Why Object Storage Is the Answer to AI’s Biggest Challenge

(Michael Traitov/Shutterstock)

The COVID-19 pandemic has underscored just how critical it is to be able to quickly analyze and interpret data — and the invaluable role that artificial intelligence and machine learning play in savvy decision-making. In the quest for a vaccine, the whole world has witnessed a very practical (and life-saving) application of machine learning, which drives the required training and ongoing fine-tuning of models used by AI for inference in real-time.

Because better trained models result in faster and more accurate AI, it stands to reason that AI’s biggest challenge is properly training its ML models.

Well-trained ML models must be fed a steady diet of big data so they can adapt and improve. As the training data sets grow, learning algorithms perform better and become more accurate. Simply put: the more data, the better the outcome. Obviously, huge amounts of data call for huge amounts of storage, but all storage solutions aren’t created equal in this context. As enterprises evaluate how they can best leverage their own AI/ML applications, it’s imperative that they don’t overlook storage infrastructure in the process.

An organization’s ability to successfully sift through and glean actionable insights from the massive, ever-growing data sets necessary for model training is dependent on having a storage architecture that can keep up with exceedingly demanding requirements across all stages of the data pipeline.

7 Reasons Object Storage Is a Must-Have for Supporting Effective ML Models

Here’s why object storage is the most suitable and, quite frankly, the only adequate solution to help solve AI/ML’s model-training challenge:

(CG Alex/Shutterstock)

• Infinite scalability: Huge amounts of data necessitate huge amounts of storage, and AI/ML workloads require a solution that can infinitely scale as the data grows. Legacy file and block storage solutions will hit a scalability ceiling after a few hundreds of terabytes. Object storage is the only storage type that can scale limitlessly to tens of petabytes and beyond within a single, global namespace. Being able to scale elastically and seamlessly based on demand, by deploying new nodes non-disruptively, whenever and wherever needed, is a great advantage.

• Built-in data protection: Regularly backing up a multi-petabyte training data set is not only cost and time prohibitive, it’s downright unrealistic. Most object storage systems, by design, do not require backups. Rather, they store data with sufficient redundancy so it’s always protected. Because object storage solutions are often designed as a distributed architecture — a collection of distributed servers operating in parallel requiring no special machine(s) to provide or manage specific services — all responsibilities are divided and don’t require a central ‘control’ machine. Thus, there’s zero risk of a single point of failure (SPOF) in the architecture. Distributed object storage systems offer extreme data durability with self-healing capabilities. A system may be configured to tolerate the failure of multiple nodes or even an entire geo-distributed data center.
• Inherent metadata search and classification: An absolute must in the data preparation phase required for building and training effective ML models is detailed and descriptive metadata, which makes it possible to easily tag, search for, locate and analyze data. Storage architecture influences the ability to gather metadata. Whereas file and block systems do not enable application or user-defined extended attributes, object storage systems offer unique ways to identify data with incredibly rich, customizable metadata. Its unrestricted nature enables easy tagging, robust and lightning-fast searchability as well as efficient management of huge data sets.

• Multi-tenancy functionality: Isolating workloads through multi-tenancy allows multiple teams of data scientists to simultaneously work with the same data source without impacting each other or competing for resources. Object storage systems designed to service multi-tenant use cases make it simple to securely manage tenant data from within a single scalable, AWS S3-compatible interface.

• Sustained throughput performance for shorter training time: The ability to sustain the data pipeline at an optimal rate is crucial for training ML models. Infrastructure efficiency is required to yield rapid results, otherwise the calculations that run on vast data sets will be slowed or interrupted. Modern object storage systems maintain high data throughput and, furthermore, enable scale-out to increase overall system capacity and performance, both independently and linearly. This is achieved by adding system resources in the form of storage servers, which provide the compute (CPU and memory) and capacity (flash and HDD) to be managed by the storage software as a single pool.

• Lingua franca for AI/ML algorithms run in the cloud: No matter where data resides, integration with the public cloud is important, especially as public cloud platforms offer some ready-made and attractive tool sets for AI/ML.. Of all storage architectures, object storage is the most suitable for training and tuning ML models because it permits seamless access and mobility between on-premises/private cloud environments and public cloud storage thanks to its de facto language — AWS S3 API. The best object storage solutions enable users to manage cloud-based and local data within a single, unified namespace, eliminating data silos and allowing resources to be used cooperatively and interchangeably without any loss in functionality no matter where they are.

• Low total cost of ownership (TCO): A storage infrastructure designed for AI/ML workloads must provide not only capacity and performance, but also cost-effectiveness with regard to storing, moving and managing multi-petabytes of data required for optimal model training. By leveraging standard server technology, and the ability to operate at large scale in a single system, object storage delivers that in spades, coming in at a fraction of the cost of traditional proprietary enterprise storage. Software-defined solutions can be hosted on affordable standard x86 servers and grow across multiple hardware generations to reduce cost.

Enterprises seeking to realize the full value of their AI applications must understand the critical nature — and potential challenge— of properly training and fine-tuning their ML models. The smart ones will be just as conscientious about choosing the right storage infrastructure as they are about compute requirements. The wisest will come to the conclusion that object storage solutions provide the optimal foundation for ultimately extracting fast and accurate analytical insights, life-saving and otherwise.

About the author: Maziar Tamadon is product and solution marketing director at Scality. Prior to joining Scality, Maziar held product marketing, product management and engineering positions at Seagate, Broadcom, Emulex, Brocade, and Hewlett-Packard, in the US and France. He holds MS in EE and CS from the Institut National Polytechnique de Grenoble in France.

Related Items:

Fast Object Storage: Meeting the Demands of Modern Data

The Object (Store) of Your Desire

Rethinking Architecture at Massive Scale