The Future of Databases Is Now

(Immersion Imagery/Shutterstock)

Databases–those workhorses of data management–have come a long way over the past 10 years. Users no longer must accept the tradeoffs that were commonplace in 2013, and the array of features and capabilities in relational and NoSQL databases is growing every month. In some ways, we’ve already arrived at the glorious future data architects envisioned for us way back then. So what’s holding us back?

The biggest change in databases is the cloud. While you could get a DynamoDB instance from AWS as far back as 2012, the cloud was pretty much an afterthought for databases. But thanks to continued investments by database vendors, cloud database services greatly improved, and by 2018, Gartner estimated that managed cloud database services accounted for $10.4 billion of the $46.1 billion DBMS market, or about a 23% share.

One year later, Gartner went out on a limb when it declared that cloud had become the default deployment method for databases. “On-premises is the new legacy,” the analysts declared. By 2020, thanks to COVID-19 pandemic, cloud migrations had kicked into high gear, and managed cloud deployments accounted for $39.2 billion in revenue by 2022, a whopping 49% of the total $80 billion market, Gartner found.

Today, the cloud is the default mechanism for database deployments. Database vendors work hard to eliminate as much as the complexity as possible from deployments, utilizing containerization technology to create serverless database instances that scale up and down on demand. While data modeling continues to occupy customers’ time, operating and managing the database has practically been eliminated.

A Modern NoSQL Database

Ravi Mayuram, the senior vice president of products and engineering at NoSQL database vendor Couchbase, remembers the bad old days when database administrators (DBAs) would dictate what could and couldn’t be done with the database.

“We need to go to a place where the front- and back-end friction should go away, where more of the operational tasks of the databases are hidden, automated, and made autonomous,” Mayuram says. “All that stuff should basically go away when you’re getting to a point where the database is on tap, so to say. You just have a URL end point, you start writing to it, and it takes care of the rest of the stuff.”

NoSQL databases like Couchbase, Cassandra, and MongoDB emerged in response to limitations in relational databases, specifically the schema rigidity and lack of scale-out capabilities. Developers love the schema flexibility of document databases like Couchbase and MongoDB, which store data in JSON-like formats, while their distributed architectures allow them to scale out to meet growing data needs.

Since their introduction, many NoSQL vendors have also added multi-modal capabilities, which allows the database to shapeshift and serve different use cases from the same data store, such as search, analytics, time-series, and graph. And most NoSQL vendors have even embraced good old SQL, despite having their own query language optimized for their particular data store.

That flexibility appeals to customers and prospects alike, Mayuram says. “In Couchbase, you can write the data once and I can do a key-value lookup, I can do a relational query on it, I can do a full ACID transaction on it, I can search tokens. I can do analytics,” he says. “It’s more like a smartphone. It’s about five different data services in one place.”

Like Teslas, the differences with modern databases are under the hood  (Alexander Kondratenko/Shutterstock)

While Couchbase delivers some of the same functionality as a relational database, it goes about it in a completely different manner. Newer databases, such as Couchbase’s, are completely different animals than the relational databases that have roamed the land for the past 40 years. Today’s modern databases are more complex in some ways than the old guard, and it will take some time for enterprise to adjust to the new paradigm, Mayuram says.

“Sometimes you have to go slow to go fast,” he says. “There is going to be an amount of time in which we have to carry sort of both sides, if you will, until we can sort of cut over. That is not an easy task. It’s a generational shift. It’s going to take a little bit of time before your investment that you made in the past has to be transformed to the investment that we make for the future. There is a learning curve as well as an experience curve that you will go through.”

Familiarity will be critical to giving customers a sense of comfort as they slowly swap out the old databases for the newer generation of more-capable databases, Mayuram says.

“You can say there is no difference between Tesla and a regular car because it’s got the same steering wheel, the same tires, the same gas pedal, so what’s the difference?” he says. “What we are losing is our comfort. We just need to go to the next level to tackle the problem. Tthat doesn’t mean you break away completely. You have to have the same SQL available to you. It’s the same steering wheel. Don’t take away the steering wheel. That’s where the comfort lives. Change the gas engine, which is saving all the pollution and, you know, dependency on oil and all that stuff. Change that.”

New Relational DBs

A similar but slightly different journey has taken place in the world of relational databases, which has seen its share of new entries. Vendors like Cockroach Labs, Fauna, and Yugabyte have sought to remake the RDBMS into a scale-out data store that can provide ACID guarantees for a globally distributed cluster. And like their NoSQL brethren, the new generation of relational databases can run in the cloud in a serverless manner.

Yugabyte, for example, has found success by fitting the open source Postgres database into the new distributed and cloud-first world. “Our unique advantage is we don’t enable one feature at a time,” says Karthik Ranganathan, Yugabyte’s founder and CTO. “We enable a class of features at a time.”

By starting with Postgres, YugabyteDB ensures compatibility not only with applications that have already been built for Postgres, but also ensures that the database works with the large ecosystem of Postgres tools, Ranganathan says.

However, unlike plain vanilla Postgres, YugabyteDB is a full-fledged distributed database, providing ACID guarantees for transactions in globally distributed clusters. Not every organization needs that level of capability, but the world’s biggest enterprises certainly do.

Yugabyte wraps that Postgres compatibility and distributed capability in a cloud-native delivery vehicle, dubbed YugabyteDB Managed, enabling users to scale their database clusters up and down as needed. In addition to scaling out by adding more nodes on the fly, YugabyteDB can also scale vertically.

Yugabyte has brought together all of these features into a single package, and it’s resonating in the market, Ranganathan says.

The cloud is now driving the bulk of database revenue, per Gartner

“You need the availability, resilience, and scale in order to be cloud-native because cloud is, after all, commodity hardware and it’s prone to failures and it’s a bursty environment,” he says. “And all of the features are there and the architectural way of thinking how to build an application [is there]…because they have the ecosystem, the tooling and the feature set. So that marriage has been amazing, and we’re getting incredible pull from companies.”

Many enterprises that would have traditionally looked to the trusted relational database vendors–the Oracles, IBMs, and Microsofts of the world–are looking to open source Postgres to save money. They’re short-listing the Postgres offerings from cloud vendors, such as Amazon Aurora and Amazon RDS, and giving YugabyteDB a try in the process.

YugabyteDB is winning its share of business. Kroger, for example, relies on the database to power its ecommerce shopping cart. Another customer is General Motors, which uses YugabyteDB to manage data collected from 20 million smart vehicles. And Temenos, which is one of the world’s largest banking solutions providers, is also running core transaction processing on YugabyteDB.

Ranganathan admits that some of this success is luck. He certinaly couldn’t have foreseen that Postgres would become the world’s most popular database when he and his colleagues started work on the stealth project 10 years ago. But Ranganathan and his colleagues also deserve credit for doing the hard work to create a database that contains the other features enterprises want, which is the resiliency and scale of distributed processing and the ease of use that comes with cloud.

“Sometimes we get pulled into the conversation by the research the customers do and they tell us ‘Can you help us with this?’ So we’re kind of getting the ask handed to us,” Ranganathan says. “It’s still difficult. Don’t get me wrong…But we just really love the place we’re in and where the market is pulling us.”

The times are changing when it comes to databases. Today’s cloud-native databases provide better scalability, more flexibility, and are easier to use than the relational databases of old. For customers looking to modernize their data workhorses and reap the data-driven benefits that come with it, the future has never looked brighter than it does right now.

Related Items:

Cloud Databases Are Maturing Rapidly, Gartner Says

Cloud Now Default Platform for Databases, Gartner Says

Who’s Winning the Cloud Database War