Neon Looks to Light Up Cloud Postgres Market

When it comes to relational databases, Postgres reigns supreme, especially in the cloud. However, running the open source database in the modern cloud manner leaves something to be desired. That’s the functionality gap that NewSQL database veteran Nikita Shamgunov is hoping to fill with his latest startup, Neon.

Shamgunov was a co-founder and later CEO of MemSQL, a distributed SQL database that can simultaneously handle analytical and transactional workloads. Now called SingleStore, the super-scalable database continues to successfully serve the high-end of the market, Shamgunov says. But when it comes to the bulk of transactional workloads on relational databases, Postgres is the hands-down winner.

“Postgres is basically unstoppable at this point,” Shamgunov tells Datanami in an interview last week. “It’s becoming Linux.”

The data certainly back that up. Last month, Postgres was named the database of the year for 2023 by DB-Engines.com. The database was the number one database in Stack Overflow’s 2023 Developer Survey, besting database stalwarts MySQL, SQL Server, and MongoDB.

Its plug-in architecture allows Postgres to quickly and easily adapt to handle different data types like time-series, geolocation, and vector embeddings, which has made it the Swiss Army Knife of relational databases. All that’s missing is a column store for analytical workloads, but “the Postgres ecosystem will probably eventually solve that,” Shamgunov says.

Postgres has emerged as the number one database in the cloud (monticello/Shutterstock)

All three cloud giants offer Postgres as a service, but AWS is the undisputed heavyweight champion in this fight. According to Shamgunov, Amazon Aurora pulls in $4 billion per year while Amazon Relational Database Service (RDS) pulls in $7 billion per, amounting to 11% of a global database market that Gartner estimated was worth $100 billion in 2023. “Everything else is just a rounding error,” says the former Microsoft SQL Server engineer.

While Postgres dominates in the cloud, the database does so without the sort of features and capabilities one would expect in this day and age, Shamgunov says. Companies like AWS and Google Cloud have done the engineering work to separate compute and storage in their Postgres offerings, which allows them to deliver serverless Postgres instances that can be spun up and spun down on a dime. However, these are not open source offerings. At the end of 2024, Aurora Serverless V1, which spins all the way down to zero, will be put out to pasture, to customers’ great chagrin.

What the database market lacks, Shamgunov says, was a serverless Postgres offering that developers can easily spin up in the cloud while simultaneously being open source and maintaining full compatibility with the massive open source Postgres ecosystem. That’s essentially what has been delivered with Neon, which Shamgunov co-founded in 2021 with Postgres contributor Heikki Linnakangas and Stas Kelvich.

The startup, which came out of stealth in June 2022, focused early on the hard engineering work of separating compute from storage in the database, which is necessary to deliver a serverless experience. The company developed its own storage engine for Postgres that enables it to use Amazon S3 as backend network storage for the database, without introducing incompatibility in the data stream.

“What we’ve done is we’ve separated that storage and moved it into network attached storage that is custom built for Postgres,” Shamgunov says. “The API is not a file system API. It’s the API that Postgres understands.”

Nikita Shamgunov is the CEO and co-founder of Neon

The Neon storage engine plugs into Postgres at “an incredibly low level,” which is a key factor enabling full Postgres compatibility, Shamgunov says.

The Neon storage engines is composed of two parts: The Pageserver component, the scalable storage backend that sits next to the compute nodes, and the Safekeepers, which serve as a redundant write ahead log (WAL) service that receives WALs from the compute node and stores it durably until it’s been processed by the Pageserver and uploaded to cloud server, according to the Neon GitHub page.

As long as the Neon storage engine returns the data within the timeframe expected, the query engine doesn’t know the difference, Shamgunov says. That means that nothing else in the Postgres stack is impacted, and all the of Postgres extensions and applications just work, he says.

“It’s super important for us be 100% compatible with Postgres,” he adds, “and also position ourselves as Postgres, not some other database.”

This approach brings several benefits, starting with practically unlimited scalability, Shamgunov says. Since Neon is built upon a shared-storage architecture as opposed to the shared-nothing architectures that other Postgres-compatible databases use, it scales basically linearly based on how many read replicas you have, he says.

“With shared-storage system like us, AWS Aurora, and [Google Cloud’s] AlloyDB, your compute for each query is a single node compute,” Shamgunov explains. “You can have multiple read replicas there, but each individual query is processed by a single node compute. But that compute is attached to storage, and storage is distributed, so you can basically push your IOPS onto the distributed storage. Now are your IOPS are kind of infinite.”

Developers also benefit from this approach, Shamgunov says. Developer activities like cloning or branching a database are relatively trivial acts, thanks to the serverless characteristic of Neon. That makes Neon much easier to work with for developers, he says.

“When you look at databases today, they’re nowhere to be found there. They’re not built for modern cloud consumption and they’re not built from modern developer lifecycle,” Shamgunov says. “The foundational feature of that is the ability to branch. Just like Git allows you to branch things, Neon allows you to branch things. So you can have a database in production and the database is the URL. So we have a URL, which represents your database in the cloud. You can branch it. Now you have a different URL and you instantly have a full copy of that data with a with a separate endpoint, which is isolated also.”

When a developer builds an application, they can branch the database on every pull request or even on every commit, the Neon CEO says. “So now you have breadcrumbs,” he says. “You can build isolated environments, which if you don’t have that feature, it’s incredibly expensive.” Neon is integrated with GitHub and Vercel for source code management, and its API can easily be incorporated into a CI/CD pipeline using a tool like Jenkins, Shamgunov says.

Microsoft offers similar developer-centric capabilities with SQL Server Hyperscale, says Shamgunov, who previously worked on the SQL Server team. However, that database is not compatible with Postgres, which puts it at a disadvantage in today’s database marketplace.

The Neon database is available under a permissive Apache 2.0 license from the Neon GitHub project, which sports more than 11,000 stars. Users are free to download the source code and compile their own Postgres database. Snowflake has even adopted open source Neon into Snowpark, Shamgunov says.

In addition to the open source bits, the company is also offering an enterprise version of Neon that it hosts for customers in the cloud, a la the MongoDB or Databricks models, he says. “This is Mongo Atlas for Postgres,” he says.

Alternatively, developers can spin up their own hosted database under the Neon Free Tier, which is offered as a technical preview. Free Tier customers are allowed one Neon project with up to 10 branches, with 3GB of storage per branch. Neon is currently managing more than 500,000 database environments, the company says.

Shamgunov has finally built a database that retains what he believes are the two most critical characteristics that a modern database must have: a cloud architecture, which delivers scalability, and open source, which removes lock-in (or fear of lock-in). SingleStore/MemSQL had cloud scalability, but that database was never made open source. Amazon Aurora, the $4 billion Postgres juggernaut, similarly is not open source, makes it vulnerable to Postgres adopters who demand openness, Shamgunov says.

With so much momentum developed in such a short time, the future certainly looks bright for Neon. The company isn’t profitable yet, but it’s signing up new users at a rapid rate, with the hope that it will convert them into steady paying customers. The company so far has raised $104 million across five rounds, including a $46 million Series B in August 2023 that was led by Menlo Ventures with participation by the venture arms of Databricks, Snowflake, and Google.

“This architecture is just the right one, and then value starts getting layered on that architecture like Lego bricks,” Shamgunov says. “It’s heavily inspired by Amazon Aurora, but think of it like V3 of Aurora. If V1 storage is Aurora, V2 storage is Microsoft SQL Server Hyperscale, then V3 is a re-implementation that takes all the learnings from those two systems and comes up with a modern implementation of storage.”

Related Items:

AWS Cancels Serverless Postgres Service That Scales to Zero

Postgres Rolls Into 2024 with Massive Momentum. Can It Keep It Up?

How Broad Is Your Database’s Data Ecosystem? Gartner Takes a Look