4 ms·
I'm pretty bullish on this tech. A clean, low-friction separation of storage and compute for a top-tier database like Postgres is huge. I think it will take so
by jmull 3y ago
I'm pretty bullish on this tech.
A clean, low-friction separation of storage and compute for a top-tier database like Postgres is huge. I think it will take some time for the implications to sink in though, because it changes some assumptions about how to integrate a databases with a system that are so fundamental that we rarely think about it... (not to mention, it probably doesn't make sense to change existing systems, where the fundamental system decisions have already been made.)
We tend to centralize the database in system design because they are complex and expensive to run, and you can amortize those costs the more you put into a single database. But of course, that also makes the central database bigger, more complex and more expensive. And once everything is in a central store, we tend to bind everything together (with joins, references, procs, etc. Each step in that direction tends to make sense on its own, but the end result is the "big ball of mud" architecture where everything is interconnected and there's no separation of concerns, which makes the system fragile and increasingly difficult to change.
With this tech the pressure to centralize the database is relieved.
E.g., if you have a situation where you are adding some new, logically distinct data to your system -- maybe just a few tables and maybe with a variable load, especially at first -- traditionally you probably would lean hard toward adding this to an existing central database since the overhead of spinning up and running a new one would be prohibitive. But now you can just make that a separate database -- spinning up a managed one is easy and you only need to pay for as much storage and compute as you actually need to use (which is really about the same as if you had put the tables in an existing central database anyway).
Of course, systems would still tend to have significant central data, but that could be based on actual app concerns, not operational ones.
Anyway, I'm loving this and can't wait to see the long term impact.
You can have as many databases as logically makes senseseparate data logically without exploding the operational cost of your system. E.g. you can spin out a separate a database that has light (or even no load) most of the time yet can seamlessly scale to heavy load. And it can do this independent of how much data is in it, and you don't need to worry about it's affect on day-to-day data access in your system since you
- atombender 3y agoWe already have it with YugabyteDB, which is open source (unlike Neon), but doesn't seem to be the success story it should have been. At least I've heard barely a peep about it. Maybe they're not so great at marketing. Yugabyte is architected similarly to other similar databases, with an SQL layer on top of a distributed, replicated key/value store based on RocksDB. The different being that the SQL layer is Postgres. Yugabyte tightly couples the two, though, though it seems like this is mostly for convenience, in that the "TServer" (tablet server) is managed by Yugabyte itself. I wonder if it could support an architecture more similar to TiDB, where the storage is independently scalable from compute. CockroachDB and TiDB are based on a similar design, but Yugabyte stands out as being a "real" Postgres. It even supports normal Postgres extensions via "CREATE EXTENSION".
- eqoram 3y agoDid I miss something? I always thought that Neon is open source.
- atombender 3y agoFor some reason I thought it wasn't open source. Looks like I'm wrong!
- vineyardmike 3y agoI mostly agree with the commenter you’re replying to - but I think you bring up a good point. Once you start disconnecting tables from one another, and end up with lots of small data stores, does it still make sense to bother with the Postgres at all? I think that its value as a layer becomes unclear. Sure its battle-tested and has great query planning and similar tools, but those base based on assumptions that are largely violated with the re-architecture. I think the architectural decoupling benefits mentioned are basically already proven though. I’ve worked at places that had many NoSQL tables per team, each logically distinct, and at places that had one database with 50+ tables in it. The breakup can be nice, in the same way that “serverless” is nice - data storage becomes entirely abstracted to the developer.
- nikita 3y agoNeon is open source and under a good license: Apache 2
- nikita 3y agoThe problem that shared nothing systems lose compatibility with the single node system. So planetscale and yugabyte are not 100% compatible with Postgres. They are also very expensive at the low end hence the free tiers are discontinued. I have a thread about it on twitter: https://x.com/nikitabase/status/1723579854266966399?s=46 https://x.com/nikitabase/status/1723579854266966399?s=46
- nikita 3y agoOne thing that is very important is what you charge for. Software or service. We are only charging for the cloud service and it has major implications in the “company architecture”. When you charge for software like yuga and roach and for the cloud you end up with two companies in one. Counterexample is snowflake - they got it right !
- nikita 3y agoThis is pretty much the first part of the strategy: database is just a url. We have more cool stuff coming!