3 ms·

Architecture Decisions in Neon

48 points by nikita 4y ago

fishtockos 4y ago
> In Neon, incoming WAL is processed as it arrives, and indexed and buffered in memory. When the buffer fills up, it is written to a new file This sounds cool, but what happens if the server crashes? How do you prevent losing the buffered WAL? Do you have WAL of WAL or something?
nikita 4y ago
Heikki was behind major decision and has done a lot of prototyping. Happy to add more color
dfee 4y ago
Cool to see this (I’d somehow missed the early access link until last night). Your storage strategy makes sense - building from first principles. To be honest, I figured you’d not done that though, and I imagine this makes portability a bit harder. I’m eager to play with the service, but I wonder how local access patterns (e.g. dev workflows) differ - upon the right abstraction it shouldn’t really matter, right? I.e. a conn is a conn and a cursor is a cursor - the storage is opaque to the client. But does this somehow fall over with transactions and connection limits?
- nikita 4y ago
  Please email beta@neon.tech and we will bump you up on the waitlist. Portability is key - we can't fork Postgres. Apart from branching it's the same Postgres - so you can develop locally against vanilla Postgres. We are thinking how to connect local and cloud environments. What would be some good patterns here?
infogulch 4y ago
Splitting the architecture across compute/storage and then pageserver/walserver seems like a great design decision because the pieces fit snugly onto the underlying layers of cloud services. The design is general enough that it could work for other block-device-based databases (~all of them) so you could put a SQLite or MySQL compute layer on top of the storage layer instead of postgres. Discussion: https://github.com/neondatabase/neon/discussions/1958 https://github.com/neondatabase/neon/discussions/1958 The fact that they're targeting vanilla postgres is also intriuging because that means it will be a lot easier to integrate with other interesting PgSQL-based technologies; EdgeDB in particular seems like it would be a pretty great fit.
- nikita 4y ago
  As a matter fact lots of things can potentially benefit. Elastic, duckdb, but more importantly the new projects. I was discussing with Wes that we can put arrow compute on top of neon storage.
  - nikita 4y ago
    And yes. MySQL people reached out as well. Off strategy for us right now, but definitely doable.

48 points by nikita 4y ago

> In Neon, incoming WAL is processed as it arrives, and indexed and buffered in memory. When the buffer fills up, it is written to a new file This sounds cool, but what happens if the server crashes? How do you prevent losing the buffered WAL? Do you have WAL of WAL or something?

nikita 4y ago

Heikki was behind major decision and has done a lot of prototyping. Happy to add more color

dfee 4y ago

Cool to see this (I’d somehow missed the early access link until last night). Your storage strategy makes sense - building from first principles. To be honest, I figured you’d not done that though, and I imagine this makes portability a bit harder. I’m eager to play with the service, but I wonder how local access patterns (e.g. dev workflows) differ - upon the right abstraction it shouldn’t really matter, right? I.e. a conn is a conn and a cursor is a cursor - the storage is opaque to the client. But does this somehow fall over with transactions and connection limits?

nikita 4y ago
Please email beta@neon.tech and we will bump you up on the waitlist. Portability is key - we can't fork Postgres. Apart from branching it's the same Postgres - so you can develop locally against vanilla Postgres. We are thinking how to connect local and cloud environments. What would be some good patterns here?

infogulch 4y ago

Splitting the architecture across compute/storage and then pageserver/walserver seems like a great design decision because the pieces fit snugly onto the underlying layers of cloud services. The design is general enough that it could work for other block-device-based databases (~all of them) so you could put a SQLite or MySQL compute layer on top of the storage layer instead of postgres. Discussion: https://github.com/neondatabase/neon/discussions/1958 https://github.com/neondatabase/neon/discussions/1958 The fact that they're targeting vanilla postgres is also intriuging because that means it will be a lot easier to integrate with other interesting PgSQL-based technologies; EdgeDB in particular seems like it would be a pretty great fit.

nikita 4y ago
As a matter fact lots of things can potentially benefit. Elastic, duckdb, but more importantly the new projects. I was discussing with Wes that we can put arrow compute on top of neon storage.
- nikita 4y ago
  And yes. MySQL people reached out as well. Off strategy for us right now, but definitely doable.