22 ms·

Neon – Serverless Postgres

667 points by nikolay 4y ago

devit 4y ago
Why not use normal networked storage (nbd/drbd, AWS EBS, etc.) instead of the ad-hoc Pageserver + Safekeeper architecture? Or even better, use simple local SSD/HDD storage where data is small enough.
- mattashii 4y ago
  Because all of those don't or can't have both low-latency reads and regional persistence guarantees, which is what we're after.
XorNot 4y ago
Well this seems like a big deal. "Needs a database" has been an ongoing pain point for working with serverless apps. Staying in the postgres command library, if it works, means you can now just prototype and assume "postgres will be there". I have a couple of apps I would've used this exact service for in the recent past. Looking forward to trying it in the future.
riedel 4y ago
This sounds really interesting! I wonder what kind of scaling use cases neon is good for. Is it e.g. good for custom scenarios like a geospatial timeseries database on top of postgres? We have admittedly not really a clue about current database cluster tech as we are IoT/ML researchers, but we are running a custom timescaledb cluster that receives constant nonchunked write load from a lot of devices and may encounter some long running queries on an around 500GB DB filled with geolo (even timing out if users are too creative), why we splitted into a single ingress master and multiple outgres WAL readonly replicated query clients to relax the consistency and sync, that seemed to be killing us (we need postgres because of postgis and have no capacity to rewrite the front-end). I wonder if neon would be good for such a use case and if it easily supports postgres extension like timescaledb hypertables and postgis). Most of the time our system just measurements, but sometimes we really need to scale up for PoCs, which makes dimensioning really hard (for us).
- mattashii 4y ago
  Neon has the ability to make a Copy-on-write replica of a database, which for you would allow you to create an instant read-only copy of the data that has been ingested up to that point, without significant storage overhead. The new data would still be written to the primary database, and long running queries would only see the snapshot that the instance was started with (using their own pool of CPU and memory resources). Assuming that the extensions that you use are compatible (that is, they don't access database files in a way that PostgreSQL doesn't, and the licence is compatible) then Neon could be a good solution to that issue.
pgt 4y ago
Incredibly good sales copy by someone who understands their target market.
ranguna 4y ago
Just yesterday I was comparing managed serverless postgres offers and was sad to temporarily end my investigation with a compromise of using managed aws RDS for development, hoping that a fully serverless postgres with a nice free tier would pop up before going to production, and here we are! Congrats to the team for what feels like an amazing product. Signed up for the early access, can't wait to get my hands on this! For anyone interested, these ere the DB offers I looked into: * DO managed postgres, no free tier but price scaling was not too aggressive, the issue is that it's not natively serverless and we're gonna get 100s of ephemeral connections. * Cockroach, was the best option for our use case but it doesn't support triggers and stored procedures, so we can't use it right now (closely following https://github.com/cockroachdb/cockroach/issues/28296 https://github.com/cockroachdb/cockroach/issues/28296) * Fly.io price scaling is too aggressive 6$ -> 33 -> 154 -> 1000s a month and no free tier that I could find. * Aurora serverless v2 is only for aws internal access and we are using gcp. * Aurora v1 was what we were gonna go with, but a lot of people online have showed their negative opinion around slow scaling. I didn't investigate enough but I'm thinking we'd need to setup RDS proxy for it handle all our connections, which would've bumped up the price by a good amount. Also no free tier. * Alloydb looked promising but also no free tier and starting price is a bit much for our current phase of development, but it was definitely something we'd look into in the future. And now Neon, natively serverless with a (hopefully) good free tier to test things out and some hints about cross region data replication, amazing stuff!
- lysecret 4y ago
  "Aurora serverless v2 is only for aws internal access and we are using gcp." You can have public access to serverless v2. I'm using it with retool for example. That said I moved a Postgres DB to Aurora, the process was hilarious in how crazy it was. Also they haven't implemented scaling to 0 yet!!!! And the minimum 0.5 Compute unites are actually pretty expensive.
  - ranguna 4y ago
    Nice point about the minimum 0.5 ACU, forgot about that one. From what I've read 0.5 on v2 is the same price as 1 on v1, which seems pretty dumb coming from aws. Could you elaborate more on this: > You can have public access to serverless v2 Because the docs mention the following: > You can’t give an Aurora Serverless DB cluster a public IP address; you can only access it from within a VPC based on the Amazon VPC service. Potentially I could setup an RDS proxy or vpn inside the vpc and give that public access, but that seems a bit of a roundabout way of handling this. https://aws.amazon.com/blogs/database/best-practices-for-working-with-amazon-aurora-serverless/ https://aws.amazon.com/blogs/database/best-practices-for-wor...
kosolam 4y ago
Cool! Can be used in production already?
- mattashii 4y ago
  We cannot yet recommend users to run their critical workloads on Neon.
- kelvich 4y ago
  We are planning to open public tech preview next month. We'll go to general availability when we see a decent SLO for several months. Tech preview should be okay for hobby projects and staging workloads, though.
rad_gruchalski 4y ago
What’s the licensing story? Can I run this in my org and expose this as a service to users outside of my org? Or does it have strings attached, like CockroachDB? Asking because github says Apache 2 but the devil is in details.
- kelvich 4y ago
  You can, no strings attached
  - rad_gruchalski 4y ago
    Awesome, will take it for a spin.
- mattashii 4y ago
  Yep, for now the storage engine and the modifications on PostgreSQL are licensed under APLv2. Our cloud offering which orchestrates the scale-to-0 and other glue between the components are closed-source and not covered by that license. Do note however that the licensing story is not entirely fleshed out yet, as the product we're building is still in closed beta. As we work towards a paid cloud offering, we'll further flesh out the license model for the code, but for now we're planning on keeping this license.
smallerfish 4y ago
Could you document some of your differentiation against aurora, both on price and architecture? I don't care about scale to 0. I care about more scale to NNN, efficiently & reliably, with minimal devops needs.
- nikita 4y ago
  We will put more of the Aurora differentiation docs up in a bit. Architecture wise we are slightly different b/c of the S3 integration and splitting storage to safekeepers and page servers. This is very technical and we will be highlighting both this and user visible features, such as branching.
talkingtab 4y ago
Why is this a good idea? In my experience, getting Postgres up and running is trivial. Docker anyone? And in many cases your data is your business so why hand it off? And if you are going to offer this product why not just call it what it is, "Postgres as service", instead of serverless which seems a bit misleading. Really it is simply Postgres running on your server.
- chimen 4y ago
  Not everyone can manage a database properly and, sorry to say this to you but, Docker is a terrible idea for a database in general. Setting up your own database somewhere still puts your trivial data on someone else's server more or less. All these "Serverless" keywords pretty much mean you don't have to be spinning up servers (cloud) or setting up & maintaining one. Nothing is "Serverless" per-se so it's time to move on from picking on this, I agree, bad choice of words.
  - smokey_circles 4y ago
    > Docker is a terrible idea for a database in general Why? Genuine question, my gut feel is there's something wrong about it too but I can't put words to it nor have I found a benchmark that convinced me, but it's worth noting I'm not sure what I'm looking for
    - chimen 4y ago
      I manage about 200 servers and docker crashing accounts for 20%+ of my issues so far. The servers are brought up easily on crash and that's not an issue for my services. For a database docker is nothing more than an extra layer of complications on top with iptables, volume system and all the layers it brings. It's just a bad wrapper for a production database which needs stability.
      - Art9681 4y ago
        To be fair, that sounds like a problem with your infra. There are plenty of enterprises running Postgres in Docker/Kubernetes successfully. A lot of technical problems are not the stack or the engineer's capability, but the company culture and lack of funding, time or resources. If you had more of all 3, im sure your problem could be "solved" to an acceptable margin of error. Likely by you.
captnObvious 4y ago
I hope y’all have a plan for when AWS decides to pick up your open source project and turn it into a managed cloud solution. It’s a pattern of theirs. And with the way egress charges are structured they’re likely to snap up any clients straddling their cloud and yours.
- mattashii 4y ago
  AWS already has Aurora, which is their own in-house closed-source variant that does very similar things. We think we'll be able to provide a better experience at lower cost for smaller developers, while having some very useful quality-of-life features like zero-cost branching and instant PITR.
- onphonenow 4y ago
  AWS has Aurora Serverless v2 out already for postgresql along with RDS for postgresql. Not scale to zero though / bottom is 40/month or so
- oxfordmale 4y ago
  AWS already has this, Google for Aurora Serverless. It is not cheap though, and this might well be cheaper.
SonOfLilit 4y ago
> Neon allows to instantly branch your Postgres database to support a modern development workflow. You can create a branch for your test environments for every code deployment in your CI/CD pipeline. > Branches are virtually free and implemented using the "copy on write" technique. Unless I missed that everyone supports this, this here could be a killer feature and should be advertised higher.
- jhgb 4y ago
  It sounds like something you might be able to accomplish with a copy-on-write VFS on top of a Firebird database file. (Not sure about PostgreSQL, but with Firebird, you only deal with one file, so with Firebird, this should definitely work.)
  - nikita 4y ago
    There is an enterprise company called Delphix that does it on top of Zfs - so the idea was in the air. Instead of duct taping this together with a filesystem we purpose built database storage. The advantage to this is that we can much tighter control execution paths and can profile them end-to-end. Additionally this allows us to integrate with S3 and make it much much cheaper to run.
    - jhgb 4y ago
      Technically Firebird just requires a block device, so you might not even need a filesystem.
- zxspectrum1982 4y ago
  You can get that feature on any Postgres server by installing Citus
  - mattashii 4y ago
    Does Citus provide any such storage-level multi-cluster features? I can't seem to find any documentation on that...
funstuff007 4y ago
How do I keep my application servers physically close to Neon?
- mattashii 4y ago
  Right now, we have our hosting in AWS (us-west-2). We're planning on expanding to other regions and cloud providers eventually, though.
  - GordonS 4y ago
    Having data in Europe will be important for many users, hope you expand to EU soonish.
Sytten 4y ago
I am glad someone is taking on this challenge! Its been a few years since I was saying that the last piece of the serverless puzzle is a good postgres serverless database. If I was not working on my startup I would apply for sure. It would be nice to present the project on the CMU database group youtube channel at some point to dive into the implementation.
- nikita 4y ago
  I'll work with Andy Pavlo who is a friend to set up a presentation. And yes, we are hiring! So if and when you are ready let us know.
jasonhansel 4y ago
Does this handle sharding? Or is it like ordinary Postgres, which only performs replication?
- mattashii 4y ago
  It is ordinary PostgreSQL, in which only the storage interface with the file system is replaced with our own storage interface.
edude03 4y ago
This is really amazing, super excited to try it out. I read the docs and I noticed you can run it locally, but has the kubernetes bits been made available? I see https://github.com/neondatabase/helm-charts https://github.com/neondatabase/helm-charts and https://github.com/neondatabase/neon/tree/main/.circleci/helm-values https://github.com/neondatabase/neon/tree/main/.circleci/hel... but I think there is some charts missing?
- mattashii 4y ago
  Correct: we do not yet use k8s for provisioning the Safekeepers and PageServers for our closed-beta cloud offering, and the PostgreSQL instances are managed in k8s by our closed-source console. As such, there's little we can open-source at the orchestration level at this point in time.
  - chwzr 4y ago
    Is there a timeline on releasing orchestration things?
madelyn 4y ago
This is really interesting. Are there restrictions or limitations on the PageServer + Safekeeper design when running OLAP queries on larger datasets? Phrased another way, would a query that needs to access a relatively large amount of data (10-100 GB) ever need to read from s3, incurring extra latency?
- mattashii 4y ago
  In general, all data of live clusters will also be stored locally at a PageServer. Only in recovery scenarios will a PageServer not hold the data that is needed to serve the requests of a compute node - but that would recover quickly as the local cache of the PageServer is repopulated with data from S3.
  - madelyn 4y ago
    Awesome, thank you!
cja 4y ago
In what way is this serverless? Surely it's running on a server?
- ttfkam 4y ago
  Refers to management and autoscaling, not whether physical hardware exists somewhere. From your point of view it has DNS entries, you've defined how high/low it's allowed to scale and some other configs, you've defined network access rules, and allows client connections. That's it. You don't manage the server instances. Hence: "serverless"
  - cja 4y ago
    Thank you for explaining. There must be a better term than serverless though. "Auto-scaled cloud database" would be more accurate, for example
TobyTheDog123 4y ago
I'm a bit concerned that the free trial mentions "compute up to 1 vCPU / 256 MB" Why would I need to worry about this for a serverless database provider?
- nikita 4y ago
  We are limiting the amount of compute we are giving for free during the technical preview. We will open it up to much large computes later. Like those will be in the paid tier.
  - TobyTheDog123 4y ago
    That sounds good, but my question was (and I probably phrased it poorly), as a solo developer, why would I need to concern myself with resources associated with servers (compute, CPUs, MBs of RAM) on a serverless platform? I could maybe see it in the sense if it's only applicable when Postgres is executing complex custom queries or whatever, but for any operation? That's a huge red flag, for me at least. When I think of serverless, I think of things like Cloudflare Workers, FaunaDB, Ably.io, all of which have pricing based on the usage of their features, not the consumption of their resources. The whole point being that I can way more easily calculate how many messages my users are sending, than how much CPU all those messages are taking to send. Maybe I'm operating on the wrong definition of serverless, and all of the other features look amazing, but that concept is really a dealbreaker.
_xnmw 4y ago
Are they going to stay up-to-date with the latest version of Postgres? One problem with Yugabyte, TimescaleDB, Aurora etc. is that they are stuck on older versions of Postgres, which makes it feel like an entirely different product after a few years.
- mattashii 4y ago
  We are planning on supporting the latest stable version of PostgreSQL. Right now, we're a bit behind (we're at 14.1, latest PosgreSQL is 14.3) but that shouldn't be much of an issue. We don't yet know how we're going to do major version migrations, as the product is still not even out of private beta.
  - rattray 4y ago
    "how will this handle major version upgrades" was one of my first/biggest questions when reading through the homepage fwiw
    - nikita 4y ago
      Major versions don't come out suddenly. We have time to test and make sure they work well with our storage. Customers don't always want to upgrade to new version when old versions just work. However this can lead to version creep. Ideally we want to always run the latest version of Postgres. We will hold this line as long as possible.
      - rattray 4y ago
        Sorry, I meant less "how long until they catch up to trunk" and more "what is the downtime / operational burden of performing major upgrades like". With separate storage and compute it seems tantalizingly possible to have low-effort zero downtime upgrades for the user, which would be a huge selling point.
- akulkarni 4y ago
  "One of these things is not like the other" TimescaleDB, because it is packaged as a PostgreSQL extension (and not a fork, unlike the others), stays compatible with mainline PostgreSQL, especially as PostgreSQL improves. This is one of the key advantages of our approach. (Timescale co-founder)
nojvek 4y ago
Nikita, I am a huge fan of SingleStore - amazing what you’ve accomplished there and now what you’re building at Neon. Do you plans to build a columnstore index on top of postgres that supports insert/update/delete? Love how MSSQL has a columnstore index for a subset of columns on a row store table. Always wondered why nobody has built something like that for postgres yet. Citus is nice but it’s append only, which is a huge restriction.
- nikita 4y ago
  We are not building it in house right now - we are focusing on OLTP. But have plenty ideas of how to do it in the future. In the first iteration we are thinking about an integration with Singlestore, Snowflake, and BigQuery. Singlestore will be best due to low latency updates, but the other two are very popular so integration with Snowflake and Big Query just makes sense. In the future we can integrate a columnstore right into the engine to make a smooth on system experience. There are some awesome open source implementations: arrow and duckdb. Updatability is tricky but doable as proven by Singlestore and SQL Server (I'm ex SQL Server and a huge fan of this feature). Not this year.
  - nojvek 4y ago
    Makes sense. I agree. DuckDB is pretty phenomenal. I enjoyed reading its source code and playing around with it. Another open source nice C++ codebase is Typesense for Hyperfast text search (algolia competitor). It’s been on my mind for many months how to build indexing like this as postgres extensions. I love how versatile postgres is with so many indexing datastructures.
    - nikita 4y ago
      Again it is a big commitment to do a good job on implementing columnstores. If you don't do it all the way it not very usable and you are confusing your users by giving them too many options. The performance expectation is now set by great columstore implementations and you just can't afford a half ass job here. Here is an example from Google AlloyDB: https://twitter.com/mim_djo/status/1527900193626025984 https://twitter.com/mim_djo/status/1527900193626025984. My understanding that DuckDB is even faster on the TPCH benchmark. TPCDS is much harder and I doubt AllowDB can even run it at any reasonable scale.
rattray 4y ago
FYI, the image at the bottom says "Zenith's prices" instead of "Neon's prices" https://neon.tech/static/saas-illustration-lg-410ada378df755f415b2ab6c15364b17.svg https://neon.tech/static/saas-illustration-lg-410ada378df755...
- nikita 4y ago
  Fixed. Thank you for catching in. Renaming a company is hard even pre launch :)
  - rattray 4y ago
    I know the pain :) Good luck, this is a very exciting project. I'm extremely curious to see how it unfolds…
Komodai 4y ago
Typo in footer 'Maid in SF and the World' should be 'Made in SF and the World'
- nikita 4y ago
  Fixed - sorry, sleepless night.
rattray 4y ago
This sounds awesome, but one of my first reactions to the notion of separated compute/storage Postgres with copy-on-write is "okay, so… slow Postgres?" Is there anything the dev team can share on read/write performance compared to RDS, NVMe EC2 instances, EBS-backed EC2, etc? In what situations would this setup perform poorly, and in what situations would it excel?
cultofmetatron 4y ago
don't know too much about this yet but i've been desperately wanting there to exist a postgres equivalent to planetscale. Bravo guys! I'll keep you guys in mind for my next mvp. PS: btw why call it Neon? thats already the name of the rust/nodejs interop library which I assume you know about since your storage layer is written in rust. PS: imagine a colab between this and fly.io. I wish thsi stuff was available when I was starting Blinq
- nikita 4y ago
  We already have a shared slack channel with Kurt - CEO of fly and a number of folks from Fly.io. We are discussing our partnership.
  - cultofmetatron 4y ago
    thats awesome. what will the pricing be when its done? my startup currently spends around 400-500/month on aurora. would be curious what the differences would be here in terms of TOC. the ability to branch databasees alone is a pretty compelling killer feature imho
rapind 4y ago
I wonder how well this will work with PostgREST. Looks very interesting.
- nikita 4y ago
  We will be launching REST api with PostgREST later this year.
rektide 4y ago
Really interesting. I've seen so much disagregated database work, and so so so much of that exposes postgres interfaces. But all the good stuff has been closed source! I'm very very excited to hear about a team taking this effort to postgres itself, in an open source fashion! From the Architecture[1] section of the README: > A Neon installation consists of compute nodes and Neon storage engine. > Compute nodes are stateless PostgreSQL nodes, backed by Neon storage engine. > *Neon storage engine consists of two major components: A) Pageserver. Scalable storage backend for compute nodes. B) WAL service. The service that receives WAL from compute node and ensures that it is stored durably. Sounds like a very reasonable disaggregation strategy. Really hope to hear about this wonderful effort for many more years. Ticks the boxes: open-source with a great service offering: nice. Rust: nice.
- rattray 4y ago
  [1] https://github.com/neondatabase/neon#architecture-overview https://github.com/neondatabase/neon#architecture-overview
  - rektide 4y ago
    Oops thanks!
- nikita 4y ago
  We are committed to building a durable company and we are well funded. So yes, you will hear from us for years to come as we will be shipping more and more features.
  - avinassh 4y ago
    I could not find funding information on the Neon site. Is that information not public? edit: I found the info here: https://boards.greenhouse.io/neondatabase/jobs/4506003004 https://boards.greenhouse.io/neondatabase/jobs/4506003004
    - nikita 4y ago
thomasfromcdnjs 4y ago
Anyone got a spare invite code? =D
- felixmeziere 4y ago
  Same here please!
  - oknoorap 4y ago
    same here ^_^
    - nikita 4y ago
      We got a lot of "join waitlist" submission. Thank you HackerNews for this! Please email beta@neon.tech for the invite code. We will ask for feedback in return.
peterkelly 4y ago
Please stop this abuse of language. Here's serverless sqlite: https://www.sqlite.org/serverless.html https://www.sqlite.org/serverless.html
- dandanua 4y ago
  Modern marketing makes me mad sometimes. They call something "serverless" while in fact it runs on their servers somewhere in the world, where you have even less control than on your cloud. Though, I believe it was popularized by Amazon, which doesn't care much about being honest in its expansionism.
- jjeaff 4y ago
  From your link: Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity. Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers. It sounds like this is neo-serverless but serverless nonetheless.
  - eyelidlessness 4y ago
    It “is” in the generous sense that the quoted section interprets generously for the purpose of interpreting it generously.
- infogulch 4y ago
  This ship is so sailed already that the stories of its voyages are legendary classics that were told to our great grandparents.
  - CharlesW 4y ago
    > This ship is so sailed already that the stories of its voyages are legendary classics that were told to our great grandparents. Specifically, the term is 10 years old this October: http://readwrite.com/2012/10/15/why-the-future-of-software-and-apps-is-serverless/ http://readwrite.com/2012/10/15/why-the-future-of-software-a... It has never, ever meant that the software wasn't running on a server. From TFA: "The phrase 'serverless' doesn’t mean servers are no longer involved. It simply means that developers no longer have to think that much about them. Computing resources get used as services without having to manage around physical capacities or limits."
coder543 4y ago
> Branches are virtually free and implemented using the "copy of write" technique. Copy on write, presumably.
- kelvich 4y ago
  Fixed, thank you!
deleted 4y ago
[deleted]
anilgulecha 4y ago
This is the missing piece on cloud for masses * we already have compute scale-to-zero (cloudrun, lambda, fly.io). * Network is default pay for use. Storage (S3) is default pay for use. * The only piece in the stack that was always-on was the database (only serverless db thus far was firestore, or something like sqlite+litestream) With something like this we get a solid RDBMS engineered to be scale-to-zero, and with good developer experience. This opens up a world of try-out mini applications that cost cents to host. serverless db (postgres) + serverless compute (cloud-run) + use as you go storage+network. This is a paradigm-shift stack. Exciting days ahead.
- pid-1 4y ago
  Fly.io does not scale to zero. Lambda has many limitations. In particular, for some reason AWS is allergic to providing a container deployment service that actually scales to zero.
  - russellendicott 4y ago
    > AWS is allergic to providing a container deployment service that actually scales to zero Isn't this what Fargate is?
    - pid-1 4y ago
      No.
  - hamandcheese 4y ago
    Not yet, but soon.
- jokethrowaway 4y ago
  I don't see the benefits over a 5$ VPS. Even admitting I'll save a few cents over a VPS (which is absolutely not guaranteed), the cost saving is so minimal I won't bother rewriting everything under the serverless paradigm just for it. Of course if your cloud doesn't cost an eye and an arm. I can understand people excited to save on expensive Aws instances but maybe you should just consider dumping Aws. Scale doesn't matter for mini applications and scaling vertically (=throw money for a bigger server) will work for 99% of the companies. The 1% who need horizontal scaling will have custom everything regardless and will need to hire experts, not a good niche to release a product.
vira28 4y ago
Amazing work by the Team. Congrats y'all. It was one of the best presentations in the PGcon22. I did email Heikki the following questions, in case if someone from Neon is around here. a) How does Neon compare to polardb https://github.com/ApsaraDB/PolarDB-for-PostgreSQL https://github.com/ApsaraDB/PolarDB-for-PostgreSQL. b) The readme mentions a component "Repository - Neon storage implementation". Does it use any special FileSystem? Any links to read more about it? c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?
- ololobus 4y ago
  > c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s? Yeah, as Nikita mentioned it's 2 seconds now. We did some tests and measurements and on bare metal, it's sub 500 ms usually, so the remaining part is the k8s (+ our own control plane) orchestration overhead. For example, with plain Docker (which we use in CI in addition to k8s) it's around 1 second already. K8s provides a convenient abstraction layer, though. So I think that we'll continue using it and optimization will come with pods pool / over-provisioning and it'll be realistic to bring the startup time closer to bare-metal. -- Cloud engineer @ Neon
- nikita 4y ago
  Thank you! a. PolarDB is based on a similar idea. https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-sigmod21.pdf https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-s.... This paper describes it. The biggest difference that I see glancing through the paper is that we really integrated S3 into the storage. In Neon architecture branches, backups, checkpoint are all the same thing and instant to run. This simplifies a good amount of database management AND deliver on better costs. S3 is cheap. b. Neon doesn't need a special filesystem. Neon storage is in a way a filesystem, however it doesn't expose filesystem API. It's a key value store - serves 8k pages to Postgres and a consensus - update API to the key value store. Pages are organized in LSM trees and background processes put layers of the LSM trees to S3. c. The cols start is 2sec right now. There is a dependency on K8S. Bare metal implementation will require new code to orchestrate starts and stops.
gorgoiler 4y ago
Postgres is mind boggling, coming from sqlite. In a good way, and both are amazing tools. with ordinal jsonb_* ‘3 minutes’::interval create index on my_json ->> ‘a key’ It’s amazing how much stuff there is available. All the toys!
- CGamesPlay 4y ago
  Just a quick point in defense of SQLite: that last one is almost verbatim possible in SQLite, and it is possible to calculate ordinals, although the syntax is with standard SQL rather than a custom syntax. The SQLite docs mention that they never found a use case for jsonb that ended up being faster or more efficient than json, so they left it out, although they do reserve the BLOB data type for jsonb if such a use case is discovered.
  - gorgoiler 4y ago
    Well this is a doozy: so you’re saying they are both equally awesome as opposed to being individually awesome in different ways. What a time to be a developer.
canadiantim 4y ago
Can you use extensions with it like normal postgres?
- takeda 4y ago
  I'm guessing no, or at most just have set of extensions that you can use (kind of like it's done in AWS RDS or AWS Aurora). The claim is "serverless" i.e. you don't have access to the server and if you could install any extension you essentially have full access to the server as there's no restriction what you could do in an extension. I don't think that would be allowed.
- mpartel 4y ago
  https://neon.tech/docs/cloud/compatibility/ https://neon.tech/docs/cloud/compatibility/ says "During technical preview Neon has restrictions on user ability to install PostgreSQL extensions. Following PostgreSQL extensions come pre-installed: [..]"
- kelvich 4y ago
  Sure. So far, we just precompiled a few popular extensions, and they are available for installation. Ultimately we want to provide an option to bring your own extensions e.g., by specifying a docker image based on our base image. But that is some work on a security front: with a custom extension, you have access to the corresponding Unix user and can construct malicious WAL, send requests to the control plane, etc. Disclaimer: Neon co-founder
ushakov 4y ago
that's one of the best lookin landing pages i've seen this year! very well executed! congrats!
- nikita 4y ago
  Thank you. PixelPoint (https://pixelpoint.io/ https://pixelpoint.io/) designed the landing page. Copy stuff may change we want to make the story clearer and highlight more features.
Aeolun 4y ago
I need some kind of pricing for me to consider this. Did I miss the page describing that?
- nikita 4y ago
  We haven't launched pricing yet. In fact our official launch is in a few weeks. This HN post "leaked". We will launch, allow people to use it for free and introduce pricing later this year.
lewisl9029 4y ago
Seems like this might implement database branching in the way most people would assume: branching both the data and schema? I remember being a bit disappointed to learn that PlanetScale's database "branching" was only for the schema [1], which is still quite useful, but this would be so much cooler! I couldn't find much info about the replication models available/planned however. I would consider this to be table stakes at this point for a serverless database with the recent trend of pushing compute to the edge. This is much more interesting to me than scaling to 0, which is only really useful during the prototyping phase. PlanetScale is single primary with eventually consistent read replicas, Fauna has strongly consistent global writes (or regional if you choose, but no option for replication between regions if you do) with a write latency cost, Dynamo/Cosmos are active-active eventual consistently replicated with fast writes globally. All useful in different scenarios, but I'd love to have one DB tech that can operate in all of these modes for different use cases within the same app, using the same programming model to interact with data across the board. I think the decoupled storage engine here would open up some really interesting strategies around replication. What are the team's plans here? [1] https://docs.planetscale.com/concepts/branching https://docs.planetscale.com/concepts/branching
- nikita 4y ago
  Great questions! 1. Yes schema and data via "copy on write". This will let you instantly create test environments, backups, and run CI/CD. There is a long video here that shows a prototype with GitLab: https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s. 2. We don't have this feature at the launch, but Matthias van de Meent is already working on it. We will publish and RFC and solicit comments from the community. 3. We are working on two: regional read replicas and consistent multi-region writes (together with Dan Abadi who helped design FaunaDB). Former is much, MUCH easier. 4. An obvious one is a time machine - we want to allow you query at LSN (or timestamp). A less obvious one is templates: you can start your project with a pre-populated database. We will allow you to create and publish such "templates". Disclaimer - it might not be called templated when we ship it.
  - rattray 4y ago
zwily 4y ago
How exciting! What version of Postgres are they targeting? And do they have a strategy for keeping up to date with new versions?
- mattashii 4y ago
  Right now, we're based on PG14 (14.1 to be precise). We're looking into supporting PG15 when that comes out too, but I wouldn't hold my breath on that as we still have a lot to do.
- jadbox 4y ago
  Says 14 in the docs
- nikita 4y ago
  One idea we have is to allow running future unreleased and beta versions of Postgres. Do people want this?
IgorPartola 4y ago
I am trying to understand how it works without digging into the code. It sounds like the disk-backed storage here uses S3 which would introduce some severe latency as well as orders of magnitude more access errors (S3 is not going to be more reliable than EBS, let alone physical disk arrays on a day to day basis). Also how do they mitigate latency from their network to mine? In other words why would I run this over a local install if performance mattered at all to me?
- mattashii 4y ago
  How it works is: PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low commit latencies. After Safekeepers acknowledge the WAL, a PageServer will receive the WAL from these Safekeepers and transform it into LSM-tree "Layers" - blocks of lookup-optimized changelogs, which (when complete) are sent to S3. At that point, the data is considered fully persisted against most, if not all, outages. The PageServer (which serves as the long-term data server for the running compute nodes) maintains a local cache of Layers. Still, by design, that is only a cache -- it allows for fast responses but is not strictly necessary for the persistence model.
  - cpursley 4y ago
    Can we set our own logical replication? I'm listening to the WAL the same way Supabase does it: https://github.com/cpursley/walex#logical-replication https://github.com/cpursley/walex#logical-replication
    - mattashii 4y ago
      In our cloud offering logical replication is not yet supported. There's an open epic to support receiving a logical replication stream [0], but we've not yet checked on sending logical or physical replication streams. [0] https://github.com/neondatabase/neon/issues/1632 https://github.com/neondatabase/neon/issues/1632
      - 4y ago
ngcc_hk 4y ago
The site just invite to trial run. Any basic doc, tutorial etc. come from db2 …
- mattashii 4y ago
  The docs are available on https://neon.tech/docs/cloud/about/ https://neon.tech/docs/cloud/about/ . It contains some information, but we're looking to expand that while working towards a public beta that's arriving soon.
TruthWillHurt 4y ago
"We separated storage and compute" No. just no. Compute where the data lives, else you incur traffic cost and latency. A lesson learned in life, known from the dawn of time..
- wnolens 4y ago
  Easy trade-off to get scalability. Writing log off-host is fast with 10G network and nvme on the other side (see Amazon Aurora, Microsoft hyper scale, etc)
- manigandham 4y ago
  This isn't binary as much as it's a continuum of abstractions already. Cloud servers already use virtualized disks where the actual drives can be in an entirely different building from the compute racks. Do you consider that separate? Separating storage and compute at the database nodes is just one architecture method that allows for simpler and more efficient scaling in modern cloud-based deployments where you have large pools object storage that's fast and close. I think you'd be surprised by what you can achieve with some metadata and caching.
- userbinator 4y ago
  The goal of the "cloud" is to extract maximum $$$ from its users. Once you realise that, it's not hard to see through the marketing propaganda and the rest of the decisions following. else you incur traffic cost and latency Thus, they are separated so they can charge you (more) for them separately.
  - panarky 4y ago
    Zero bandwidth cost within the same zone.
- dunkeylim 4y ago
  What? This isn't controversial, Snowflake did this years ago and this is the main reason it became a $40B business. Amazon came out with Redshift, a cloud OLAP database, but it tied compute with data so teams couldn't scale compute and data separately and thus had to pay disproportional costs to their required workloads. Sure, there's good reasons to keep compute and data together. But there is obviously a massive market for technologies that keep them separate...
manigandham 4y ago
From the teams page, the CEO of Neon is the cofounder of MemSQL/Singlestore which is one of the best database products I've used. Looks like a solid team to get this done. Very similar approach to Yugabyte (real postgres compute layer + custom scale out data layer) and many others in the OLAP space.
- ignoramous 4y ago
  Manish Jain of dgraph.io noted that building on top of Postgres or betting on Postgres seems like a necessary condition for database startups to be successful. Some are commodotizing Postgres' wire format but implementing their own query and storage layers (like CockroachDB / Aurora / AlloyDB), while others are modifying parts of Postgres (like Timescale / EdgeDB / YugaByte), and others still are building atop it (Supabase). https://twitter.com/manishrjain/status/1496174276474732544 https://twitter.com/manishrjain/status/1496174276474732544
  - manigandham 4y ago
    Interesting note but that seems to be recency bias with news than anything concrete. Companies from MongoDB to Snowflake to FaunaDB have been successful. Manish himself is from DGraph which is a brand new graph database with no relational to Postgres.
- nikita 4y ago
  Thank you for the kind words Mani! Singlestore is indeed an amazing product and company. I'm really proud of it!
tomatowurst 4y ago
how does it differ from fly.io? https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/
- tptacek 4y ago
  Our Postgres is just a cluster of Postgres servers. It's not "serverless"; it's server-y. :)
tuukkah 4y ago
In case anyone else is wondering, it was called Zenith / ZenithDB before launch: https://github.com/neondatabase/zenith https://github.com/neondatabase/zenith
- nikita 4y ago
  Neon is a better name!
hestefisk 4y ago
What is “bottom-less” storage? This just seems like a managed hosting of PostgreSQL, nothing else…
- kelvich 4y ago
  We wrote a custom storage layer for postgres, so in our setup, postgres node (k8s pod actually) doesn't store any data, and it is easy to start/stop/reschedule it. So while we actually are DBaaS, the closest analogy here is Aurora or Alloy, not RDS-like setups. A bit more details are on https://github.com/neondatabase/neon https://github.com/neondatabase/neon
- excuses_ 4y ago
  Typically it means S3 and alike. Not being bounded by disk size of your server. Or in their case they might refer to horizontal scalability of their storage layer which is independent from computing.
  - nikita 4y ago
    Yes, you will never run out of storage with Neon. The reason to it that in our tiered storage implementation S3 is the cheapest tier and we will offload data into S3 if it gets too big.
infogulch 4y ago
How fast can a "scaled to zero" database start up? Does Neon use a "uninitialized hot spare" strategy to reduce startup latency like crdb? How much memory do they expect a typical single postgresql compute instance to take? I saw that Neon is targeting 'thousands' of postgresql processes per server, though with giant multi-TB servers these days that doesn't really narrow it down. Are the postgresql processes multi-tenant as well, or is multitenancy isolated to the storage layer? --- Heikki from the Neon team presented a talk about why they chose to develop Neon in Rust and what their experience was in Rust Finland 2022. https://www.youtube.com/watch?v=kAQeout-mh8 https://www.youtube.com/watch?v=kAQeout-mh8
- Shorn 4y ago
  https://youtu.be/kAQeout-mh8?t=700 https://youtu.be/kAQeout-mh8?t=700 Nobody knew Rust, so they started out by hiring someone who did. Good move. Business idea: consultancy that hires out competent Rust devs to new projects.
- nikita 4y ago
  Postgres process in single tenant. Right now both provisioning a new Postgres instance (we call it project) and cold start is 2 seconds. We will be improving on that.
nikita 4y ago
Nikita - CEO of Neon here. We intended to post this at the launch next month, but since it here, I'm happy to answer any questions. We have been hard at work and looking to open the service to the public soon.
- zeusly 4y ago
  Hey Nikita, could you maybe put some more legal information on the webpage? I'm trying to find out if you're a company and where you are located. Is there no legal entity behind this? Do you have a privacy policy?
  - mattashii 4y ago
    The company is Neon, Inc., which is registered in the USA. We're a remote company, with a significant portion of the developers being located in Europe. Privacy policy and related stuff will be ready when we publish the public beta, which we expect to happen soon.
    - nikita 4y ago
      Yep, Delaware corp with top tier US investors. I'm in the Bay Area. Heikki is in Finland. Stas is in Cyprus. Majority of engineering is in Europe, some in the US and Canada. Postgres is a global phenomenon.
- akmodi 4y ago
  Hey Nikita! I was just looking at the docs but I was a bit confused about what the various compute instances were doing. Do they all serve reads and writes? If so, is there data partitioning or does this support distributed transactions?
  - nikita 4y ago
    Various compute instances are different endpoints to separate databases. So for now it's single writer system. You can get a lot of power out of a 128 core compute node. In the future will will also spin up extra compute to scale reads. In the future after that future we will introduce data partitioning - we have a cool design for it, but one step at a time.
AndrewDucker 4y ago
Having a relational database where you're charged purely for the calls you make is a game-changer. All of the relational databases I looked at in the past required you to have a gateway node on at all times, which is far too expensive for a simple hobby project.
rubenfiszel 4y ago
This is exactly what I was looking for windmill, an OSS tool for multi-step automation from scripts. By any chance, are documentation to host it on top of nomad be on the Roadmap ? If yes, I would try it immediately to replace the postgres server we currently use.
- nikita 4y ago
  No documentation of how to self host yet. It's not hard, it requires K8S and S3 compatible object store. We want to harden Neon operationally before documenting and supporting on prem production installs.
  - ccakes 4y ago
    Requires k8s as in - uses k8s APIs? Or is supported on k8s? Apologies haven’t read the docs but wanted to highlight this. A hard requirement on k8s to the exclusion of other schedulers would be a shame and an odd choice
    - mattashii 4y ago
      We've built our current cloud offering on k8s, and that is our only scale-to-0 implementation. That does not mean that you cannot run Neon outside k8s, but we are not actively maintaining nor supporting other hosting options.
  - rubenfiszel 4y ago
    Got it, makes sense. I think we are pursuing similar strategies with OSS/self-hostable products. I would love to get in touch with you about it. Could you drop me an email at ruben@windmill.dev ?
    - nikita 4y ago
      Replied
sudhirj 4y ago
Does this storage layer still require VACUUMing? The Aurora system that AWS has uses a log based storage system, so along with point in time restore there’s no longer a need to go free up old garbage rows. This system says it’s copy on write, wondering if vacuuming is still necessary.
- petergeoghegan 4y ago
  The difficulty comes from the fact that VACUUM requires most of the same state that is required when processing a user transaction. For example, VACUUM needs the commit log to check whether a given transaction ID is for a transaction that committed or aborted (VACUUM also truncates the commit log to remove information about older XIDs when it is no longer required). Aurora Postgres does still have VACUUM, which seems to work in the same way -- which is the same in Neon. AWS have in the past promoted Aurora as having significantly better performance characteristics when VACUUM is run. That may well be true, but the benefits seem to come from not having to generate full page writes in WAL, which are a way of preventing a low-level problem called torn pages. Theoretically you could just turn off full-page writes in standard Postgres if you had hardware that offered atomic page writes, though I don't think that it's a widespread practice. I spend most of my time working on problems with VACUUM in Postgres itself (I'm one of the committers). An approach to organizing storage within transactional constraints seems necessary to push vacuuming down to storage, and that would more than likely need plenty of work in Postgres itself to be practical -- since it would cross a few layers of abstraction. Currently heapam doesn't specifically try to keep tuples inserted around the same time together, so it's hard to make anything that VACUUM does work implicitly or logically. Disclaimer: I work for Neon
  - sudhirj 4y ago
    Thanks for the explanation, I’m nowhere near your experience level with this, but from what I understand Aurora is an append only log with rolling snapshots (this might be wrong). In an append only log storage system there would be no concept of page writes, right? The VACUUM process might still be necessary to find and mark dead tuples, but this seems very minor compared to filesystem operations to reclaim (sometimes gigabytes) of storage. The reclamation of storage would simply left to the background snapshotting process and would not affect DB performance in any way.
iknownothow 4y ago
I knew my bet to sticking with Postgres would pay off! This looks super exciting. I thought of doing something similar for our data warehouse with AWS Fargate and Postgres but the cold starts and limited disk space required too much engineering on top to make it work. Moving to Snowflake comes at the cost of losing so many Posgtres features in exchange for speed. Things like foreign keys, constraints, extensions etc which requires so much engineering to replace in Snowflake. I would be happy to pay 25x the price for a 10x speed increase for a specific query.
- thejosh 4y ago
  Snowflake is a data warehouse though. Completely different use case. If your data can be done via PG, highly recommend that over SF. Especially with this concept. Snowflake is great when you use a tool like dbt, their modern SQL approach and functions are fantastic. Downsides is it's pretty pricey, and can catch you out.
  - iknownothow 4y ago
    We already use DBT and the data is less than 10TB, something Postgres can handle well. And most of the data is concentrated in a few tables. With a serverless approach I'd be happy to allocate 10x resources for just a query or two and for the rest a minimal server is fine. I manage the data warehouse mostly alone because Postgres offers guarantees, unique constraints, triggers and relationships between columns of different tables. It does the work of two engineers. Snowflake is fast but not Postgres compatible. In order to move to Snowflake, I have to write tests and maintain them which Postgres does for me for free. I'd stick with Postgres at least until 20TB before considering Snowflake.
- manigandham 4y ago
  Snowflake is an OLAP system. It's an entirely different kind of "speed" designed for analyzing vast amounts of data through scans and aggregations.
  - iknownothow 4y ago

667 points by nikolay 4y ago

Why not use normal networked storage (nbd/drbd, AWS EBS, etc.) instead of the ad-hoc Pageserver + Safekeeper architecture? Or even better, use simple local SSD/HDD storage where data is small enough.

mattashii 4y ago
Because all of those don't or can't have both low-latency reads and regional persistence guarantees, which is what we're after.

XorNot 4y ago

Well this seems like a big deal. "Needs a database" has been an ongoing pain point for working with serverless apps. Staying in the postgres command library, if it works, means you can now just prototype and assume "postgres will be there". I have a couple of apps I would've used this exact service for in the recent past. Looking forward to trying it in the future.

riedel 4y ago

This sounds really interesting! I wonder what kind of scaling use cases neon is good for. Is it e.g. good for custom scenarios like a geospatial timeseries database on top of postgres? We have admittedly not really a clue about current database cluster tech as we are IoT/ML researchers, but we are running a custom timescaledb cluster that receives constant nonchunked write load from a lot of devices and may encounter some long running queries on an around 500GB DB filled with geolo (even timing out if users are too creative), why we splitted into a single ingress master and multiple outgres WAL readonly replicated query clients to relax the consistency and sync, that seemed to be killing us (we need postgres because of postgis and have no capacity to rewrite the front-end). I wonder if neon would be good for such a use case and if it easily supports postgres extension like timescaledb hypertables and postgis). Most of the time our system just measurements, but sometimes we really need to scale up for PoCs, which makes dimensioning really hard (for us).

mattashii 4y ago
Neon has the ability to make a Copy-on-write replica of a database, which for you would allow you to create an instant read-only copy of the data that has been ingested up to that point, without significant storage overhead. The new data would still be written to the primary database, and long running queries would only see the snapshot that the instance was started with (using their own pool of CPU and memory resources). Assuming that the extensions that you use are compatible (that is, they don't access database files in a way that PostgreSQL doesn't, and the licence is compatible) then Neon could be a good solution to that issue.

pgt 4y ago

Incredibly good sales copy by someone who understands their target market.

ranguna 4y ago

Just yesterday I was comparing managed serverless postgres offers and was sad to temporarily end my investigation with a compromise of using managed aws RDS for development, hoping that a fully serverless postgres with a nice free tier would pop up before going to production, and here we are! Congrats to the team for what feels like an amazing product. Signed up for the early access, can't wait to get my hands on this! For anyone interested, these ere the DB offers I looked into: * DO managed postgres, no free tier but price scaling was not too aggressive, the issue is that it's not natively serverless and we're gonna get 100s of ephemeral connections. * Cockroach, was the best option for our use case but it doesn't support triggers and stored procedures, so we can't use it right now (closely following https://github.com/cockroachdb/cockroach/issues/28296 https://github.com/cockroachdb/cockroach/issues/28296) * Fly.io price scaling is too aggressive 6$ -> 33 -> 154 -> 1000s a month and no free tier that I could find. * Aurora serverless v2 is only for aws internal access and we are using gcp. * Aurora v1 was what we were gonna go with, but a lot of people online have showed their negative opinion around slow scaling. I didn't investigate enough but I'm thinking we'd need to setup RDS proxy for it handle all our connections, which would've bumped up the price by a good amount. Also no free tier. * Alloydb looked promising but also no free tier and starting price is a bit much for our current phase of development, but it was definitely something we'd look into in the future. And now Neon, natively serverless with a (hopefully) good free tier to test things out and some hints about cross region data replication, amazing stuff!

lysecret 4y ago
"Aurora serverless v2 is only for aws internal access and we are using gcp." You can have public access to serverless v2. I'm using it with retool for example. That said I moved a Postgres DB to Aurora, the process was hilarious in how crazy it was. Also they haven't implemented scaling to 0 yet!!!! And the minimum 0.5 Compute unites are actually pretty expensive.
- ranguna 4y ago
  Nice point about the minimum 0.5 ACU, forgot about that one. From what I've read 0.5 on v2 is the same price as 1 on v1, which seems pretty dumb coming from aws. Could you elaborate more on this: > You can have public access to serverless v2 Because the docs mention the following: > You can’t give an Aurora Serverless DB cluster a public IP address; you can only access it from within a VPC based on the Amazon VPC service. Potentially I could setup an RDS proxy or vpn inside the vpc and give that public access, but that seems a bit of a roundabout way of handling this. https://aws.amazon.com/blogs/database/best-practices-for-working-with-amazon-aurora-serverless/ https://aws.amazon.com/blogs/database/best-practices-for-wor...

kosolam 4y ago

Cool! Can be used in production already?

mattashii 4y ago
We cannot yet recommend users to run their critical workloads on Neon.
kelvich 4y ago
We are planning to open public tech preview next month. We'll go to general availability when we see a decent SLO for several months. Tech preview should be okay for hobby projects and staging workloads, though.

rad_gruchalski 4y ago

What’s the licensing story? Can I run this in my org and expose this as a service to users outside of my org? Or does it have strings attached, like CockroachDB? Asking because github says Apache 2 but the devil is in details.

kelvich 4y ago
You can, no strings attached
- rad_gruchalski 4y ago
  Awesome, will take it for a spin.
mattashii 4y ago
Yep, for now the storage engine and the modifications on PostgreSQL are licensed under APLv2. Our cloud offering which orchestrates the scale-to-0 and other glue between the components are closed-source and not covered by that license. Do note however that the licensing story is not entirely fleshed out yet, as the product we're building is still in closed beta. As we work towards a paid cloud offering, we'll further flesh out the license model for the code, but for now we're planning on keeping this license.

smallerfish 4y ago

Could you document some of your differentiation against aurora, both on price and architecture? I don't care about scale to 0. I care about more scale to NNN, efficiently & reliably, with minimal devops needs.

nikita 4y ago
We will put more of the Aurora differentiation docs up in a bit. Architecture wise we are slightly different b/c of the S3 integration and splitting storage to safekeepers and page servers. This is very technical and we will be highlighting both this and user visible features, such as branching.

talkingtab 4y ago

Why is this a good idea? In my experience, getting Postgres up and running is trivial. Docker anyone? And in many cases your data is your business so why hand it off? And if you are going to offer this product why not just call it what it is, "Postgres as service", instead of serverless which seems a bit misleading. Really it is simply Postgres running on your server.

chimen 4y ago
Not everyone can manage a database properly and, sorry to say this to you but, Docker is a terrible idea for a database in general. Setting up your own database somewhere still puts your trivial data on someone else's server more or less. All these "Serverless" keywords pretty much mean you don't have to be spinning up servers (cloud) or setting up & maintaining one. Nothing is "Serverless" per-se so it's time to move on from picking on this, I agree, bad choice of words.
- smokey_circles 4y ago
  > Docker is a terrible idea for a database in general Why? Genuine question, my gut feel is there's something wrong about it too but I can't put words to it nor have I found a benchmark that convinced me, but it's worth noting I'm not sure what I'm looking for
  - chimen 4y ago
    I manage about 200 servers and docker crashing accounts for 20%+ of my issues so far. The servers are brought up easily on crash and that's not an issue for my services. For a database docker is nothing more than an extra layer of complications on top with iptables, volume system and all the layers it brings. It's just a bad wrapper for a production database which needs stability.
    - Art9681 4y ago
      To be fair, that sounds like a problem with your infra. There are plenty of enterprises running Postgres in Docker/Kubernetes successfully. A lot of technical problems are not the stack or the engineer's capability, but the company culture and lack of funding, time or resources. If you had more of all 3, im sure your problem could be "solved" to an acceptable margin of error. Likely by you.

captnObvious 4y ago

I hope y’all have a plan for when AWS decides to pick up your open source project and turn it into a managed cloud solution. It’s a pattern of theirs. And with the way egress charges are structured they’re likely to snap up any clients straddling their cloud and yours.

mattashii 4y ago
AWS already has Aurora, which is their own in-house closed-source variant that does very similar things. We think we'll be able to provide a better experience at lower cost for smaller developers, while having some very useful quality-of-life features like zero-cost branching and instant PITR.
onphonenow 4y ago
AWS has Aurora Serverless v2 out already for postgresql along with RDS for postgresql. Not scale to zero though / bottom is 40/month or so
oxfordmale 4y ago
AWS already has this, Google for Aurora Serverless. It is not cheap though, and this might well be cheaper.

SonOfLilit 4y ago

> Neon allows to instantly branch your Postgres database to support a modern development workflow. You can create a branch for your test environments for every code deployment in your CI/CD pipeline. > Branches are virtually free and implemented using the "copy on write" technique. Unless I missed that everyone supports this, this here could be a killer feature and should be advertised higher.

jhgb 4y ago
It sounds like something you might be able to accomplish with a copy-on-write VFS on top of a Firebird database file. (Not sure about PostgreSQL, but with Firebird, you only deal with one file, so with Firebird, this should definitely work.)
- nikita 4y ago
  There is an enterprise company called Delphix that does it on top of Zfs - so the idea was in the air. Instead of duct taping this together with a filesystem we purpose built database storage. The advantage to this is that we can much tighter control execution paths and can profile them end-to-end. Additionally this allows us to integrate with S3 and make it much much cheaper to run.
  - jhgb 4y ago
    Technically Firebird just requires a block device, so you might not even need a filesystem.
zxspectrum1982 4y ago
You can get that feature on any Postgres server by installing Citus
- mattashii 4y ago
  Does Citus provide any such storage-level multi-cluster features? I can't seem to find any documentation on that...

funstuff007 4y ago

How do I keep my application servers physically close to Neon?

mattashii 4y ago
Right now, we have our hosting in AWS (us-west-2). We're planning on expanding to other regions and cloud providers eventually, though.
- GordonS 4y ago
  Having data in Europe will be important for many users, hope you expand to EU soonish.

Sytten 4y ago

I am glad someone is taking on this challenge! Its been a few years since I was saying that the last piece of the serverless puzzle is a good postgres serverless database. If I was not working on my startup I would apply for sure. It would be nice to present the project on the CMU database group youtube channel at some point to dive into the implementation.

nikita 4y ago
I'll work with Andy Pavlo who is a friend to set up a presentation. And yes, we are hiring! So if and when you are ready let us know.

jasonhansel 4y ago

Does this handle sharding? Or is it like ordinary Postgres, which only performs replication?

mattashii 4y ago
It is ordinary PostgreSQL, in which only the storage interface with the file system is replaced with our own storage interface.

edude03 4y ago

This is really amazing, super excited to try it out. I read the docs and I noticed you can run it locally, but has the kubernetes bits been made available? I see https://github.com/neondatabase/helm-charts https://github.com/neondatabase/helm-charts and https://github.com/neondatabase/neon/tree/main/.circleci/helm-values https://github.com/neondatabase/neon/tree/main/.circleci/hel... but I think there is some charts missing?

mattashii 4y ago
Correct: we do not yet use k8s for provisioning the Safekeepers and PageServers for our closed-beta cloud offering, and the PostgreSQL instances are managed in k8s by our closed-source console. As such, there's little we can open-source at the orchestration level at this point in time.
- chwzr 4y ago
  Is there a timeline on releasing orchestration things?

madelyn 4y ago

This is really interesting. Are there restrictions or limitations on the PageServer + Safekeeper design when running OLAP queries on larger datasets? Phrased another way, would a query that needs to access a relatively large amount of data (10-100 GB) ever need to read from s3, incurring extra latency?

mattashii 4y ago
In general, all data of live clusters will also be stored locally at a PageServer. Only in recovery scenarios will a PageServer not hold the data that is needed to serve the requests of a compute node - but that would recover quickly as the local cache of the PageServer is repopulated with data from S3.
- madelyn 4y ago
  Awesome, thank you!

cja 4y ago

In what way is this serverless? Surely it's running on a server?

ttfkam 4y ago
Refers to management and autoscaling, not whether physical hardware exists somewhere. From your point of view it has DNS entries, you've defined how high/low it's allowed to scale and some other configs, you've defined network access rules, and allows client connections. That's it. You don't manage the server instances. Hence: "serverless"
- cja 4y ago
  Thank you for explaining. There must be a better term than serverless though. "Auto-scaled cloud database" would be more accurate, for example

TobyTheDog123 4y ago

I'm a bit concerned that the free trial mentions "compute up to 1 vCPU / 256 MB" Why would I need to worry about this for a serverless database provider?

nikita 4y ago
We are limiting the amount of compute we are giving for free during the technical preview. We will open it up to much large computes later. Like those will be in the paid tier.
- TobyTheDog123 4y ago
  That sounds good, but my question was (and I probably phrased it poorly), as a solo developer, why would I need to concern myself with resources associated with servers (compute, CPUs, MBs of RAM) on a serverless platform? I could maybe see it in the sense if it's only applicable when Postgres is executing complex custom queries or whatever, but for any operation? That's a huge red flag, for me at least. When I think of serverless, I think of things like Cloudflare Workers, FaunaDB, Ably.io, all of which have pricing based on the usage of their features, not the consumption of their resources. The whole point being that I can way more easily calculate how many messages my users are sending, than how much CPU all those messages are taking to send. Maybe I'm operating on the wrong definition of serverless, and all of the other features look amazing, but that concept is really a dealbreaker.

_xnmw 4y ago

Are they going to stay up-to-date with the latest version of Postgres? One problem with Yugabyte, TimescaleDB, Aurora etc. is that they are stuck on older versions of Postgres, which makes it feel like an entirely different product after a few years.

mattashii 4y ago
We are planning on supporting the latest stable version of PostgreSQL. Right now, we're a bit behind (we're at 14.1, latest PosgreSQL is 14.3) but that shouldn't be much of an issue. We don't yet know how we're going to do major version migrations, as the product is still not even out of private beta.
- rattray 4y ago
  "how will this handle major version upgrades" was one of my first/biggest questions when reading through the homepage fwiw
  - nikita 4y ago
    Major versions don't come out suddenly. We have time to test and make sure they work well with our storage. Customers don't always want to upgrade to new version when old versions just work. However this can lead to version creep. Ideally we want to always run the latest version of Postgres. We will hold this line as long as possible.
    - rattray 4y ago
      Sorry, I meant less "how long until they catch up to trunk" and more "what is the downtime / operational burden of performing major upgrades like". With separate storage and compute it seems tantalizingly possible to have low-effort zero downtime upgrades for the user, which would be a huge selling point.
akulkarni 4y ago
"One of these things is not like the other" TimescaleDB, because it is packaged as a PostgreSQL extension (and not a fork, unlike the others), stays compatible with mainline PostgreSQL, especially as PostgreSQL improves. This is one of the key advantages of our approach. (Timescale co-founder)

nojvek 4y ago

Nikita, I am a huge fan of SingleStore - amazing what you’ve accomplished there and now what you’re building at Neon. Do you plans to build a columnstore index on top of postgres that supports insert/update/delete? Love how MSSQL has a columnstore index for a subset of columns on a row store table. Always wondered why nobody has built something like that for postgres yet. Citus is nice but it’s append only, which is a huge restriction.

nikita 4y ago
We are not building it in house right now - we are focusing on OLTP. But have plenty ideas of how to do it in the future. In the first iteration we are thinking about an integration with Singlestore, Snowflake, and BigQuery. Singlestore will be best due to low latency updates, but the other two are very popular so integration with Snowflake and Big Query just makes sense. In the future we can integrate a columnstore right into the engine to make a smooth on system experience. There are some awesome open source implementations: arrow and duckdb. Updatability is tricky but doable as proven by Singlestore and SQL Server (I'm ex SQL Server and a huge fan of this feature). Not this year.
- nojvek 4y ago
  Makes sense. I agree. DuckDB is pretty phenomenal. I enjoyed reading its source code and playing around with it. Another open source nice C++ codebase is Typesense for Hyperfast text search (algolia competitor). It’s been on my mind for many months how to build indexing like this as postgres extensions. I love how versatile postgres is with so many indexing datastructures.
  - nikita 4y ago
    Again it is a big commitment to do a good job on implementing columnstores. If you don't do it all the way it not very usable and you are confusing your users by giving them too many options. The performance expectation is now set by great columstore implementations and you just can't afford a half ass job here. Here is an example from Google AlloyDB: https://twitter.com/mim_djo/status/1527900193626025984 https://twitter.com/mim_djo/status/1527900193626025984. My understanding that DuckDB is even faster on the TPCH benchmark. TPCDS is much harder and I doubt AllowDB can even run it at any reasonable scale.

rattray 4y ago

FYI, the image at the bottom says "Zenith's prices" instead of "Neon's prices" https://neon.tech/static/saas-illustration-lg-410ada378df755f415b2ab6c15364b17.svg https://neon.tech/static/saas-illustration-lg-410ada378df755...

nikita 4y ago
Fixed. Thank you for catching in. Renaming a company is hard even pre launch :)
- rattray 4y ago
  I know the pain :) Good luck, this is a very exciting project. I'm extremely curious to see how it unfolds…

Komodai 4y ago

Typo in footer 'Maid in SF and the World' should be 'Made in SF and the World'

nikita 4y ago
Fixed - sorry, sleepless night.

rattray 4y ago

This sounds awesome, but one of my first reactions to the notion of separated compute/storage Postgres with copy-on-write is "okay, so… slow Postgres?" Is there anything the dev team can share on read/write performance compared to RDS, NVMe EC2 instances, EBS-backed EC2, etc? In what situations would this setup perform poorly, and in what situations would it excel?

cultofmetatron 4y ago

don't know too much about this yet but i've been desperately wanting there to exist a postgres equivalent to planetscale. Bravo guys! I'll keep you guys in mind for my next mvp. PS: btw why call it Neon? thats already the name of the rust/nodejs interop library which I assume you know about since your storage layer is written in rust. PS: imagine a colab between this and fly.io. I wish thsi stuff was available when I was starting Blinq

nikita 4y ago
We already have a shared slack channel with Kurt - CEO of fly and a number of folks from Fly.io. We are discussing our partnership.
- cultofmetatron 4y ago
  thats awesome. what will the pricing be when its done? my startup currently spends around 400-500/month on aurora. would be curious what the differences would be here in terms of TOC. the ability to branch databasees alone is a pretty compelling killer feature imho

rapind 4y ago

I wonder how well this will work with PostgREST. Looks very interesting.

nikita 4y ago
We will be launching REST api with PostgREST later this year.

rektide 4y ago

Really interesting. I've seen so much disagregated database work, and so so so much of that exposes postgres interfaces. But all the good stuff has been closed source! I'm very very excited to hear about a team taking this effort to postgres itself, in an open source fashion! From the Architecture[1] section of the README: > A Neon installation consists of compute nodes and Neon storage engine. > Compute nodes are stateless PostgreSQL nodes, backed by Neon storage engine. > *Neon storage engine consists of two major components: A) Pageserver. Scalable storage backend for compute nodes. B) WAL service. The service that receives WAL from compute node and ensures that it is stored durably. Sounds like a very reasonable disaggregation strategy. Really hope to hear about this wonderful effort for many more years. Ticks the boxes: open-source with a great service offering: nice. Rust: nice.

rattray 4y ago
[1] https://github.com/neondatabase/neon#architecture-overview https://github.com/neondatabase/neon#architecture-overview
- rektide 4y ago
  Oops thanks!
nikita 4y ago
We are committed to building a durable company and we are well funded. So yes, you will hear from us for years to come as we will be shipping more and more features.
- avinassh 4y ago
  I could not find funding information on the Neon site. Is that information not public? edit: I found the info here: https://boards.greenhouse.io/neondatabase/jobs/4506003004 https://boards.greenhouse.io/neondatabase/jobs/4506003004
  - nikita 4y ago

thomasfromcdnjs 4y ago

Anyone got a spare invite code? =D

felixmeziere 4y ago
Same here please!
- oknoorap 4y ago
  same here ^_^
  - nikita 4y ago
    We got a lot of "join waitlist" submission. Thank you HackerNews for this! Please email beta@neon.tech for the invite code. We will ask for feedback in return.

peterkelly 4y ago

Please stop this abuse of language. Here's serverless sqlite: https://www.sqlite.org/serverless.html https://www.sqlite.org/serverless.html

dandanua 4y ago
Modern marketing makes me mad sometimes. They call something "serverless" while in fact it runs on their servers somewhere in the world, where you have even less control than on your cloud. Though, I believe it was popularized by Amazon, which doesn't care much about being honest in its expansionism.
jjeaff 4y ago
From your link: Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity. Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers. It sounds like this is neo-serverless but serverless nonetheless.
- eyelidlessness 4y ago
  It “is” in the generous sense that the quoted section interprets generously for the purpose of interpreting it generously.
infogulch 4y ago
This ship is so sailed already that the stories of its voyages are legendary classics that were told to our great grandparents.
- CharlesW 4y ago
  > This ship is so sailed already that the stories of its voyages are legendary classics that were told to our great grandparents. Specifically, the term is 10 years old this October: http://readwrite.com/2012/10/15/why-the-future-of-software-and-apps-is-serverless/ http://readwrite.com/2012/10/15/why-the-future-of-software-a... It has never, ever meant that the software wasn't running on a server. From TFA: "The phrase 'serverless' doesn’t mean servers are no longer involved. It simply means that developers no longer have to think that much about them. Computing resources get used as services without having to manage around physical capacities or limits."

coder543 4y ago

> Branches are virtually free and implemented using the "copy of write" technique. Copy on write, presumably.

kelvich 4y ago
Fixed, thank you!

deleted 4y ago

[deleted]

anilgulecha 4y ago

This is the missing piece on cloud for masses * we already have compute scale-to-zero (cloudrun, lambda, fly.io). * Network is default pay for use. Storage (S3) is default pay for use. * The only piece in the stack that was always-on was the database (only serverless db thus far was firestore, or something like sqlite+litestream) With something like this we get a solid RDBMS engineered to be scale-to-zero, and with good developer experience. This opens up a world of try-out mini applications that cost cents to host. serverless db (postgres) + serverless compute (cloud-run) + use as you go storage+network. This is a paradigm-shift stack. Exciting days ahead.

pid-1 4y ago
Fly.io does not scale to zero. Lambda has many limitations. In particular, for some reason AWS is allergic to providing a container deployment service that actually scales to zero.
- russellendicott 4y ago
  > AWS is allergic to providing a container deployment service that actually scales to zero Isn't this what Fargate is?
  - pid-1 4y ago
    No.
- hamandcheese 4y ago
  Not yet, but soon.
jokethrowaway 4y ago
I don't see the benefits over a 5$ VPS. Even admitting I'll save a few cents over a VPS (which is absolutely not guaranteed), the cost saving is so minimal I won't bother rewriting everything under the serverless paradigm just for it. Of course if your cloud doesn't cost an eye and an arm. I can understand people excited to save on expensive Aws instances but maybe you should just consider dumping Aws. Scale doesn't matter for mini applications and scaling vertically (=throw money for a bigger server) will work for 99% of the companies. The 1% who need horizontal scaling will have custom everything regardless and will need to hire experts, not a good niche to release a product.

vira28 4y ago

Amazing work by the Team. Congrats y'all. It was one of the best presentations in the PGcon22. I did email Heikki the following questions, in case if someone from Neon is around here. a) How does Neon compare to polardb https://github.com/ApsaraDB/PolarDB-for-PostgreSQL https://github.com/ApsaraDB/PolarDB-for-PostgreSQL. b) The readme mentions a component "Repository - Neon storage implementation". Does it use any special FileSystem? Any links to read more about it? c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?

ololobus 4y ago
> c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s? Yeah, as Nikita mentioned it's 2 seconds now. We did some tests and measurements and on bare metal, it's sub 500 ms usually, so the remaining part is the k8s (+ our own control plane) orchestration overhead. For example, with plain Docker (which we use in CI in addition to k8s) it's around 1 second already. K8s provides a convenient abstraction layer, though. So I think that we'll continue using it and optimization will come with pods pool / over-provisioning and it'll be realistic to bring the startup time closer to bare-metal. -- Cloud engineer @ Neon
nikita 4y ago
Thank you! a. PolarDB is based on a similar idea. https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-sigmod21.pdf https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-s.... This paper describes it. The biggest difference that I see glancing through the paper is that we really integrated S3 into the storage. In Neon architecture branches, backups, checkpoint are all the same thing and instant to run. This simplifies a good amount of database management AND deliver on better costs. S3 is cheap. b. Neon doesn't need a special filesystem. Neon storage is in a way a filesystem, however it doesn't expose filesystem API. It's a key value store - serves 8k pages to Postgres and a consensus - update API to the key value store. Pages are organized in LSM trees and background processes put layers of the LSM trees to S3. c. The cols start is 2sec right now. There is a dependency on K8S. Bare metal implementation will require new code to orchestrate starts and stops.

gorgoiler 4y ago

Postgres is mind boggling, coming from sqlite. In a good way, and both are amazing tools. with ordinal jsonb_* ‘3 minutes’::interval create index on my_json ->> ‘a key’ It’s amazing how much stuff there is available. All the toys!

CGamesPlay 4y ago
Just a quick point in defense of SQLite: that last one is almost verbatim possible in SQLite, and it is possible to calculate ordinals, although the syntax is with standard SQL rather than a custom syntax. The SQLite docs mention that they never found a use case for jsonb that ended up being faster or more efficient than json, so they left it out, although they do reserve the BLOB data type for jsonb if such a use case is discovered.
- gorgoiler 4y ago
  Well this is a doozy: so you’re saying they are both equally awesome as opposed to being individually awesome in different ways. What a time to be a developer.

canadiantim 4y ago

Can you use extensions with it like normal postgres?

takeda 4y ago
I'm guessing no, or at most just have set of extensions that you can use (kind of like it's done in AWS RDS or AWS Aurora). The claim is "serverless" i.e. you don't have access to the server and if you could install any extension you essentially have full access to the server as there's no restriction what you could do in an extension. I don't think that would be allowed.
mpartel 4y ago
https://neon.tech/docs/cloud/compatibility/ https://neon.tech/docs/cloud/compatibility/ says "During technical preview Neon has restrictions on user ability to install PostgreSQL extensions. Following PostgreSQL extensions come pre-installed: [..]"
kelvich 4y ago
Sure. So far, we just precompiled a few popular extensions, and they are available for installation. Ultimately we want to provide an option to bring your own extensions e.g., by specifying a docker image based on our base image. But that is some work on a security front: with a custom extension, you have access to the corresponding Unix user and can construct malicious WAL, send requests to the control plane, etc. Disclaimer: Neon co-founder

ushakov 4y ago

that's one of the best lookin landing pages i've seen this year! very well executed! congrats!

nikita 4y ago
Thank you. PixelPoint (https://pixelpoint.io/ https://pixelpoint.io/) designed the landing page. Copy stuff may change we want to make the story clearer and highlight more features.

Aeolun 4y ago

I need some kind of pricing for me to consider this. Did I miss the page describing that?

nikita 4y ago
We haven't launched pricing yet. In fact our official launch is in a few weeks. This HN post "leaked". We will launch, allow people to use it for free and introduce pricing later this year.

lewisl9029 4y ago

Seems like this might implement database branching in the way most people would assume: branching both the data and schema? I remember being a bit disappointed to learn that PlanetScale's database "branching" was only for the schema [1], which is still quite useful, but this would be so much cooler! I couldn't find much info about the replication models available/planned however. I would consider this to be table stakes at this point for a serverless database with the recent trend of pushing compute to the edge. This is much more interesting to me than scaling to 0, which is only really useful during the prototyping phase. PlanetScale is single primary with eventually consistent read replicas, Fauna has strongly consistent global writes (or regional if you choose, but no option for replication between regions if you do) with a write latency cost, Dynamo/Cosmos are active-active eventual consistently replicated with fast writes globally. All useful in different scenarios, but I'd love to have one DB tech that can operate in all of these modes for different use cases within the same app, using the same programming model to interact with data across the board. I think the decoupled storage engine here would open up some really interesting strategies around replication. What are the team's plans here? [1] https://docs.planetscale.com/concepts/branching https://docs.planetscale.com/concepts/branching

nikita 4y ago
Great questions! 1. Yes schema and data via "copy on write". This will let you instantly create test environments, backups, and run CI/CD. There is a long video here that shows a prototype with GitLab: https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s. 2. We don't have this feature at the launch, but Matthias van de Meent is already working on it. We will publish and RFC and solicit comments from the community. 3. We are working on two: regional read replicas and consistent multi-region writes (together with Dan Abadi who helped design FaunaDB). Former is much, MUCH easier. 4. An obvious one is a time machine - we want to allow you query at LSN (or timestamp). A less obvious one is templates: you can start your project with a pre-populated database. We will allow you to create and publish such "templates". Disclaimer - it might not be called templated when we ship it.
- rattray 4y ago

zwily 4y ago

How exciting! What version of Postgres are they targeting? And do they have a strategy for keeping up to date with new versions?

mattashii 4y ago
Right now, we're based on PG14 (14.1 to be precise). We're looking into supporting PG15 when that comes out too, but I wouldn't hold my breath on that as we still have a lot to do.
jadbox 4y ago
Says 14 in the docs
nikita 4y ago
One idea we have is to allow running future unreleased and beta versions of Postgres. Do people want this?

IgorPartola 4y ago

I am trying to understand how it works without digging into the code. It sounds like the disk-backed storage here uses S3 which would introduce some severe latency as well as orders of magnitude more access errors (S3 is not going to be more reliable than EBS, let alone physical disk arrays on a day to day basis). Also how do they mitigate latency from their network to mine? In other words why would I run this over a local install if performance mattered at all to me?

mattashii 4y ago
How it works is: PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low commit latencies. After Safekeepers acknowledge the WAL, a PageServer will receive the WAL from these Safekeepers and transform it into LSM-tree "Layers" - blocks of lookup-optimized changelogs, which (when complete) are sent to S3. At that point, the data is considered fully persisted against most, if not all, outages. The PageServer (which serves as the long-term data server for the running compute nodes) maintains a local cache of Layers. Still, by design, that is only a cache -- it allows for fast responses but is not strictly necessary for the persistence model.
- cpursley 4y ago
  Can we set our own logical replication? I'm listening to the WAL the same way Supabase does it: https://github.com/cpursley/walex#logical-replication https://github.com/cpursley/walex#logical-replication
  - mattashii 4y ago
    In our cloud offering logical replication is not yet supported. There's an open epic to support receiving a logical replication stream [0], but we've not yet checked on sending logical or physical replication streams. [0] https://github.com/neondatabase/neon/issues/1632 https://github.com/neondatabase/neon/issues/1632
    - 4y ago

ngcc_hk 4y ago

The site just invite to trial run. Any basic doc, tutorial etc. come from db2 …

mattashii 4y ago
The docs are available on https://neon.tech/docs/cloud/about/ https://neon.tech/docs/cloud/about/ . It contains some information, but we're looking to expand that while working towards a public beta that's arriving soon.

TruthWillHurt 4y ago

"We separated storage and compute" No. just no. Compute where the data lives, else you incur traffic cost and latency. A lesson learned in life, known from the dawn of time..

wnolens 4y ago
Easy trade-off to get scalability. Writing log off-host is fast with 10G network and nvme on the other side (see Amazon Aurora, Microsoft hyper scale, etc)
manigandham 4y ago
This isn't binary as much as it's a continuum of abstractions already. Cloud servers already use virtualized disks where the actual drives can be in an entirely different building from the compute racks. Do you consider that separate? Separating storage and compute at the database nodes is just one architecture method that allows for simpler and more efficient scaling in modern cloud-based deployments where you have large pools object storage that's fast and close. I think you'd be surprised by what you can achieve with some metadata and caching.
userbinator 4y ago
The goal of the "cloud" is to extract maximum $$$ from its users. Once you realise that, it's not hard to see through the marketing propaganda and the rest of the decisions following. else you incur traffic cost and latency Thus, they are separated so they can charge you (more) for them separately.
- panarky 4y ago
  Zero bandwidth cost within the same zone.
dunkeylim 4y ago
What? This isn't controversial, Snowflake did this years ago and this is the main reason it became a $40B business. Amazon came out with Redshift, a cloud OLAP database, but it tied compute with data so teams couldn't scale compute and data separately and thus had to pay disproportional costs to their required workloads. Sure, there's good reasons to keep compute and data together. But there is obviously a massive market for technologies that keep them separate...

manigandham 4y ago

From the teams page, the CEO of Neon is the cofounder of MemSQL/Singlestore which is one of the best database products I've used. Looks like a solid team to get this done. Very similar approach to Yugabyte (real postgres compute layer + custom scale out data layer) and many others in the OLAP space.

ignoramous 4y ago
Manish Jain of dgraph.io noted that building on top of Postgres or betting on Postgres seems like a necessary condition for database startups to be successful. Some are commodotizing Postgres' wire format but implementing their own query and storage layers (like CockroachDB / Aurora / AlloyDB), while others are modifying parts of Postgres (like Timescale / EdgeDB / YugaByte), and others still are building atop it (Supabase). https://twitter.com/manishrjain/status/1496174276474732544 https://twitter.com/manishrjain/status/1496174276474732544
- manigandham 4y ago
  Interesting note but that seems to be recency bias with news than anything concrete. Companies from MongoDB to Snowflake to FaunaDB have been successful. Manish himself is from DGraph which is a brand new graph database with no relational to Postgres.
nikita 4y ago
Thank you for the kind words Mani! Singlestore is indeed an amazing product and company. I'm really proud of it!

tomatowurst 4y ago

how does it differ from fly.io? https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/

tptacek 4y ago
Our Postgres is just a cluster of Postgres servers. It's not "serverless"; it's server-y. :)

tuukkah 4y ago

In case anyone else is wondering, it was called Zenith / ZenithDB before launch: https://github.com/neondatabase/zenith https://github.com/neondatabase/zenith

nikita 4y ago
Neon is a better name!

hestefisk 4y ago

What is “bottom-less” storage? This just seems like a managed hosting of PostgreSQL, nothing else…

kelvich 4y ago
We wrote a custom storage layer for postgres, so in our setup, postgres node (k8s pod actually) doesn't store any data, and it is easy to start/stop/reschedule it. So while we actually are DBaaS, the closest analogy here is Aurora or Alloy, not RDS-like setups. A bit more details are on https://github.com/neondatabase/neon https://github.com/neondatabase/neon
excuses_ 4y ago
Typically it means S3 and alike. Not being bounded by disk size of your server. Or in their case they might refer to horizontal scalability of their storage layer which is independent from computing.
- nikita 4y ago
  Yes, you will never run out of storage with Neon. The reason to it that in our tiered storage implementation S3 is the cheapest tier and we will offload data into S3 if it gets too big.

infogulch 4y ago

How fast can a "scaled to zero" database start up? Does Neon use a "uninitialized hot spare" strategy to reduce startup latency like crdb? How much memory do they expect a typical single postgresql compute instance to take? I saw that Neon is targeting 'thousands' of postgresql processes per server, though with giant multi-TB servers these days that doesn't really narrow it down. Are the postgresql processes multi-tenant as well, or is multitenancy isolated to the storage layer? --- Heikki from the Neon team presented a talk about why they chose to develop Neon in Rust and what their experience was in Rust Finland 2022. https://www.youtube.com/watch?v=kAQeout-mh8 https://www.youtube.com/watch?v=kAQeout-mh8

Shorn 4y ago
https://youtu.be/kAQeout-mh8?t=700 https://youtu.be/kAQeout-mh8?t=700 Nobody knew Rust, so they started out by hiring someone who did. Good move. Business idea: consultancy that hires out competent Rust devs to new projects.
nikita 4y ago
Postgres process in single tenant. Right now both provisioning a new Postgres instance (we call it project) and cold start is 2 seconds. We will be improving on that.

nikita 4y ago

Nikita - CEO of Neon here. We intended to post this at the launch next month, but since it here, I'm happy to answer any questions. We have been hard at work and looking to open the service to the public soon.

zeusly 4y ago
Hey Nikita, could you maybe put some more legal information on the webpage? I'm trying to find out if you're a company and where you are located. Is there no legal entity behind this? Do you have a privacy policy?
- mattashii 4y ago
  The company is Neon, Inc., which is registered in the USA. We're a remote company, with a significant portion of the developers being located in Europe. Privacy policy and related stuff will be ready when we publish the public beta, which we expect to happen soon.
  - nikita 4y ago
    Yep, Delaware corp with top tier US investors. I'm in the Bay Area. Heikki is in Finland. Stas is in Cyprus. Majority of engineering is in Europe, some in the US and Canada. Postgres is a global phenomenon.
akmodi 4y ago
Hey Nikita! I was just looking at the docs but I was a bit confused about what the various compute instances were doing. Do they all serve reads and writes? If so, is there data partitioning or does this support distributed transactions?
- nikita 4y ago
  Various compute instances are different endpoints to separate databases. So for now it's single writer system. You can get a lot of power out of a 128 core compute node. In the future will will also spin up extra compute to scale reads. In the future after that future we will introduce data partitioning - we have a cool design for it, but one step at a time.

AndrewDucker 4y ago

Having a relational database where you're charged purely for the calls you make is a game-changer. All of the relational databases I looked at in the past required you to have a gateway node on at all times, which is far too expensive for a simple hobby project.

rubenfiszel 4y ago

This is exactly what I was looking for windmill, an OSS tool for multi-step automation from scripts. By any chance, are documentation to host it on top of nomad be on the Roadmap ? If yes, I would try it immediately to replace the postgres server we currently use.

nikita 4y ago
No documentation of how to self host yet. It's not hard, it requires K8S and S3 compatible object store. We want to harden Neon operationally before documenting and supporting on prem production installs.
- ccakes 4y ago
  Requires k8s as in - uses k8s APIs? Or is supported on k8s? Apologies haven’t read the docs but wanted to highlight this. A hard requirement on k8s to the exclusion of other schedulers would be a shame and an odd choice
  - mattashii 4y ago
    We've built our current cloud offering on k8s, and that is our only scale-to-0 implementation. That does not mean that you cannot run Neon outside k8s, but we are not actively maintaining nor supporting other hosting options.
- rubenfiszel 4y ago
  Got it, makes sense. I think we are pursuing similar strategies with OSS/self-hostable products. I would love to get in touch with you about it. Could you drop me an email at ruben@windmill.dev ?
  - nikita 4y ago
    Replied

sudhirj 4y ago

Does this storage layer still require VACUUMing? The Aurora system that AWS has uses a log based storage system, so along with point in time restore there’s no longer a need to go free up old garbage rows. This system says it’s copy on write, wondering if vacuuming is still necessary.

petergeoghegan 4y ago
The difficulty comes from the fact that VACUUM requires most of the same state that is required when processing a user transaction. For example, VACUUM needs the commit log to check whether a given transaction ID is for a transaction that committed or aborted (VACUUM also truncates the commit log to remove information about older XIDs when it is no longer required). Aurora Postgres does still have VACUUM, which seems to work in the same way -- which is the same in Neon. AWS have in the past promoted Aurora as having significantly better performance characteristics when VACUUM is run. That may well be true, but the benefits seem to come from not having to generate full page writes in WAL, which are a way of preventing a low-level problem called torn pages. Theoretically you could just turn off full-page writes in standard Postgres if you had hardware that offered atomic page writes, though I don't think that it's a widespread practice. I spend most of my time working on problems with VACUUM in Postgres itself (I'm one of the committers). An approach to organizing storage within transactional constraints seems necessary to push vacuuming down to storage, and that would more than likely need plenty of work in Postgres itself to be practical -- since it would cross a few layers of abstraction. Currently heapam doesn't specifically try to keep tuples inserted around the same time together, so it's hard to make anything that VACUUM does work implicitly or logically. Disclaimer: I work for Neon
- sudhirj 4y ago
  Thanks for the explanation, I’m nowhere near your experience level with this, but from what I understand Aurora is an append only log with rolling snapshots (this might be wrong). In an append only log storage system there would be no concept of page writes, right? The VACUUM process might still be necessary to find and mark dead tuples, but this seems very minor compared to filesystem operations to reclaim (sometimes gigabytes) of storage. The reclamation of storage would simply left to the background snapshotting process and would not affect DB performance in any way.

iknownothow 4y ago

I knew my bet to sticking with Postgres would pay off! This looks super exciting. I thought of doing something similar for our data warehouse with AWS Fargate and Postgres but the cold starts and limited disk space required too much engineering on top to make it work. Moving to Snowflake comes at the cost of losing so many Posgtres features in exchange for speed. Things like foreign keys, constraints, extensions etc which requires so much engineering to replace in Snowflake. I would be happy to pay 25x the price for a 10x speed increase for a specific query.

thejosh 4y ago
Snowflake is a data warehouse though. Completely different use case. If your data can be done via PG, highly recommend that over SF. Especially with this concept. Snowflake is great when you use a tool like dbt, their modern SQL approach and functions are fantastic. Downsides is it's pretty pricey, and can catch you out.
- iknownothow 4y ago
  We already use DBT and the data is less than 10TB, something Postgres can handle well. And most of the data is concentrated in a few tables. With a serverless approach I'd be happy to allocate 10x resources for just a query or two and for the rest a minimal server is fine. I manage the data warehouse mostly alone because Postgres offers guarantees, unique constraints, triggers and relationships between columns of different tables. It does the work of two engineers. Snowflake is fast but not Postgres compatible. In order to move to Snowflake, I have to write tests and maintain them which Postgres does for me for free. I'd stick with Postgres at least until 20TB before considering Snowflake.
manigandham 4y ago
Snowflake is an OLAP system. It's an entirely different kind of "speed" designed for analyzing vast amounts of data through scans and aggregations.
- iknownothow 4y ago