8 ms·
Show HN: Serverless Postgres
This is a MVP for Serverless Postgres.
1/ It uses Fly.io[0], which can automatically pause your database after all connections are released (and start it again when new connections join).
2/ It uses Oriole[1], a Postgres extension with experimental support for S3 / Decoupled Storage[2].
3/ It uses Tigris[3], Globally Distributed S3-Compatible Object Storage. Oriole will automatically backup the data to Tigris using background workers.
I wouldn't recommend using this in production, but I think it's in a good spot to provoke some discussion and ideas. You can get it running on your own machine with the steps provided - connecting to a remote Tigris bucket (can also be an AWS S3 bucket).
[0] https://fly.io https://fly.io
[1] https://www.orioledb.com/ https://www.orioledb.com/
[2] Oriole Experiemental s3: https://www.orioledb.com/docs/usage/decoupled-storage https://www.orioledb.com/docs/usage/decoupled-storage
[3] Tigris: https://www.tigrisdata.com/ https://www.tigrisdata.com/
- deleted 2y ago[deleted]
- thomasfromcdnjs 2y agoWhen would it make sense to use such a setup? What kind of applications?
- cowboylowrez 2y agoI like to think this sort of project makes sense when putting screendoors on submarines.
- kiwicopple 2y agoThe “scale to zero” aspects mean that it’s useful for very infrequently accessed applications Once I have experimented with Tigris’ global data replication then it could be useful for a distributed user-base doing a lot of reads. The idea is that you can point a fresh postgres instance to the S3 bucket for read access (rather than keeping a hot standby using Postgres native replication)
- AgharaShyam 2y agoIn addition to infrequent access type use-cases, the ability to trivially create new DBs without incurring significant costs is the killer feature. This makes it feasible to build multi-tenant applications with 1-db-per-tenant. I know the same can be achieved with RLS policies or application-layer permissions, but nothing beats the peace mind (for developer and customers alike) of having tenant data in entirely separate databases. This also makes it easier to have custom schema for different subsets of users. Side note: SQLite (the OG serverless DB) also has the same benefit but is decidedly less feature-full than postgres/mysql. Another pattern that I have seen emerging is that certain products allow end-users to create DBs on the fly as a part of their own product offering. For example, Retool (which uses Neon) allows users to create a new database in seconds, even on their free plan. I don't think this would have been possible (both cost and DX wise) if they provisioned a new Postgres cluster for each customer. On that note, and in light of what happened with the Planetscale free plan, this would make the Supabase free offering and branching to be significantly less expensive to operate. btw: thank you for the free plan and all the amazing OSS contributions from the supabase team!!
- internet101010 2y agoYou have data updates that happen at irregular intervals and don't want to be bothered with turning on and turning off a db instance.
- slt2021 2y agowhy keep separate postgres instance for this use case? One could easily have a single small Postgres instance for a few dollars that hosts all these small scale peanut sized toy databases and forget about it. if your main business critical database is not doing transactions and stays idle, you should be working on bringing more transactions (sales sales sales) - not trying to save pennies on downsized postgres by moving from SSD-fast postgres to S3-slow-postgres. I get that there are benefits to scaling up/down, but to me: Serverless is a billing model for multi-tenant services, where user is billed by service usage, not the actual underlying resource consumption. Because in multi-tenant SAAS all your tenants share the underlying infrastructure
- sea-gold 2y agoHow does this compare to Neon[1], Crunchy PostgreSQL for Kubernetes[2], and Citus[3]? [1] https://neon.tech https://neon.tech / https://github.com/neondatabase/neon https://github.com/neondatabase/neon [2] https://access.crunchydata.com/documentation/postgres-operator/latest https://access.crunchydata.com/documentation/postgres-operat... / https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator [3] https://www.citusdata.com https://www.citusdata.com / https://github.com/citusdata/citus https://github.com/citusdata/citus
- craigkerstiens 2y agoAt very first glance this would be much closer to neon, with the separated storage and compute. Crunchy Postgres for Kubernetes is great if you're running Postgres inside Kubernetes, but is more of standard Postgres than something serverless. Citus also not really serverless at all, Citus is more focused on performance scaling where things are very co-located and you're outgrowing the bounds of a single node.
- scirob 2y ago>citus is a bit out of date, now mostly native azure not flexible at all >neon has not released all of their kubernetes operator so its not fully open source yet > Crunchy is just a kubernetes operator for HA postgres. but they are working on a cool thing to natively use IceBerg as FDW. But their attitude is that this serverless separation of compute and storage is not the right approach for the primary operational data store
- paulddraper 2y agoI think the only relevant comparison here is Neon.
- yencabulator 2y agoNeon has a consensus cluster of >=3 WAL storage servers using local storage that is not scale-to-zero. Only with multitenancy can you amortize it to scale-down-to-not-much. It's not clear to me that kiwicopple's work horizontally scales compute, ever. It seems it has to be just one server, and if you ran multiple they'd corrupt the S3 storage. Neon: >= 3 nodes kiwicopple/serverless-postgres: 0..1 nodes
- canadiantim 2y agoVery cool! I’m glad to see someone working with oriole, been wanting to do something similar for a long while. How tricky is it to with the S3 storage? Esp. In regards to permissions, etc.? Thanks for sharing eh, love what you’re doing
- kiwicopple 2y agoI think it’s pretty simple with S3, and the permissions are basically “keep the bucket private” That said, this is very much an MVP and I’m sure I’ll encounter some dragons as I push for scale or as I receive feedback from customers
- justinclift 2y agoThis seems more like "PostgreSQL using remote storage for the database" rather than "Serverless"? It seems like it still needs memory/cpu resources locally (sized properly) to handle the returned data. It also seems like an incredibly bad idea to use with any remote data store that charges for network traffic. ie Amazon AWS You could easily give yourself a many thousand dollar bill for a month if you start using that backend data store in a serious way. :( :( :( --- That being said, it's a nifty idea conceptually. Just be aware of likely operational costs if you try using it. (!)
- kiwicopple 2y agoI guess the term “Serverless” is pretty ambiguous, but since this originated from AWS Lambda (firecracker) and that’s the same primitive as Fly.io then it seemed fitting. You can scale up/down compute and snapshot memory for fast startups The other tricky thing is data: there are several ways to handle this. Since Tigris has a lot of the data replication features that Postgres would traditionally handle then it looks promising. That said - it comes with tradeoffs that you point out (transfer costs) but these are also costs that you’d pay for postgres replication. I’ll try benchmark some of these costs and compare them across several implementations. At this stage I just wanted to get an MVP working: the harder stuff is still ahead Thanks for your comment - a lot of good points
- justinclift 2y ago> these are also costs that you’d pay for postgres replication. Some places define a 2nd (private) network interface for their VMs, and you can transfer data between your VMs over that private interface without cost. Not sure if AWS does it that way though.
- chipdart 2y ago> I guess the term “Serverless” is pretty ambiguous, (...) I don't this is the case though. The core principle of serverless computing is that a cloud provider supplies a managed service and scales computational resources allocated to the service to meet it's computational needs. This "serverless postgres" meets none of the items in the definition. You mentioned AWS Lambdas. AWS allocates all computational resources when a function is invoked. Let's face it: this was an attempt to abuse a buzzword.
- victorbjorklund 2y agoVery cool.
- scirob 2y agonow it would be mega cool if someone got a grant to do this with the open source stack. fly uses firecracker underneath etc...
- scirob 2y agoOh fun this is from the CEO of supabase, so i guess that's where the grant is coming from :))))
- scirob 2y agodouble funny, looks like they already acquihired the main devs from Orioledb https://github.com/akorotkov https://github.com/akorotkov now supabase https://github.com/pashkinelfe https://github.com/pashkinelfe now supabase silicon valley isn't capitalism awesome
- re-thc 2y agoThey acquired Oriole, see: https://supabase.com/blog/supabase-aquires-oriole https://supabase.com/blog/supabase-aquires-oriole
- BurnGpuBurn 2y agoServerless != serverless You use a lot of servers, you just call them resources. Your S3 storage is a server, your background workers run on a server, and your postgress, well, it doesn't run on a bike. I hate this confabulating stuff. It's like people believing "the cloud" is an actual cloud. Couldn't we just leave this stuff to the marketing people?
- INTPenis 2y agoNevertheless serverless is an actual concept where you don't deploy any server instances, you use only services. Those services might scale up with server instances, but you don't have to know about that. I was against it too at first, but then I realized how to use it.
- tmountain 2y agoNovel, and maybe I am just old school, but I truly believe that a right sized Postgres deployment is good enough for most use cases. I would trust my data to something vanilla barring a super niche scenario.
- sgarland 2y agoA vanilla Postgres or MySQL instance, when properly tuned, will take you EXTREMELY far. People forget how much latency network hops add compared to native disks. Even improperly tuned, it’ll take you much further than one would be led to believe from the SaaS shills.
- xarope 2y agoInstead of all this cloud native stuff, I wish there was an embedded version of Postgres, ala sqlite or duckdb
- irjustin 2y agoCan this run on AWS Lambda with a EBS Blockstore? Mentally this is the cheapest w/o all those egress costs, but I'm not sure. Combine that with we don't like managing resources outside AWS unless absolutely forced. I really wanted this because RDS serverless v1 could do zero compute, but v2 has a minimum compute requirement. If I can run my Postgres reliably on AWS Lambda, the costs drop to near zero for non production instances, which is where a lot of people play.
- kiwicopple 2y agoSince this is dockerized, the best AWS primitive would be Fargate or ECS As far as I know you can’t mount a file system like EBS to Lambda, and tbh even if you could the implementation of Lambda+Postgres sounds particularly cursed.
- irjustin 2y agoAh sorry I should've said EFS, got my acronyms wrong. That one is mountable to Lambda. But Lambda can't accept multiple connections to the same instance, so it's allowing multiple pg servers connect to the same data source.
- fulafel 2y agoWhy EBS and not the decoupled storage that this does?
- irjustin 2y agoAh sorry I should've said EFS, got my acronyms wrong.
- paulddraper 2y agoWell A. EFS is dogs*** for this B. The EFS protocol is NFS, so that's the question.
- 2y ago
- nijuashi 2y agoFor a moment I was excited to see sqlite with postgres compatibility.
- fulafel 2y agoNeat stuff. AWS threw in the towel with their serverless PG (Aurora v2 can no longer scale to zero).
- joshstrange 2y agoTheir serverless Aurora was always terrible, even v1. It cost way too much and didn’t live up to the promise. I left them as they went to v2 (double the price, even worse features) for PlanetScale before then moving to Neon (I still like PS but it wasn’t a good fit for my business/tech model).
- dzan 2y agoCool idea, but it sounds dangerous. Data is often the most valuable thing for the company / project / app.
- ovaistariq 2y agoI am curious why this sounds dangerous. Data is still getting persisted to storage. It’s just a different architecture where compute and storage are not colocated on the same machine.
- broner 2y agoCould you use this to create 100s of on-demand read replicas and get 100s of GB/s out of S3? Seems like a nice way to cheaply and quickly do batch jobs.