7 ms·

There are plenty of serverless database options already: Firestore, DynamoDB, CosmosDB, FaunaDB, even MongoDB, and there are "newsql" distributed relational sys

There are plenty of serverless database options already: Firestore, DynamoDB, CosmosDB, FaunaDB, even MongoDB, and there are "newsql" distributed relational systems like CockroachDB and Planetscale with serverless plans.

pid-1 4y ago
Most stuff you mentioned: 1 - Is not an actual relational DB 2- Doesn't really scale to zero Planetscale does scale to zero, but has a ridiculous billing model.
- acjohnson55 4y ago
  What's the billing model / how is it ridiculous?
  - pid-1 4y ago
    Per row read / write billing -> how do you estimate that?
    - derekperkins 4y ago
      Either you have an existing db running and check the metrics, or you don't have a workload to move and you can just try it out from scratch. You can always migrate away if the billing model doesn't work for you. Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.
- manigandham 4y ago
  Both CRDB and Planetscale "scale" to zero - but neither expose any concept of individual instances so I'm still not sure what difference it makes.
jillesvangurp 4y ago
Many teams would prefer a postgresql compatible database with full sql support without compromises, missing features, etc. So, this could break that market open a little. Both AWS and Google are unreasonably expensive for this stuff. Most teams don't need a huge database and would be able to run postgresql on a tiny instance and get away with it. Have 2 of those and failover and backups and it's good enough for a lot of small shops. Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard. So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.
- davidzweig 4y ago
  We have a single big bare-metal machine. We run Postgres with a ~1TB DB, moderate load, on a Hetzner AX101 (16C/128GB ram). It has 2* 3.84TB nvme drives (zfs mirror with hourly snapshots) used for postgres storage only, and a seperate pair of mirrored sata drives for system/boot (had to request the extra drive, ask support to change boot option in BIOS, and reinstall OS using rescue system). It's about 100 EUR/mo with unlimited data. We bounce all incoming requests from clients (the machine also runs a node backend) through a digital ocean machine (NGINX proxy), as their peering agreements are better, without this some users in Brazil, Turkey etc. have very slow access. OVH I think would be even better for this use (better? peering and IIRC cheaper data). ZFS snapshots are backed up with sanoid to a machine under my desk with spinning disks. AX101 can be fitted with up six 3.84TB drives, that's almost 12TB of mirrored storage, we should be good for a while. You can (should) use at least lz4 compression on zfs.. can consider zstd-1, bit slower, that could double the effective space. The compression also applies to the in-ram ZFS cache, that can be 100GB+. We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.
geysersam 4y ago
Yes, but they are not "scale-to-zero"
- manigandham 4y ago
  Both CRDB and Planetscale are.
rektide 4y ago
MongoDB & CockroachDB are the only open source ones, the only ones we can hack on & improve & grow. Neon seems like a vast vast improvement & great & desperately needed potential leap for mankind.
- manigandham 4y ago
  Planetscale is Vitess which is also opensource: https://vitess.io/ https://vitess.io/ > "great & desperately needed potential leap for mankind" Are you being serious? That's very hyperbolic if so.
  - rektide 4y ago
    > Are you being serious? That's very hyperbolic if so. Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much. Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them. Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data. We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]: > This is the missing piece on cloud for masses [1] https://news.ycombinator.com/item?id=31537313 https://news.ycombinator.com/item?id=31537313
    - manigandham 4y ago
      There are tons of databases, I just named several and there are hundreds more for every possible niche. > "But ones that are not ours, that we cant hack on, that we cant run ourselves." Who is "we"? Is your entire issue that not everything is open-source? > "these are not humanity's heritdge & humanity is cut off from active participation with them." This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking https://db-engines.com/en/ranking
anilgulecha 4y ago
(Some of these are options I've not looked deeper into - Fauna, Planetscale) This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy. Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only. So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.
- manigandham 4y ago
  What is the effective difference to you? Technically all compute can enter hibernation by dumping RAM (or just using virtualized memory backed only by SSDs). CockroachDB already does true scale-to-zero if that's your requirement: https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-serverless/ https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-...
  - anilgulecha 4y ago
    The effective difference is commoditization of the paradigm - there's confidence in a commodity technology that a proprietary cannot give (standardized use and wide support and community, multiple providers, self-hosting). Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet. Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.
    - manigandham 4y ago
      > "minimum self-deployable unit of cockroachdb" If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers. The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.
nikita 4y ago
Hi Mani! For sure there are many serverless options - fewer that separated storage and compute and fewer that are open source end-to-end. Neon is also 100% compatible with Postgres (unlike CockroachDB) because compute is Postgres. Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.
- boomskats 4y ago
  You should update your HN profile :). Neon looks really great. Far more interesting to me and my team than S2. I noticed you mention Azure BS in your RFCs as a potential backend. Have you done much work towards that yet?
  - mattashii 4y ago
    For now we've focused on making the product production-ready on AWS, so that we can get users to try it out. Once the model has been proven we'll likely branch out to other clouds. However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/ https://github.com/neondatabase/neon/
    - boomskats 4y ago
      Exact reason why I asked :) Thanks for the response & all the work you're putting in, it looks really promising.
  - hlinnaka 4y ago
    Haven't done anything with Azure yet. Shouldn't be hard to add though, we don't rely on any special cloud storage features, just simple upload/download of files.

pid-1 4y ago

Most stuff you mentioned: 1 - Is not an actual relational DB 2- Doesn't really scale to zero Planetscale does scale to zero, but has a ridiculous billing model.

acjohnson55 4y ago
What's the billing model / how is it ridiculous?
- pid-1 4y ago
  Per row read / write billing -> how do you estimate that?
  - derekperkins 4y ago
    Either you have an existing db running and check the metrics, or you don't have a workload to move and you can just try it out from scratch. You can always migrate away if the billing model doesn't work for you. Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.
manigandham 4y ago
Both CRDB and Planetscale "scale" to zero - but neither expose any concept of individual instances so I'm still not sure what difference it makes.

jillesvangurp 4y ago

Many teams would prefer a postgresql compatible database with full sql support without compromises, missing features, etc. So, this could break that market open a little. Both AWS and Google are unreasonably expensive for this stuff. Most teams don't need a huge database and would be able to run postgresql on a tiny instance and get away with it. Have 2 of those and failover and backups and it's good enough for a lot of small shops. Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard. So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.

davidzweig 4y ago
We have a single big bare-metal machine. We run Postgres with a ~1TB DB, moderate load, on a Hetzner AX101 (16C/128GB ram). It has 2* 3.84TB nvme drives (zfs mirror with hourly snapshots) used for postgres storage only, and a seperate pair of mirrored sata drives for system/boot (had to request the extra drive, ask support to change boot option in BIOS, and reinstall OS using rescue system). It's about 100 EUR/mo with unlimited data. We bounce all incoming requests from clients (the machine also runs a node backend) through a digital ocean machine (NGINX proxy), as their peering agreements are better, without this some users in Brazil, Turkey etc. have very slow access. OVH I think would be even better for this use (better? peering and IIRC cheaper data). ZFS snapshots are backed up with sanoid to a machine under my desk with spinning disks. AX101 can be fitted with up six 3.84TB drives, that's almost 12TB of mirrored storage, we should be good for a while. You can (should) use at least lz4 compression on zfs.. can consider zstd-1, bit slower, that could double the effective space. The compression also applies to the in-ram ZFS cache, that can be 100GB+. We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.

geysersam 4y ago

Yes, but they are not "scale-to-zero"

manigandham 4y ago
Both CRDB and Planetscale are.

rektide 4y ago

MongoDB & CockroachDB are the only open source ones, the only ones we can hack on & improve & grow. Neon seems like a vast vast improvement & great & desperately needed potential leap for mankind.

manigandham 4y ago
Planetscale is Vitess which is also opensource: https://vitess.io/ https://vitess.io/ > "great & desperately needed potential leap for mankind" Are you being serious? That's very hyperbolic if so.
- rektide 4y ago
  > Are you being serious? That's very hyperbolic if so. Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much. Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them. Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data. We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]: > This is the missing piece on cloud for masses [1] https://news.ycombinator.com/item?id=31537313 https://news.ycombinator.com/item?id=31537313
  - manigandham 4y ago
    There are tons of databases, I just named several and there are hundreds more for every possible niche. > "But ones that are not ours, that we cant hack on, that we cant run ourselves." Who is "we"? Is your entire issue that not everything is open-source? > "these are not humanity's heritdge & humanity is cut off from active participation with them." This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking https://db-engines.com/en/ranking

anilgulecha 4y ago

(Some of these are options I've not looked deeper into - Fauna, Planetscale) This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy. Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only. So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.

manigandham 4y ago
What is the effective difference to you? Technically all compute can enter hibernation by dumping RAM (or just using virtualized memory backed only by SSDs). CockroachDB already does true scale-to-zero if that's your requirement: https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-serverless/ https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-...
- anilgulecha 4y ago
  The effective difference is commoditization of the paradigm - there's confidence in a commodity technology that a proprietary cannot give (standardized use and wide support and community, multiple providers, self-hosting). Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet. Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.
  - manigandham 4y ago
    > "minimum self-deployable unit of cockroachdb" If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers. The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.

nikita 4y ago

Hi Mani! For sure there are many serverless options - fewer that separated storage and compute and fewer that are open source end-to-end. Neon is also 100% compatible with Postgres (unlike CockroachDB) because compute is Postgres. Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.

boomskats 4y ago
You should update your HN profile :). Neon looks really great. Far more interesting to me and my team than S2. I noticed you mention Azure BS in your RFCs as a potential backend. Have you done much work towards that yet?
- mattashii 4y ago
  For now we've focused on making the product production-ready on AWS, so that we can get users to try it out. Once the model has been proven we'll likely branch out to other clouds. However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/ https://github.com/neondatabase/neon/
  - boomskats 4y ago
    Exact reason why I asked :) Thanks for the response & all the work you're putting in, it looks really promising.
- hlinnaka 4y ago
  Haven't done anything with Azure yet. Shouldn't be hard to add though, we don't rely on any special cloud storage features, just simple upload/download of files.