12 ms·

How we built a serverless SQL database

265 points by TheresaBraccio 5y ago

geoduck14 5y ago
How does this compare to Snowflake?
- reilly3000 5y ago
  Snowflake is OLAP and billed based on usage time + storage. This looks like it’s a regular OLTP SQL DB and pay by request.
  - geoduck14 5y ago
    I see. Looking at the docs, the pricing for storing data is measured in Gigs (I think $1 per gig per month) - not Tb. This makes me think it is intended for smallish data. I have a database now with 40 Tb - this would cost $40k a month just to store!
    - reilly3000 5y ago
      Yikes! I think this is really positioned for side projects and microservices. There is a clear path to migrating to a full instance or Postgres cluster over certain usage levels, but for many cases being able to scale to zero is still economically and operationally valuable. 40Tb is a ton-o-data! Snowflake sounds like a good fit.
      - geoduck14 5y ago
        Yeah, they get style points for "scaling to zero". I've seen plenty of vendors who "are scalable" - but with a floor of $70k/yr. I like their product offering - I'd be interested in using them for hobby projects, but not for work atm.
andydb 5y ago
To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
- corentin88 5y ago
  Do you have a getting started guide on CockroachDB Serverless? I couldn’t find one in your docs [1]. This looks very interesting. [1] https://www.cockroachlabs.com/docs/v21.1/example-apps.html https://www.cockroachlabs.com/docs/v21.1/example-apps.html
  - shampeon 5y ago
    The Quickstart [1] is what you're looking for. Examples for NodeJS, Python, Go, and Java. [1]https://www.cockroachlabs.com/docs/cockroachcloud/quickstart.html https://www.cockroachlabs.com/docs/cockroachcloud/quickstart...
- chillfox 5y ago
  How low can CocroachDB go with resource usage? Is the open source version viable for small hobby projects?
- jawns 5y ago
  What's your elevator pitch for why my organization should use CockroachDB Serverless vs. something like AWS Aurora Serverless, particularly if we're already relatively invested in the AWS ecosystem?
  - qaq 5y ago
    Not CDB employee but CDB scales beyond what Aurora can support.
  - andydb 5y ago
    Oh boy, I'm an engineer, but I'll do my best to pretend I'm on the sales or marketing team for a minute... First of all, CockroachDB Serverless is available on AWS, and should integrate quite well with that ecosystem, including with Serverless functions offered by AWS Lambda. Here are a few advantages of CockroachDB Serverless that Aurora will struggle to match (note that we're still working on Serverless multi-region support): 1. Free-forever tier. We offer a generous "free forever" tier that doesn't end after a month or a year. As the blog post outlines, our architecture is custom-built to make this economical. 2. No ceiling on write scalability. Even non-Serverless Aurora runs into increasing trouble as the number of writes / second increases past what a single machine can handle. CockroachDB just keeps going. We've had multiple high-scale customers who hit Aurora limits and had to move over to Cockroach to support business growth. 3. True multi-region support. Aurora only allows read-only, stale replicas in other regions, while CRDB allows full ACID SQL transactions. If you want to move into other regions of the world and have latency concerns or GDPR concerns, CRDB is custom-built to make the full SQL experience possible. 4. No Cloud lock-in. Perhaps this is not a concern for you company, but many companies don't like getting completely locked in to a single Cloud provider. CockroachDB works on multiple cloud providers and doesn't have a monetary interest in locking you in to just one. 5. Online schema changes. CockroachDB supports operations like adding/removing columns, renaming tables, and adding constraints without any downtime. You can perform arbitrary schema changes without disturbing your running application workloads. SQL DDL "just works". 6. Cold start in an instant. CockroachDB clusters automatically "scale to zero" when they're not in use. When traffic arrives, they resume in a fraction of a second. Compare that to Aurora, where you need to either have a minimum compute reservation, or you need to endure multi-second cold starts. 7. Great support. We've got a friendly Slack room where you can get free support and rub shoulders with fellow CockroachDB users, as well as CockroachDB folks like myself. We also have 24/7 paid support for deeper problems you might encounter. Taken altogether, CockroachDB can go wherever your business needs it to go, without all the constraints that traditional SQL databases usually have. Do you want thousands of clusters for testing/development/tiny apps at a reasonable cost? Could your business take off and need the scale that CRDB offers? Could your business need to expand into multiple geographic regions? Are some of your workloads erratic or periodic, but still should start up instantly when needed? It's not just about what you need now, but what you may need in the future. It makes sense to plan ahead and go with a database that has "got you covered" wherever you need to go.
rabaut 5y ago
Does CockroachDB Serverless expose an HTTP api? This sounds like a great fit for use with Cloudflare Workers, but that requires an http api.
- andydb 5y ago
  Great question. We recognize how important this is and are actively working on it.
  - webmaven 5y ago
    Is it possible to use PostgREST in front of CRDB to solve this use case?: https://postgrest.org/ https://postgrest.org/
    - andydb 5y ago
      We're working on enabling PostgREST. There is some small-ish functionality that we need do in CockroachDB for it to fully work.
- sharps_xp 5y ago
  Why do you prefer an HTTP API versus a DB connection? Isn't the former going to inherently have the overhead cost of creating the connection + TLS handshakes? My question is similar, which is, is CockroachDB going to have an equivalent RDS proxy so that apps can handle traffic spikes and not have to deal with problems with DB connection pools
  - pistoriusp 5y ago
    I think we won't be getting socket connections in some of the wasm powered JS runtime engines soon. Using http solves that, and a bunch of caching issues.
    - estambar 5y ago
      there are some great projects that present an http api for postgres that you could use with CRDB I think. I'm thinking of something like https://github.com/pramsey/pgsql-http https://github.com/pramsey/pgsql-http
timwis 5y ago
This sounds great! I’ve wanted to create an open data portal for a while that lets you spin up a (ephemeral, read-only) Postgres database of a dataset and run queries on it, maybe with a notebook. Sounds like this might be perfect!
- chatmasta 5y ago
  You might like what we’re building at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect
  - timwis 5y ago
    Oh wow, very relevant indeed! I guess I thought ephemeral DBs would be better so that a user’s expensive query wouldn’t bog down the db for other users. And rather than just limiting them, enabling them to do whatever queries they could with pg running locally
    - rad_gruchalski 5y ago
      This is something we are currently building at Klarrio on top of YugabyteDB. We have opted for YugabyteDB table spaces to solve the isolation problem. We allocate a number of dedicated tablet servers to a tenant so that their db objects are close to each other. YugabyteDB executes queries only on those tablet servers where the data relevant to the query is. A resource heavy tenant doesn’t have an impact on other tenants. It’s not really “serverless” but does seem to solve the problem at a cost. Tenants receive complete databases with grant permissions so they can do as they please. Because YugabyteDB uses Postgres 11 under the hood, we have an additional sanity layer in form of a Postgres process utility hook which prevents the tenant from escaping the sandbox.
    - chatmasta 5y ago
      The query layer is composed of ephemeral Postgres instances that get created on-the-fly. These instances either proxy to live data via FDW, or they lazily download the columnar fragments necessary to resolve a query. This is meant to perform as a horizontally scalable caching layer for shared queries. For more optimized/predictable use-cases you can “check out” a version of a data image ahead-of time. In this case you use the `sgr` command line tool to load the objects of an image into a Postgres database that you can then scale vertically as normal. Our site is a bit out of date — we’ve got a new marketing site coming shortly, and we’ve gathered an all-star engineering team to take this to the next level. Stay tuned :) (Or better yet if you want a unified data stack for your team, please get in touch… we can load data from 100+ sources into versioned Splitgraph images, or we can index and proxy to existing data sources with federated querying).
kendru 5y ago
The lack of a serverless option was the only reason that I did not use Cockroach on a recent project. I am excited to see the serverless offering now, and the architectural details in the post are awesome. Nice work!
pachico 5y ago
I really see the greatness of the serverless option. Congratulations! What I can't really understand is why would someone use the dedicated cluster in AWS at that price.
- qaq 5y ago
  pricing
- dmitriid 5y ago
  - Pricing - storage and db access is still mostly through hoops - long running processes - pain to develop, test and debug
ed25519FUUU 5y ago
> And you’ll never be surprised by a bill, because you can set a guaranteed monthly spend limit. It’s amazing that this a killer and not standard feature, but here we are!
babelfish 5y ago
Really great post, thanks for sharing. I spent a lot of time a couple months ago researching 'DBaaS' offerings (for fun, not business) and found it difficult to find any posts outlining the architecture of a DBaaS. Really cool to see CRDB putting this out in the open.
- andydb 5y ago
  It's been something we've done since the start and plan to continue doing. If you read back over our engineering blog, you'll find a surprisingly thorough description of the entire CockroachDB stack, from the lowest storage layer to distributed transactions, Raft consensus, SQL => key-value mapping, online schema changes, cost-based SQL optimizer, Kubernetes usage, and so on. In fact, when we onboard new engineers, we typically point them to a stack of external blog entries to read in order to get up-to-speed on how CockroachDB works. Being open on how we solve hard problems is the way to build our collective knowledge as a developer community. Certainly CockroachDB itself has benefited enormously from all that has gone before and been published in the open.
estambar 5y ago
One of the first apps to migrate from CockroachDB Dedicated to Serverless is live now if someone wants to try it out. https://web.flightchop.com/dashboard https://web.flightchop.com/dashboard - posting for a friend.
phoboslab 5y ago
> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even funny anymore.
- deleted 5y ago
  [deleted]
- wyager 5y ago
  Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.
- mjb 5y ago
  What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability. What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility. What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.
  - phoboslab 5y ago
    Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.
boynamedsue 5y ago
Any plans to expand the region availability for AWS beyond us-west-2 in the US? I am interested in us-east-2.
- andydb 5y ago
  Yes, definitely, we'll be expanding to more regions soon.
mdasen 5y ago
I think some of the value of Cockroach Serverless may depend on what the RUs (request units) map to. Looking at some competitors... Google says that Cloud Spanner should get you around 7,000 read queries per second and 1,800 write queries per second for $650/mo. If a simple indexed read is 1 RU, $650 on Cockroach Serverless would get you around 2,500 reads/second. Of course, I think it's completely reasonable for a Serverless option to cost a bit more given that you'd need to over-provision Cloud Spanner (even if you smartly increased/decreased the amount of compute allocated based on demand). Planet Scale charges $15 per 100M rows read and $15 per 10M rows written. If an RU is a row read, then Cockroach Serverless would be $10 per 100M rows read. If a write takes 10 RUs, Cockroach would cost $10 per 10M rows written. Both of those would be less than Planet Scale's cost - but it's possible that a row read will cost more than 1 RU. Let's say that an indexed lookup of a row costs 5 RUs. Then Cockroach Serverless starts costing 3.3x more than Planet Scale. AWS DynamoDB charges $1.25 per million write request units and $0.25 per million read request units. If I can get 10M reads from Cockroach Serverless for $1 and 4M reads from DynamoDB for $1, Cockroach's pricing looks pretty good. Of course, if I need 5 RUs to do an indexed read, the pricing doesn't look as good anymore. I do respect Cockroach Labs somewhat ambiguous description here. Planet Scale's $15 per 10M rows written feels like something that could become bad. What if I define hundreds of indexes on the table? What if I'm inserting very large blob/text columns that are 50MB in size? Likewise, what if I index no columns and end up forcing a full table scan, but only 10 rows are returned? Do they consider that I "read" 10 rows or "read" all the rows in the table? If it's the former, I'm putting a lot of strain on their system without paying for it. If it's the latter, I'm going to just define indexes that might not be worth it if I were paying for the IO needed to do all that writing. Still, it would be nice if Cockroach Labs offered some indication of what could be accomplished with 1 RU. "An indexed read of 1 row or an indexed read of a few rows in sequence; for example, 'SELECT * FROM people WHERE age > 18 ORDER BY age, name LIMIT 10' where there exists an index on (age, name)." That would let me know what to expect. "A write of a row under 4KB in size with no secondary indexes will cost 3 RUs; expect secondary indexes to increase the cost by 1 RU each" would give me an idea of what's going on. I think there are definitely cases that one can't easily enumerate. For example, "UPDATE people SET age = 18 WHERE EXISTS (SELECT * FROM legacy_info WHERE people.id = legacy_info.people_id AND legacy_info.is_adult = true)". That's potentially going to require lots of stuff that's harder to predict. However, at this point I don't know if an indexed read of a row costs 1 RU or 10 RU. If an indexed read of a single row costs 1 RU, if I read 10 rows sequentially will that mean 10 RUs or will 1 RU have enough IO to cover that since they're sequential (or will the billing just over-charge since there's tangibly more rows and that's easy to explain)? I think a decent amount of the value depends on the pricing and it's hard to judge that right now. One thing I will note is that the storage seems expensive. It's slightly cheaper than Planet Scale, but a lot more than the $0.30/GB of Cloud Spanner or the $0.23/GB of FaunaDB or $0.25/GB of DynamoDB. I've been wondering a bit about the storage pricing of Planet Scale since $1.25/GB seems expensive. Cockroach Serverless is coming in at $1/GB which also seems expensive compared compared to alternatives. If Cloud Spanner is basically offering a third the price, is the "Serverless" flexibility worth it given that Spanner can be scaled up/down pretty easily in very granular increments. Actually, one thing that could be useful might be noting how many request units per month one of the dedicated instances would have. A 2 vCPU CockroachDB instance costs $350-400. Would that be 4 billion request units per month (assuming you were fully utilizing the box)? Would it be more like 15 billion request units per month since you're presumably paying a premium for the Serverless flexibility?
WatchDog 5y ago
This offering seems a lot more compelling than the serverless databases offered by AWS directly. At least if cost and scaling to zero is important to you. Dynamodb addresses the cost issue, but it's so painful to use, and is unsuitable for a lot of use-cases. Aurora Serverless takes 30 seconds to cold-start, so it's also usually a non-starter, other than for batch type workloads. So great work, but I have a couple of questions: Any plans to support AWS privatelink? I played around with this a bit, and it seems you need to connect to it over the internet which isn't always ideal. Will there be a limit on how many free databases you can create? I think there are some valid use-cases where one might want to create a bunch of them, but I would be scared to do this at the risk of being kicked off your platform for abuse.
- andydb 5y ago
  Our product roadmap will be heavily influenced by customer asks. So if there are things about CockroachDB Serverless that prevent you from using it (like requiring ingress through a public IP), we definitely want to hear about it. Regarding a cluster limit, we currently allow up to 5 clusters per customer account. I'd like to hear what kind of use-cases you have in mind for having a lot more clusters. One I've thought about is CI runs, where you'd want dozens or hundreds of temporary clusters running at once, in order to run your testing in parallel.
  - WatchDog 5y ago
    I guess two main use-cases come to mind, providing isolated multi-tenant saas services, also easily creating test/dev environments. In a similar vein to how you have made cockroach multi-tenant, not too long ago I worked on building a multi-tenant sass version of a business intelligence app. The app uses a relational DB, initially we used separate schemas on the same db cluster, but we had problems with noisy neighbors, as well as concerns about it's security. We later opted to run dedicated database clusters for each tenant, however it greatly increases the marginal cost, and makes it difficult to provide a free tier of service, which is a valuable way to gain new customers.
    - 5y ago

How does this compare to Snowflake?

reilly3000 5y ago
Snowflake is OLAP and billed based on usage time + storage. This looks like it’s a regular OLTP SQL DB and pay by request.
- geoduck14 5y ago
  I see. Looking at the docs, the pricing for storing data is measured in Gigs (I think $1 per gig per month) - not Tb. This makes me think it is intended for smallish data. I have a database now with 40 Tb - this would cost $40k a month just to store!
  - reilly3000 5y ago
    Yikes! I think this is really positioned for side projects and microservices. There is a clear path to migrating to a full instance or Postgres cluster over certain usage levels, but for many cases being able to scale to zero is still economically and operationally valuable. 40Tb is a ton-o-data! Snowflake sounds like a good fit.
    - geoduck14 5y ago
      Yeah, they get style points for "scaling to zero". I've seen plenty of vendors who "are scalable" - but with a floor of $70k/yr. I like their product offering - I'd be interested in using them for hobby projects, but not for work atm.

To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.

corentin88 5y ago
Do you have a getting started guide on CockroachDB Serverless? I couldn’t find one in your docs [1]. This looks very interesting. [1] https://www.cockroachlabs.com/docs/v21.1/example-apps.html https://www.cockroachlabs.com/docs/v21.1/example-apps.html
- shampeon 5y ago
  The Quickstart [1] is what you're looking for. Examples for NodeJS, Python, Go, and Java. [1]https://www.cockroachlabs.com/docs/cockroachcloud/quickstart.html https://www.cockroachlabs.com/docs/cockroachcloud/quickstart...
chillfox 5y ago
How low can CocroachDB go with resource usage? Is the open source version viable for small hobby projects?
jawns 5y ago
What's your elevator pitch for why my organization should use CockroachDB Serverless vs. something like AWS Aurora Serverless, particularly if we're already relatively invested in the AWS ecosystem?
- qaq 5y ago
  Not CDB employee but CDB scales beyond what Aurora can support.
- andydb 5y ago
  Oh boy, I'm an engineer, but I'll do my best to pretend I'm on the sales or marketing team for a minute... First of all, CockroachDB Serverless is available on AWS, and should integrate quite well with that ecosystem, including with Serverless functions offered by AWS Lambda. Here are a few advantages of CockroachDB Serverless that Aurora will struggle to match (note that we're still working on Serverless multi-region support): 1. Free-forever tier. We offer a generous "free forever" tier that doesn't end after a month or a year. As the blog post outlines, our architecture is custom-built to make this economical. 2. No ceiling on write scalability. Even non-Serverless Aurora runs into increasing trouble as the number of writes / second increases past what a single machine can handle. CockroachDB just keeps going. We've had multiple high-scale customers who hit Aurora limits and had to move over to Cockroach to support business growth. 3. True multi-region support. Aurora only allows read-only, stale replicas in other regions, while CRDB allows full ACID SQL transactions. If you want to move into other regions of the world and have latency concerns or GDPR concerns, CRDB is custom-built to make the full SQL experience possible. 4. No Cloud lock-in. Perhaps this is not a concern for you company, but many companies don't like getting completely locked in to a single Cloud provider. CockroachDB works on multiple cloud providers and doesn't have a monetary interest in locking you in to just one. 5. Online schema changes. CockroachDB supports operations like adding/removing columns, renaming tables, and adding constraints without any downtime. You can perform arbitrary schema changes without disturbing your running application workloads. SQL DDL "just works". 6. Cold start in an instant. CockroachDB clusters automatically "scale to zero" when they're not in use. When traffic arrives, they resume in a fraction of a second. Compare that to Aurora, where you need to either have a minimum compute reservation, or you need to endure multi-second cold starts. 7. Great support. We've got a friendly Slack room where you can get free support and rub shoulders with fellow CockroachDB users, as well as CockroachDB folks like myself. We also have 24/7 paid support for deeper problems you might encounter. Taken altogether, CockroachDB can go wherever your business needs it to go, without all the constraints that traditional SQL databases usually have. Do you want thousands of clusters for testing/development/tiny apps at a reasonable cost? Could your business take off and need the scale that CRDB offers? Could your business need to expand into multiple geographic regions? Are some of your workloads erratic or periodic, but still should start up instantly when needed? It's not just about what you need now, but what you may need in the future. It makes sense to plan ahead and go with a database that has "got you covered" wherever you need to go.

rabaut 5y ago

Does CockroachDB Serverless expose an HTTP api? This sounds like a great fit for use with Cloudflare Workers, but that requires an http api.

andydb 5y ago
Great question. We recognize how important this is and are actively working on it.
- webmaven 5y ago
  Is it possible to use PostgREST in front of CRDB to solve this use case?: https://postgrest.org/ https://postgrest.org/
  - andydb 5y ago
    We're working on enabling PostgREST. There is some small-ish functionality that we need do in CockroachDB for it to fully work.
sharps_xp 5y ago
Why do you prefer an HTTP API versus a DB connection? Isn't the former going to inherently have the overhead cost of creating the connection + TLS handshakes? My question is similar, which is, is CockroachDB going to have an equivalent RDS proxy so that apps can handle traffic spikes and not have to deal with problems with DB connection pools
- pistoriusp 5y ago
  I think we won't be getting socket connections in some of the wasm powered JS runtime engines soon. Using http solves that, and a bunch of caching issues.
  - estambar 5y ago
    there are some great projects that present an http api for postgres that you could use with CRDB I think. I'm thinking of something like https://github.com/pramsey/pgsql-http https://github.com/pramsey/pgsql-http

timwis 5y ago

This sounds great! I’ve wanted to create an open data portal for a while that lets you spin up a (ephemeral, read-only) Postgres database of a dataset and run queries on it, maybe with a notebook. Sounds like this might be perfect!

chatmasta 5y ago
You might like what we’re building at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect
- timwis 5y ago
  Oh wow, very relevant indeed! I guess I thought ephemeral DBs would be better so that a user’s expensive query wouldn’t bog down the db for other users. And rather than just limiting them, enabling them to do whatever queries they could with pg running locally
  - rad_gruchalski 5y ago
    This is something we are currently building at Klarrio on top of YugabyteDB. We have opted for YugabyteDB table spaces to solve the isolation problem. We allocate a number of dedicated tablet servers to a tenant so that their db objects are close to each other. YugabyteDB executes queries only on those tablet servers where the data relevant to the query is. A resource heavy tenant doesn’t have an impact on other tenants. It’s not really “serverless” but does seem to solve the problem at a cost. Tenants receive complete databases with grant permissions so they can do as they please. Because YugabyteDB uses Postgres 11 under the hood, we have an additional sanity layer in form of a Postgres process utility hook which prevents the tenant from escaping the sandbox.
  - chatmasta 5y ago
    The query layer is composed of ephemeral Postgres instances that get created on-the-fly. These instances either proxy to live data via FDW, or they lazily download the columnar fragments necessary to resolve a query. This is meant to perform as a horizontally scalable caching layer for shared queries. For more optimized/predictable use-cases you can “check out” a version of a data image ahead-of time. In this case you use the `sgr` command line tool to load the objects of an image into a Postgres database that you can then scale vertically as normal. Our site is a bit out of date — we’ve got a new marketing site coming shortly, and we’ve gathered an all-star engineering team to take this to the next level. Stay tuned :) (Or better yet if you want a unified data stack for your team, please get in touch… we can load data from 100+ sources into versioned Splitgraph images, or we can index and proxy to existing data sources with federated querying).

kendru 5y ago

The lack of a serverless option was the only reason that I did not use Cockroach on a recent project. I am excited to see the serverless offering now, and the architectural details in the post are awesome. Nice work!

pachico 5y ago

I really see the greatness of the serverless option. Congratulations! What I can't really understand is why would someone use the dedicated cluster in AWS at that price.

qaq 5y ago
pricing
dmitriid 5y ago
- Pricing - storage and db access is still mostly through hoops - long running processes - pain to develop, test and debug

ed25519FUUU 5y ago

> And you’ll never be surprised by a bill, because you can set a guaranteed monthly spend limit. It’s amazing that this a killer and not standard feature, but here we are!

babelfish 5y ago

Really great post, thanks for sharing. I spent a lot of time a couple months ago researching 'DBaaS' offerings (for fun, not business) and found it difficult to find any posts outlining the architecture of a DBaaS. Really cool to see CRDB putting this out in the open.

andydb 5y ago
It's been something we've done since the start and plan to continue doing. If you read back over our engineering blog, you'll find a surprisingly thorough description of the entire CockroachDB stack, from the lowest storage layer to distributed transactions, Raft consensus, SQL => key-value mapping, online schema changes, cost-based SQL optimizer, Kubernetes usage, and so on. In fact, when we onboard new engineers, we typically point them to a stack of external blog entries to read in order to get up-to-speed on how CockroachDB works. Being open on how we solve hard problems is the way to build our collective knowledge as a developer community. Certainly CockroachDB itself has benefited enormously from all that has gone before and been published in the open.

estambar 5y ago

One of the first apps to migrate from CockroachDB Dedicated to Serverless is live now if someone wants to try it out. https://web.flightchop.com/dashboard https://web.flightchop.com/dashboard - posting for a friend.

phoboslab 5y ago

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even funny anymore.

deleted 5y ago
[deleted]
wyager 5y ago
Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.
mjb 5y ago
What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability. What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility. What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.
- phoboslab 5y ago
  Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.

boynamedsue 5y ago

Any plans to expand the region availability for AWS beyond us-west-2 in the US? I am interested in us-east-2.

andydb 5y ago
Yes, definitely, we'll be expanding to more regions soon.

mdasen 5y ago

I think some of the value of Cockroach Serverless may depend on what the RUs (request units) map to. Looking at some competitors... Google says that Cloud Spanner should get you around 7,000 read queries per second and 1,800 write queries per second for $650/mo. If a simple indexed read is 1 RU, $650 on Cockroach Serverless would get you around 2,500 reads/second. Of course, I think it's completely reasonable for a Serverless option to cost a bit more given that you'd need to over-provision Cloud Spanner (even if you smartly increased/decreased the amount of compute allocated based on demand). Planet Scale charges $15 per 100M rows read and $15 per 10M rows written. If an RU is a row read, then Cockroach Serverless would be $10 per 100M rows read. If a write takes 10 RUs, Cockroach would cost $10 per 10M rows written. Both of those would be less than Planet Scale's cost - but it's possible that a row read will cost more than 1 RU. Let's say that an indexed lookup of a row costs 5 RUs. Then Cockroach Serverless starts costing 3.3x more than Planet Scale. AWS DynamoDB charges $1.25 per million write request units and $0.25 per million read request units. If I can get 10M reads from Cockroach Serverless for $1 and 4M reads from DynamoDB for $1, Cockroach's pricing looks pretty good. Of course, if I need 5 RUs to do an indexed read, the pricing doesn't look as good anymore. I do respect Cockroach Labs somewhat ambiguous description here. Planet Scale's $15 per 10M rows written feels like something that could become bad. What if I define hundreds of indexes on the table? What if I'm inserting very large blob/text columns that are 50MB in size? Likewise, what if I index no columns and end up forcing a full table scan, but only 10 rows are returned? Do they consider that I "read" 10 rows or "read" all the rows in the table? If it's the former, I'm putting a lot of strain on their system without paying for it. If it's the latter, I'm going to just define indexes that might not be worth it if I were paying for the IO needed to do all that writing. Still, it would be nice if Cockroach Labs offered some indication of what could be accomplished with 1 RU. "An indexed read of 1 row or an indexed read of a few rows in sequence; for example, 'SELECT * FROM people WHERE age > 18 ORDER BY age, name LIMIT 10' where there exists an index on (age, name)." That would let me know what to expect. "A write of a row under 4KB in size with no secondary indexes will cost 3 RUs; expect secondary indexes to increase the cost by 1 RU each" would give me an idea of what's going on. I think there are definitely cases that one can't easily enumerate. For example, "UPDATE people SET age = 18 WHERE EXISTS (SELECT * FROM legacy_info WHERE people.id = legacy_info.people_id AND legacy_info.is_adult = true)". That's potentially going to require lots of stuff that's harder to predict. However, at this point I don't know if an indexed read of a row costs 1 RU or 10 RU. If an indexed read of a single row costs 1 RU, if I read 10 rows sequentially will that mean 10 RUs or will 1 RU have enough IO to cover that since they're sequential (or will the billing just over-charge since there's tangibly more rows and that's easy to explain)? I think a decent amount of the value depends on the pricing and it's hard to judge that right now. One thing I will note is that the storage seems expensive. It's slightly cheaper than Planet Scale, but a lot more than the $0.30/GB of Cloud Spanner or the $0.23/GB of FaunaDB or $0.25/GB of DynamoDB. I've been wondering a bit about the storage pricing of Planet Scale since $1.25/GB seems expensive. Cockroach Serverless is coming in at $1/GB which also seems expensive compared compared to alternatives. If Cloud Spanner is basically offering a third the price, is the "Serverless" flexibility worth it given that Spanner can be scaled up/down pretty easily in very granular increments. Actually, one thing that could be useful might be noting how many request units per month one of the dedicated instances would have. A 2 vCPU CockroachDB instance costs $350-400. Would that be 4 billion request units per month (assuming you were fully utilizing the box)? Would it be more like 15 billion request units per month since you're presumably paying a premium for the Serverless flexibility?

WatchDog 5y ago

This offering seems a lot more compelling than the serverless databases offered by AWS directly. At least if cost and scaling to zero is important to you. Dynamodb addresses the cost issue, but it's so painful to use, and is unsuitable for a lot of use-cases. Aurora Serverless takes 30 seconds to cold-start, so it's also usually a non-starter, other than for batch type workloads. So great work, but I have a couple of questions: Any plans to support AWS privatelink? I played around with this a bit, and it seems you need to connect to it over the internet which isn't always ideal. Will there be a limit on how many free databases you can create? I think there are some valid use-cases where one might want to create a bunch of them, but I would be scared to do this at the risk of being kicked off your platform for abuse.

andydb 5y ago
Our product roadmap will be heavily influenced by customer asks. So if there are things about CockroachDB Serverless that prevent you from using it (like requiring ingress through a public IP), we definitely want to hear about it. Regarding a cluster limit, we currently allow up to 5 clusters per customer account. I'd like to hear what kind of use-cases you have in mind for having a lot more clusters. One I've thought about is CI runs, where you'd want dozens or hundreds of temporary clusters running at once, in order to run your testing in parallel.
- WatchDog 5y ago
  I guess two main use-cases come to mind, providing isolated multi-tenant saas services, also easily creating test/dev environments. In a similar vein to how you have made cockroach multi-tenant, not too long ago I worked on building a multi-tenant sass version of a business intelligence app. The app uses a relational DB, initially we used separate schemas on the same db cluster, but we had problems with noisy neighbors, as well as concerns about it's security. We later opted to run dedicated database clusters for each tenant, however it greatly increases the marginal cost, and makes it difficult to provide a free tier of service, which is a valuable way to gain new customers.
  - 5y ago