4 ms·

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with

by groundzeros2015 3mo ago

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding Did you max out the capacity of the best server you can buy? Such a database can serve millions of customers (the numbers given). You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer inspect the essential state of your system is the day your company better be included in NASDAQ and ready to pay a few hundred engineers 300k salaries.

anonzzzies 3mo ago
Well, they are selling this thing so they don't want you to buy a big server (with a read replica) as that's much cheaper.
- farslan 3mo ago
  That's not true! We have demand for bigger machines and we also sell them. You can go checkout the PlanetScale website. See: https://x.com/samlambert/status/2077197049129587150 https://x.com/samlambert/status/2077197049129587150 There are truly customers that bigger machine no longer cuts. Disclaimer: I'm an Engineer at PS.
  - anonzzzies 3mo ago
    I did not say there is no market, but I am willing to bet whatever that the majority of your client base would be much cheaper off with a simple setup where they can scale the entirety of their company’s existence with one machine. Not even a large one. This cloud stuff has its place but not for most. For them it’s either themselves overspending or wasting VC money.
- bddicken 3mo ago
  We have tons of customers who do exactly this. It's great. Sharding is for customers who out grow this path.
  - groundzeros2015 3mo ago
    Why would multiple machines be cheaper than one? Don’t you pay an overhead per machine?
anichhangani 3mo ago
Surely the cost of running sql server on premium hardware with replication would be more than running on commodity hardware with sharding?
- groundzeros2015 3mo ago
  Only if your engineering resources and free and unbounded. Even then I would probably use those resources to optimize software instead. In the physical world of trucks and cranes no company would make that mistake to try to save 30-80k.
- hilariously 3mo ago
  It depends on who you are paying for it, but generally a distributed system is harder to reason about, harder to fix, has weirder edge cases, and much more easily get into situations where it requires even MORE expertise to fix than just having a big honking server. When you start calculating things that are not just the server, the single server looks cheaper and cheaper. How do you get a consistent backup? How do you do DR? How do you tune queries when it could go to this node or that node? Now writes are going to be significantly slower if you need multi-node commit because no matter what you are racing the speed of light on the network.
  - marginalia_nu 3mo ago
    Another thing worth thinking about when running anything in parallel is the rather massive increase in error rate. When you're essentially rolling 768x the number of dice, failure outcomes will happen a lot more often than if you just roll the one or two. It's a scale that transforms a 10 year MTBF to a ~5 day MTBF.
    - hilariously 3mo ago
      A great point, you will also eventually be in a constant state of failure, and have to turn that into something you can manage, automatically.
p1necone 3mo ago
Most non FAANG orgs could probably serve all their customers from postgres on a laptop.
- don-code 3mo ago
  Let's hypothesize this is true. I worked at a startup that had _money_ - a lot of it - flowing in. Stakeholders wanted to see growth, which meant features. They did not want to see us making platform improvements; those didn't show well to clients. Best throw money at the problem, rather than sacrifice the all-important features. We had a number of architectural quirks that reflected this. One was that we ran our databases in AWS on ephemeral instance store. Yes, the kind that doesn't persist when you stop or lose the instance. The kind that you should never run a database on. We did it so that we could join them eight of them together in RAID 0 on the biggest instance types, and pull 400,000 IOPS to the disk that way. We paired this with hot spares of each shard, as well as hourly backups. We did this to support a Rails / ActiveRecord data model, for data that should never have been part of a Rails data model. We had exactly one person working part-time on query optimization. The maintenance burden was immense. The AWS bill, even more so. But again, we are hypothesizing that we could host all of this data on a laptop. To an extent, maybe up until our thousandth customer or so, I actually think we could have. But we wouldn't have spent most of our time working on _features_ - we'd have instead spent most of our time trying to performance optimize. That would've meant the investors wouldn't have seen growth, which in turn would've highly impacted our success. We had a well defined domain, and optimizing the queries for performance was not it.
  - groundzeros2015 3mo ago
    What if there was someone involved who had built web applications than run on laptops before and made different architectural decisions than the path your company went down.
    - senorrib 3mo ago
      If my grandma had wheels she would have been a bike. Most of these decisions are made by someone very early on in the process, either a cofounder or an underpaid junior engineer, when the company has zero revenue and no PMF. It's unrealistic to expect a startup to build a product (or at least it was prior to LLMs generating a lot of boilerplate) with best practices in place.
jackb4040 3mo ago
I found the article very helpful from a technical perspective, and didn't focus on the number too much, as it could easily be swapped and the decision-making process for when to shard is kinda out of scope. But I hadn't considered this, so thanks for pushing back. Good to keep in mind their incentives. I will say, since their product is a proxy whose interface is a single SQL connection, you should in theory be able to do dev queries through that black box, much the same as application queries? What is so scary here that it would require a hundred engineers?
- groundzeros2015 3mo ago
  > you should in theory be able to do dev queries through that black box Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel. A few questions I would start with: - is the system even ACID compliant? In my reading of this article, no. - is my sql feature set limited? Will it enforce all constraints? Or are their cross-shard limitations? For example the article doesn’t discuss transactions or how they would roll back, or how they guarantee a consistent view of data. Next: - how does it respond to error? - how does it respond to load? This is a complex system and complexity breeds bugs. But now you don’t have the tools or procedures to investigate those bugs because you can’t poke the system at desk with tools. You can’t run experiments; you can’t even see all the data.
  - jackb4040 3mo ago
    Thank you, good points. I'm just learning about this in real time. It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vitess-to-do-cross-shard-joins-or/ https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites... The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach. Like you I'd love to learn more about the internals of their actual SQL engine, which is just barely touched on in the article. The idea of ripping out the layer of a SQL interpreter that does just enough to route it to a real server tickles my brain in the same way as when I learned how node.js ripped the js interpreter out of a browser.
skeptic_ai 3mo ago
Hetzner has this for 100usd a month, 256gb ram can handle quite a lot of traffic already €84.70 max. per month €0.1357 per hour CPU Intel Xeon E5-1650V3 RAM 256 GB Drives 2 × 6.0 TB Enterprise HDD Location #FSN1-DC1 Information IPv4 ECC iNIC
- Tostino 3mo ago
  Lol that isn't a database server. Come on, if it has hard drives rather than (good) SSD in 2026, it's not a server you should be using for databases unless it's for something with next to no load, and you are also running your application server on the same machine.
exiguus 3mo ago
As I understand the author: They describe the journey of scale from a single database server over read replicas to sharding. And its not only about a single server, its also about network (backup, restore).
bddicken 3mo ago
Spreading requests out across hundreds, thousands, and in some cases even more is precisely what is done in the industry for big databases! Good examples: cashapp: https://code.cash.app/planetscale-metal https://code.cash.app/planetscale-metal github: https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ https://github.blog/engineering/infrastructure/partitioning-... etsy: https://www.etsy.com/codeascraft/migrating-etsyas-database-sharding-to-vitess https://www.etsy.com/codeascraft/migrating-etsyas-database-s... These companies could not realistically operate off a single database server.
- groundzeros2015 3mo ago
  > These companies could not realistically operate off a single database server. I want to see it fail first. I suggest their org chart has more to do with their architecture than database performance. Even so you’re in the category I said. Hundreds of expensive engineers maintain this stuff.
  - bddicken 3mo ago
    You should read through those articles.
    - groundzeros2015 3mo ago
      Thanks for sharing. They validated exactly what I said was of concern: - loss of atomicity and transactions. - tons of custom tooling to work around limitation of custom sql (knowing which operations are allowed and consistent). - GitHub was able to operate for almost a decade on a single MySQL instance, and it’s not clear they really hard to change. Etsy said this was a huge problem and took 5 years and thousands of PRs to resolve.
  - samlambert 3mo ago
    > I want to see it fail first. your lack of experience at scale doesn't really invalidate this architecture
amazingamazing 3mo ago
and if your one beefy server goes down? oh, now you want a replica. so much for one beefy machine.
- dizhn 3mo ago
  In the last 10 year or so my 2 beefy machines had much better uptime than the top clouds. :) I am not suggesting you do this but some businesses can actually be down a little bit, like my non-IT one. I've had issues in the past with email. If the issue happens after business hours and resolves before start of business next day, nobody will even notice.

anonzzzies 3mo ago

Well, they are selling this thing so they don't want you to buy a big server (with a read replica) as that's much cheaper.

farslan 3mo ago
That's not true! We have demand for bigger machines and we also sell them. You can go checkout the PlanetScale website. See: https://x.com/samlambert/status/2077197049129587150 https://x.com/samlambert/status/2077197049129587150 There are truly customers that bigger machine no longer cuts. Disclaimer: I'm an Engineer at PS.
- anonzzzies 3mo ago
  I did not say there is no market, but I am willing to bet whatever that the majority of your client base would be much cheaper off with a simple setup where they can scale the entirety of their company’s existence with one machine. Not even a large one. This cloud stuff has its place but not for most. For them it’s either themselves overspending or wasting VC money.
bddicken 3mo ago
We have tons of customers who do exactly this. It's great. Sharding is for customers who out grow this path.
- groundzeros2015 3mo ago
  Why would multiple machines be cheaper than one? Don’t you pay an overhead per machine?

anichhangani 3mo ago

Surely the cost of running sql server on premium hardware with replication would be more than running on commodity hardware with sharding?

groundzeros2015 3mo ago
Only if your engineering resources and free and unbounded. Even then I would probably use those resources to optimize software instead. In the physical world of trucks and cranes no company would make that mistake to try to save 30-80k.
hilariously 3mo ago
It depends on who you are paying for it, but generally a distributed system is harder to reason about, harder to fix, has weirder edge cases, and much more easily get into situations where it requires even MORE expertise to fix than just having a big honking server. When you start calculating things that are not just the server, the single server looks cheaper and cheaper. How do you get a consistent backup? How do you do DR? How do you tune queries when it could go to this node or that node? Now writes are going to be significantly slower if you need multi-node commit because no matter what you are racing the speed of light on the network.
- marginalia_nu 3mo ago
  Another thing worth thinking about when running anything in parallel is the rather massive increase in error rate. When you're essentially rolling 768x the number of dice, failure outcomes will happen a lot more often than if you just roll the one or two. It's a scale that transforms a 10 year MTBF to a ~5 day MTBF.
  - hilariously 3mo ago
    A great point, you will also eventually be in a constant state of failure, and have to turn that into something you can manage, automatically.

p1necone 3mo ago

Most non FAANG orgs could probably serve all their customers from postgres on a laptop.

don-code 3mo ago
Let's hypothesize this is true. I worked at a startup that had _money_ - a lot of it - flowing in. Stakeholders wanted to see growth, which meant features. They did not want to see us making platform improvements; those didn't show well to clients. Best throw money at the problem, rather than sacrifice the all-important features. We had a number of architectural quirks that reflected this. One was that we ran our databases in AWS on ephemeral instance store. Yes, the kind that doesn't persist when you stop or lose the instance. The kind that you should never run a database on. We did it so that we could join them eight of them together in RAID 0 on the biggest instance types, and pull 400,000 IOPS to the disk that way. We paired this with hot spares of each shard, as well as hourly backups. We did this to support a Rails / ActiveRecord data model, for data that should never have been part of a Rails data model. We had exactly one person working part-time on query optimization. The maintenance burden was immense. The AWS bill, even more so. But again, we are hypothesizing that we could host all of this data on a laptop. To an extent, maybe up until our thousandth customer or so, I actually think we could have. But we wouldn't have spent most of our time working on _features_ - we'd have instead spent most of our time trying to performance optimize. That would've meant the investors wouldn't have seen growth, which in turn would've highly impacted our success. We had a well defined domain, and optimizing the queries for performance was not it.
- groundzeros2015 3mo ago
  What if there was someone involved who had built web applications than run on laptops before and made different architectural decisions than the path your company went down.
  - senorrib 3mo ago
    If my grandma had wheels she would have been a bike. Most of these decisions are made by someone very early on in the process, either a cofounder or an underpaid junior engineer, when the company has zero revenue and no PMF. It's unrealistic to expect a startup to build a product (or at least it was prior to LLMs generating a lot of boilerplate) with best practices in place.

jackb4040 3mo ago

I found the article very helpful from a technical perspective, and didn't focus on the number too much, as it could easily be swapped and the decision-making process for when to shard is kinda out of scope. But I hadn't considered this, so thanks for pushing back. Good to keep in mind their incentives. I will say, since their product is a proxy whose interface is a single SQL connection, you should in theory be able to do dev queries through that black box, much the same as application queries? What is so scary here that it would require a hundred engineers?

groundzeros2015 3mo ago
> you should in theory be able to do dev queries through that black box Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel. A few questions I would start with: - is the system even ACID compliant? In my reading of this article, no. - is my sql feature set limited? Will it enforce all constraints? Or are their cross-shard limitations? For example the article doesn’t discuss transactions or how they would roll back, or how they guarantee a consistent view of data. Next: - how does it respond to error? - how does it respond to load? This is a complex system and complexity breeds bugs. But now you don’t have the tools or procedures to investigate those bugs because you can’t poke the system at desk with tools. You can’t run experiments; you can’t even see all the data.
- jackb4040 3mo ago
  Thank you, good points. I'm just learning about this in real time. It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vitess-to-do-cross-shard-joins-or/ https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites... The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach. Like you I'd love to learn more about the internals of their actual SQL engine, which is just barely touched on in the article. The idea of ripping out the layer of a SQL interpreter that does just enough to route it to a real server tickles my brain in the same way as when I learned how node.js ripped the js interpreter out of a browser.

skeptic_ai 3mo ago

Hetzner has this for 100usd a month, 256gb ram can handle quite a lot of traffic already €84.70 max. per month €0.1357 per hour CPU Intel Xeon E5-1650V3 RAM 256 GB Drives 2 × 6.0 TB Enterprise HDD Location #FSN1-DC1 Information IPv4 ECC iNIC

Tostino 3mo ago
Lol that isn't a database server. Come on, if it has hard drives rather than (good) SSD in 2026, it's not a server you should be using for databases unless it's for something with next to no load, and you are also running your application server on the same machine.

exiguus 3mo ago

As I understand the author: They describe the journey of scale from a single database server over read replicas to sharding. And its not only about a single server, its also about network (backup, restore).

bddicken 3mo ago

Spreading requests out across hundreds, thousands, and in some cases even more is precisely what is done in the industry for big databases! Good examples: cashapp: https://code.cash.app/planetscale-metal https://code.cash.app/planetscale-metal github: https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ https://github.blog/engineering/infrastructure/partitioning-... etsy: https://www.etsy.com/codeascraft/migrating-etsyas-database-sharding-to-vitess https://www.etsy.com/codeascraft/migrating-etsyas-database-s... These companies could not realistically operate off a single database server.

groundzeros2015 3mo ago
> These companies could not realistically operate off a single database server. I want to see it fail first. I suggest their org chart has more to do with their architecture than database performance. Even so you’re in the category I said. Hundreds of expensive engineers maintain this stuff.
- bddicken 3mo ago
  You should read through those articles.
  - groundzeros2015 3mo ago
    Thanks for sharing. They validated exactly what I said was of concern: - loss of atomicity and transactions. - tons of custom tooling to work around limitation of custom sql (knowing which operations are allowed and consistent). - GitHub was able to operate for almost a decade on a single MySQL instance, and it’s not clear they really hard to change. Etsy said this was a huge problem and took 5 years and thousands of PRs to resolve.
- samlambert 3mo ago
  > I want to see it fail first. your lack of experience at scale doesn't really invalidate this architecture

amazingamazing 3mo ago

and if your one beefy server goes down? oh, now you want a replica. so much for one beefy machine.

dizhn 3mo ago
In the last 10 year or so my 2 beefy machines had much better uptime than the top clouds. :) I am not suggesting you do this but some businesses can actually be down a little bit, like my non-IT one. I've had issues in the past with email. If the issue happens after business hours and resolves before start of business next day, nobody will even notice.