4 ms·

> How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never under

> How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand. It works, with a lower cognitive burden than that of horizontally scaling. For the loading concern (i.e. is this enough to handle the load): For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k users typically has fewer than 20k concurrent requests in flight at peak. A cheap VPS running PostgreSQL can easily handle that.[1] For the "if something breaks" concern: Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. The mechanisms to reduce downtime when the single large SQL server shits the bed (failovers, RO followers, whatever) can reduce that downtime to mere minutes. What is the benefit to removing 3 minutes of downtime? $100? $1k? $100k? $1m? The business will have to decide what those 3 minutes are worth, and if that worth exceeds the cost of using something other than a single large SQL server. Until and unless you reach the load and downtime-cost of Google, Amazon, Twitter, FB, Netflix, etc, you're simply prematurely optimising for a scenario that, even in the businesses best-case projections, might never exist. The best thing to do, TBH, is ask the business for their best-case projections and build to handle 90% of that. [1] An expensive VPS running PostgreSQL can handle a lot more than you think.

brazzy 2y ago
> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.
zelphirkalt 2y ago
The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.
- lelanthran 2y ago
  > The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service. That's still a business decision. Customers don't vote with their feet based on what tech stack the business chose, they vote based on a range of other factors, few, if none, of which are related to 3m of downtime. There are little to no services I know off that would lose customers over 3m of downtime per week. IOW, 3m of downtime is mostly an imaginary problem.
  - zelphirkalt 2y ago
    That's really a too broad generalization. Services that people might leave, because of downtime are for example a git hoster, or a password manager. When people cannot push their commits and this happens multiple times, they may leave for another git hoster. I have seen this very example when gitlab was less stable and often unreachable for a few minutes. When people need some credentials, but cannot reach their online password manager, they cannot work. They cannot trust that service to be available in critical moments. Not being able to access your credentials leaves a very bad impression. Some will look for more reliable ways of storing their credentials.
    - Bjartr 2y ago
      The user experience of "often unreachable" means way more than 3m per week in practice.
    - skydhash 2y ago
      Why does a password manager needs to be online. I understand the need for synchronization, but being exclusively online is a very bad decision. And git synchronization is basically ssh, and if you mess that up on a regular basis, you have no job being in business in the first place. These are examples, but there's a few things that do not need to be online unless your computer is a thin client or you don't trust it at all.
- scott_w 2y ago
  >> What is the benefit to removing 3 minutes of downtime? > The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service. What do you think the business is doing when it evaluates what 3 minutes are worth?
ClumsyPilot 2y ago
> For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k This is a very simple distinction and I am not sure why is it not understood For some reason people design public apps the same as internal apps The largest companies employ circa 1 million people - that’s Walmart, Amazon, etc. most giants, like Shell, etc. companies have ~ 100k tops. That can be handled by 1 beefy server. Successful consumer facing apps are hundred millions to billion. It’s 3 orders of magnitude difference I have seen a company with 5k employees invest into mega-scalable microservice event driven architecture and I was was thinking - I hope they realise what they are doing and it’s just CV-driven development
lmm 2y ago
> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Agreed, but the cost/benefit should be the same at every level in the stack. I get it when people build a single-server system. I'm just baffled that so many people insist they need a load balancer and multiple instances of their application so that there's no single point of failure (which is not free by any means), but then run them all off of a single SQL server.

brazzy 2y ago

> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.

zelphirkalt 2y ago

The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.

lelanthran 2y ago
> The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service. That's still a business decision. Customers don't vote with their feet based on what tech stack the business chose, they vote based on a range of other factors, few, if none, of which are related to 3m of downtime. There are little to no services I know off that would lose customers over 3m of downtime per week. IOW, 3m of downtime is mostly an imaginary problem.
- zelphirkalt 2y ago
  That's really a too broad generalization. Services that people might leave, because of downtime are for example a git hoster, or a password manager. When people cannot push their commits and this happens multiple times, they may leave for another git hoster. I have seen this very example when gitlab was less stable and often unreachable for a few minutes. When people need some credentials, but cannot reach their online password manager, they cannot work. They cannot trust that service to be available in critical moments. Not being able to access your credentials leaves a very bad impression. Some will look for more reliable ways of storing their credentials.
  - Bjartr 2y ago
    The user experience of "often unreachable" means way more than 3m per week in practice.
  - skydhash 2y ago
    Why does a password manager needs to be online. I understand the need for synchronization, but being exclusively online is a very bad decision. And git synchronization is basically ssh, and if you mess that up on a regular basis, you have no job being in business in the first place. These are examples, but there's a few things that do not need to be online unless your computer is a thin client or you don't trust it at all.
scott_w 2y ago
>> What is the benefit to removing 3 minutes of downtime? > The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service. What do you think the business is doing when it evaluates what 3 minutes are worth?

ClumsyPilot 2y ago

> For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k This is a very simple distinction and I am not sure why is it not understood For some reason people design public apps the same as internal apps The largest companies employ circa 1 million people - that’s Walmart, Amazon, etc. most giants, like Shell, etc. companies have ~ 100k tops. That can be handled by 1 beefy server. Successful consumer facing apps are hundred millions to billion. It’s 3 orders of magnitude difference I have seen a company with 5k employees invest into mega-scalable microservice event driven architecture and I was was thinking - I hope they realise what they are doing and it’s just CV-driven development

lmm 2y ago

> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Agreed, but the cost/benefit should be the same at every level in the stack. I get it when people build a single-server system. I'm just baffled that so many people insist they need a load balancer and multiple instances of their application so that there's no single point of failure (which is not free by any means), but then run them all off of a single SQL server.