4 ms·

At least Kafka can be properly master-master HA. How people ever got away with building massively redundant fault-tolerant applications that were completely dep

by lmm 2y ago

At least Kafka can be properly master-master HA. How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand.

trog 2y ago
I think because it works pretty close to 100 percent of the time with only the most basic maintenance and care (like making sure you don't run out of disk space and keeping up with security updates). You can go amazingly far with this, and adding read only replicas gets you a lot further with little extra effort.
slt2021 2y ago
stack overflow has no problem serving entire planet from just four SQL Servers (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...) There is really nothing wrong with a large vertically scaled up SQL server. You need to be either really really large large scale - or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits
- HideousKojima 2y ago
  >or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits Sadly that's the case at my current job. Zero thought put into table design, zero effort into even formatting our stored procedures in a remotely readable way, zero attempts to cache data on the application side even when it's glaringly obvious. We actually brought in a consultant to diagnose our SQL Server performance issues (I'm sure we paid a small fortune for that) and the DB team and all of the other higher ups capable of actually enforcing change rejected every last one of his suggestions.
- lmm 2y ago
  > There is really nothing wrong with a large vertically scaled up SQL server. Large or small, it's always going to be a single point of failure. All hardware is fallible, especially if we're talking about a commodity box rather than a mainframe or something.
lelanthran 2y ago
> How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand. It works, with a lower cognitive burden than that of horizontally scaling. For the loading concern (i.e. is this enough to handle the load): For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k users typically has fewer than 20k concurrent requests in flight at peak. A cheap VPS running PostgreSQL can easily handle that.[1] For the "if something breaks" concern: Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. The mechanisms to reduce downtime when the single large SQL server shits the bed (failovers, RO followers, whatever) can reduce that downtime to mere minutes. What is the benefit to removing 3 minutes of downtime? $100? $1k? $100k? $1m? The business will have to decide what those 3 minutes are worth, and if that worth exceeds the cost of using something other than a single large SQL server. Until and unless you reach the load and downtime-cost of Google, Amazon, Twitter, FB, Netflix, etc, you're simply prematurely optimising for a scenario that, even in the businesses best-case projections, might never exist. The best thing to do, TBH, is ask the business for their best-case projections and build to handle 90% of that. [1] An expensive VPS running PostgreSQL can handle a lot more than you think.
- brazzy 2y ago
  > Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.
- zelphirkalt 2y ago
  The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.
  - lelanthran 2y ago
hylaride 2y ago
Because people are beholden to costs and it's often out of our hands when to spend money on redundancy (or opportunity costs elsewhere). It's less true today when redundancy is baked into SaS products (like AWS Aurora, where even if you have a single database instance, it's easy to spin up a replacement one if the hardware on the one running fails).
ClumsyPilot 2y ago
Ow yeah, I am looking at that problem right now

by lmm 2y ago

trog 2y ago

I think because it works pretty close to 100 percent of the time with only the most basic maintenance and care (like making sure you don't run out of disk space and keeping up with security updates). You can go amazingly far with this, and adding read only replicas gets you a lot further with little extra effort.

slt2021 2y ago

stack overflow has no problem serving entire planet from just four SQL Servers (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...) There is really nothing wrong with a large vertically scaled up SQL server. You need to be either really really large large scale - or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits

HideousKojima 2y ago
>or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits Sadly that's the case at my current job. Zero thought put into table design, zero effort into even formatting our stored procedures in a remotely readable way, zero attempts to cache data on the application side even when it's glaringly obvious. We actually brought in a consultant to diagnose our SQL Server performance issues (I'm sure we paid a small fortune for that) and the DB team and all of the other higher ups capable of actually enforcing change rejected every last one of his suggestions.
lmm 2y ago
> There is really nothing wrong with a large vertically scaled up SQL server. Large or small, it's always going to be a single point of failure. All hardware is fallible, especially if we're talking about a commodity box rather than a mainframe or something.

lelanthran 2y ago

> How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand. It works, with a lower cognitive burden than that of horizontally scaling. For the loading concern (i.e. is this enough to handle the load): For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k users typically has fewer than 20k concurrent requests in flight at peak. A cheap VPS running PostgreSQL can easily handle that.[1] For the "if something breaks" concern: Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. The mechanisms to reduce downtime when the single large SQL server shits the bed (failovers, RO followers, whatever) can reduce that downtime to mere minutes. What is the benefit to removing 3 minutes of downtime? $100? $1k? $100k? $1m? The business will have to decide what those 3 minutes are worth, and if that worth exceeds the cost of using something other than a single large SQL server. Until and unless you reach the load and downtime-cost of Google, Amazon, Twitter, FB, Netflix, etc, you're simply prematurely optimising for a scenario that, even in the businesses best-case projections, might never exist. The best thing to do, TBH, is ask the business for their best-case projections and build to handle 90% of that. [1] An expensive VPS running PostgreSQL can handle a lot more than you think.

brazzy 2y ago
> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.
zelphirkalt 2y ago
The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.
- lelanthran 2y ago

hylaride 2y ago

Because people are beholden to costs and it's often out of our hands when to spend money on redundancy (or opportunity costs elsewhere). It's less true today when redundancy is baked into SaS products (like AWS Aurora, where even if you have a single database instance, it's easy to spin up a replacement one if the hardware on the one running fails).

ClumsyPilot 2y ago

Ow yeah, I am looking at that problem right now