4 ms·

Three out of four times MySQL was involved.

mattip 6y ago
If you are running an app that is basically a UI layer over a database, where would you expect significant failures to occur? The users are already debugging possible problems with git locally by simply using it, so UI-database interactions are where all the action happens on products like github.
- KingOfCoders 6y ago
  When I was CTO in some companies we had very different problems, with DBs being only one of them (and thankfully not four hour long outages in two months).
  - aneutron 6y ago
    I might be wrong, but you probably didn't handle as much traffic as Github does. Bugs and outages probability is a asymptotic to normalized traffic. At least intuitively.
    - KingOfCoders 6y ago
      No we didn't luckily but we also didn't have that much money. Biggest problem was 50k+ people trying to reserve a limited amount of stuff at the same time and 5k+ logins/second (at first we made the mistake to write last_login to MySQL ;-)
      - d0100 6y ago
        How did you go about saving login/device data? We are going to do this soon in our app and I'm looking for good use cases and solutions
      - KingOfCoders 6y ago
        We changed to a write behind setup (e.g. with Redis PubSub or any other queue). I think everyone with some experience will use async write behind and not sync writes during a request as we did.
dx034 6y ago
What confuses me is that CPU starvation caused crashes. Sure, too few CPU ressources should cause a slow down but the database process itself shouldn't crash as a result. I haven't dealt with MySQL that much but at least for Postgres, I've never seen 100% load to cause a node to fail.
- Akronymus 6y ago
  I could see a buffer for requests filling up too much so that it causes a no memory crash. But even that seems preventable.
  - dspillett 6y ago
    OOM issues often don't _really_ cause a crash unless you are running with real memory only. Usually long before you are out of total memory you are using so much virtual memory that the kernel is thrashing IO to try keep up with all the page faults. At this point the machine hasn't crashed, but from the outside it might as well have because it can't respond to anything new, even diagnostic intervention, in a timely fashion. CPU overloading, without memory starvation, can have a similar effect, though that doesn't tend to fall off the same efficiency wall as when IO is the issue (unless something has literally kicked off a fork bomb) because of the orders of magnitude differences in latency and work throughput.
    - fulafel 6y ago
      A swap livelock would be worse than a crash, you really don't want a DB backend to start swapping in this kind of application.
      - dspillett 6y ago
        No, but from a diagnostics PoV it can be important to know the difference between failure modes that, unless you are physically near the device so can see drive lights flickering (or hear heads moving if using traditional drives), could otherwise look identical.
bob1029 6y ago
I think the hosted SQL model may be fundamentally flawed when you are trying to do something as complex as what GitHub is doing. You are basically saying "Here, MySQL/Postgres/Oracle/et.al., I trust you with 100% of my replication & persistence logic. Good luck optimizing all that SQL!". I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail. For instance, when going to persist a business object's changes, you can have a policy configured that reflects over the type and determines how many nodes need to be replicated to and which ones should replicate synchronously vs asynchronously based upon it. Log entries may only be async to 2 nodes, but you may decide that any accounting transactions should be replicated to 2 near nodes synchronously, with 2 more far nodes being async. Additionally, you could inspect various business facts in the entity (or session related to the transaction) in order to determine ideal persistence strategy. One powerful example here could be to use a user's zip code to determine the geographically-ideal node to store the primary replica of their account & profile data. This could allow for lower latency access for that specific user.
- m0xte 6y ago
  I'm not sure it even makes sense for smaller platforms any more. The metric of most business success is really growth. Tying yourself to something which is troublesome to grow because it favours doing things that are difficult to scale is an evolutionary dead end. I've spent the last 25 years constantly battling this problem in relational databases. Moore's Law has been pretty kind so far assuming you have the money to buy ridiculous computers and licenses. I am out of touch of what is available in the NoSQL side of things but about 5 years ago there were too many things which had a terrible failure mode somewhere in the stack thus I'm never sure which way to go instead.
  - nkozyra 6y ago
    Boy in 2020 I can't see describing databases as difficult to scale. Expensive, sure, but these are solved problems, and the tradeoffs for going the NoSQL route will likely exceed its perceived benefits. Modern relational databases are great at what they do.
PedroBatista 6y ago
Three out of four times MySQL was involved in the success of products and companies.

If you are running an app that is basically a UI layer over a database, where would you expect significant failures to occur? The users are already debugging possible problems with git locally by simply using it, so UI-database interactions are where all the action happens on products like github.

KingOfCoders 6y ago
When I was CTO in some companies we had very different problems, with DBs being only one of them (and thankfully not four hour long outages in two months).
- aneutron 6y ago
  I might be wrong, but you probably didn't handle as much traffic as Github does. Bugs and outages probability is a asymptotic to normalized traffic. At least intuitively.
  - KingOfCoders 6y ago
    No we didn't luckily but we also didn't have that much money. Biggest problem was 50k+ people trying to reserve a limited amount of stuff at the same time and 5k+ logins/second (at first we made the mistake to write last_login to MySQL ;-)
    - d0100 6y ago
      How did you go about saving login/device data? We are going to do this soon in our app and I'm looking for good use cases and solutions
    - KingOfCoders 6y ago
      We changed to a write behind setup (e.g. with Redis PubSub or any other queue). I think everyone with some experience will use async write behind and not sync writes during a request as we did.

dx034 6y ago

What confuses me is that CPU starvation caused crashes. Sure, too few CPU ressources should cause a slow down but the database process itself shouldn't crash as a result. I haven't dealt with MySQL that much but at least for Postgres, I've never seen 100% load to cause a node to fail.

Akronymus 6y ago
I could see a buffer for requests filling up too much so that it causes a no memory crash. But even that seems preventable.
- dspillett 6y ago
  OOM issues often don't _really_ cause a crash unless you are running with real memory only. Usually long before you are out of total memory you are using so much virtual memory that the kernel is thrashing IO to try keep up with all the page faults. At this point the machine hasn't crashed, but from the outside it might as well have because it can't respond to anything new, even diagnostic intervention, in a timely fashion. CPU overloading, without memory starvation, can have a similar effect, though that doesn't tend to fall off the same efficiency wall as when IO is the issue (unless something has literally kicked off a fork bomb) because of the orders of magnitude differences in latency and work throughput.
  - fulafel 6y ago
    A swap livelock would be worse than a crash, you really don't want a DB backend to start swapping in this kind of application.
    - dspillett 6y ago
      No, but from a diagnostics PoV it can be important to know the difference between failure modes that, unless you are physically near the device so can see drive lights flickering (or hear heads moving if using traditional drives), could otherwise look identical.

bob1029 6y ago

I think the hosted SQL model may be fundamentally flawed when you are trying to do something as complex as what GitHub is doing. You are basically saying "Here, MySQL/Postgres/Oracle/et.al., I trust you with 100% of my replication & persistence logic. Good luck optimizing all that SQL!". I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail. For instance, when going to persist a business object's changes, you can have a policy configured that reflects over the type and determines how many nodes need to be replicated to and which ones should replicate synchronously vs asynchronously based upon it. Log entries may only be async to 2 nodes, but you may decide that any accounting transactions should be replicated to 2 near nodes synchronously, with 2 more far nodes being async. Additionally, you could inspect various business facts in the entity (or session related to the transaction) in order to determine ideal persistence strategy. One powerful example here could be to use a user's zip code to determine the geographically-ideal node to store the primary replica of their account & profile data. This could allow for lower latency access for that specific user.

m0xte 6y ago
I'm not sure it even makes sense for smaller platforms any more. The metric of most business success is really growth. Tying yourself to something which is troublesome to grow because it favours doing things that are difficult to scale is an evolutionary dead end. I've spent the last 25 years constantly battling this problem in relational databases. Moore's Law has been pretty kind so far assuming you have the money to buy ridiculous computers and licenses. I am out of touch of what is available in the NoSQL side of things but about 5 years ago there were too many things which had a terrible failure mode somewhere in the stack thus I'm never sure which way to go instead.
- nkozyra 6y ago
  Boy in 2020 I can't see describing databases as difficult to scale. Expensive, sure, but these are solved problems, and the tradeoffs for going the NoSQL route will likely exceed its perceived benefits. Modern relational databases are great at what they do.

PedroBatista 6y ago

Three out of four times MySQL was involved in the success of products and companies.