4 ms·

TLDR: They still don't know why this is happenning.

rvz 5y ago
Oh dear. So other than those double outages, I should expect at the very least that GitHub should have at least one outage every month, not 3 or 5 in one month? I don't think we are going to see GitHub be up for a full month without an incident anytime soon and I guess my entire comment chain [0] on the whole situation has aged for two straight years in a row, especially yesterday's one from [1]: >> Until the next time GitHub goes down again (hopefully that won't be in another month's time). *Goes down the very next day* That says it all really. Lets reset the counter and try this again. [0] https://news.ycombinator.com/item?id=30779275 https://news.ycombinator.com/item?id=30779275 [1] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
throwdbaaway 5y ago
They mentioned nothing about CPU nor disk I/O, with the immediate cause being "reaching maximum number of connections", i.e. the following error from vitess vttablet: ResourceExhausted desc = transaction pool connection limit exceeded My unsolicited advice: this looks like a classic case of https://bugs.mysql.com/bug.php?id=53825 https://bugs.mysql.com/bug.php?id=53825, which remains unfixed after 12 years. Here's the solution space: * Oracle added NOWAIT and SKIP LOCKED to MySQL 8.0, and called it a day. This is a bad solution because it just changes from one extreme (unbounded queue) to another (no queue). * AliSQL added Statement Queue via hint (their older implementation extented the UPDATE syntax instead), which should work great, but do require modifying the write queries: https://partners-intl.aliyun.com/help/en/doc-detail/144127.htm https://partners-intl.aliyun.com/help/en/doc-detail/144127.h... * At $prevjob we added block-level queue within InnoDB, based on an older experimental patch from AliSQL which they didn't actually bring into production. No app change required. * That bug was reported by Facebook, but there is nothing relevant in their MySQL fork. They probably outsource the lock queuing logic to their Memcached layer. The problem with this bug is that, without a correct analysis, the knee-jerk reactions are typically: * adding more CPUs * throwing in even faster disks * bolting on a sharding layer like vitess * blaming your DBaaS providers (well, they are to be blamed if they intend to stick with "vanilla Oracle MySQL") Question to Github: I see you running pprof on vttablet, but have you run perf on mysqld?
- throwdbaaway 5y ago
  I gathered from another comment here that this mysql1 database is still fronted by ProxySQL? In that case, the advice remains the same: don't bother with the proxy layer, focus on mysqld instead.
elondaits 5y ago
I’m surprised when these increasingly frequent outages at large internet companies happen I don’t see tons of suggestions that Russia (or China, or North Korea) is behind them. Is all cyber warfare completely underground and invisible?
- jpgvm 5y ago
  Cyberattacks look every different to degraded query performance. You could in theory pull off an attack like that but it seems sort of dumb because it's not destructive/permanent so unless the timing it really critical it's probably a huge waste of a (likely expensive) exploit chain. If you have deep enough access to degrade performance of the database you likely have deep enough access for exfiltration or other more nefarious activities.
  - elondaits 5y ago
    I sometimes think in that direction as well... but in the case of degraded query performance it could be simple a DOS attack, where they found some combination of operations that brings the db down... and they exploit it because they don't expect it to do anything more than this. If you want your enemy to fear the repercussions of attacking you, you need him to understand the strength you have to fight back. And in the case of cyberwarfare the only way to show strength is by hacking stuff... so I imagine there's some value in smaller not-so-destructive attacks.

rvz 5y ago

Oh dear. So other than those double outages, I should expect at the very least that GitHub should have at least one outage every month, not 3 or 5 in one month? I don't think we are going to see GitHub be up for a full month without an incident anytime soon and I guess my entire comment chain [0] on the whole situation has aged for two straight years in a row, especially yesterday's one from [1]: >> Until the next time GitHub goes down again (hopefully that won't be in another month's time). *Goes down the very next day* That says it all really. Lets reset the counter and try this again. [0] https://news.ycombinator.com/item?id=30779275 https://news.ycombinator.com/item?id=30779275 [1] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821

throwdbaaway 5y ago

They mentioned nothing about CPU nor disk I/O, with the immediate cause being "reaching maximum number of connections", i.e. the following error from vitess vttablet: ResourceExhausted desc = transaction pool connection limit exceeded My unsolicited advice: this looks like a classic case of https://bugs.mysql.com/bug.php?id=53825 https://bugs.mysql.com/bug.php?id=53825, which remains unfixed after 12 years. Here's the solution space: * Oracle added NOWAIT and SKIP LOCKED to MySQL 8.0, and called it a day. This is a bad solution because it just changes from one extreme (unbounded queue) to another (no queue). * AliSQL added Statement Queue via hint (their older implementation extented the UPDATE syntax instead), which should work great, but do require modifying the write queries: https://partners-intl.aliyun.com/help/en/doc-detail/144127.htm https://partners-intl.aliyun.com/help/en/doc-detail/144127.h... * At $prevjob we added block-level queue within InnoDB, based on an older experimental patch from AliSQL which they didn't actually bring into production. No app change required. * That bug was reported by Facebook, but there is nothing relevant in their MySQL fork. They probably outsource the lock queuing logic to their Memcached layer. The problem with this bug is that, without a correct analysis, the knee-jerk reactions are typically: * adding more CPUs * throwing in even faster disks * bolting on a sharding layer like vitess * blaming your DBaaS providers (well, they are to be blamed if they intend to stick with "vanilla Oracle MySQL") Question to Github: I see you running pprof on vttablet, but have you run perf on mysqld?

throwdbaaway 5y ago
I gathered from another comment here that this mysql1 database is still fronted by ProxySQL? In that case, the advice remains the same: don't bother with the proxy layer, focus on mysqld instead.

elondaits 5y ago

I’m surprised when these increasingly frequent outages at large internet companies happen I don’t see tons of suggestions that Russia (or China, or North Korea) is behind them. Is all cyber warfare completely underground and invisible?

jpgvm 5y ago
Cyberattacks look every different to degraded query performance. You could in theory pull off an attack like that but it seems sort of dumb because it's not destructive/permanent so unless the timing it really critical it's probably a huge waste of a (likely expensive) exploit chain. If you have deep enough access to degrade performance of the database you likely have deep enough access for exfiltration or other more nefarious activities.
- elondaits 5y ago
  I sometimes think in that direction as well... but in the case of degraded query performance it could be simple a DOS attack, where they found some combination of operations that brings the db down... and they exploit it because they don't expect it to do anything more than this. If you want your enemy to fear the repercussions of attacking you, you need him to understand the strength you have to fight back. And in the case of cyberwarfare the only way to show strength is by hacking stuff... so I imagine there's some value in smaller not-so-destructive attacks.