7 ms·

I find it surprising that: - The team was not able to sufficiently diagnose the bad query program, and/or devise emergency mitigations, by 1400 UTC March 17 (2

I find it surprising that: - The team was not able to sufficiently diagnose the bad query program, and/or devise emergency mitigations, by 1400 UTC March 17 (24h after the first incident). - They were not able to reduce the load on the db more quickly (eg; cranking up rate limiting, shutting down async or noncritical services and features, etc). At $prevjob we had the ability to do things like this at the push of a button, and generally would have within 90min of incident onset to help systems heal. Here they did eventually figure out they could throttle webhooks but it took several days of repeated incidents. - They did not see connections creeping up towards their limits and have emergency mitigations prepared in case of spikes like this. - The simple failover described on March 22nd (for example) took almost 3h to complete. - They decided to enable profiling during the approximate time window of their load spikes, without sufficient resources. - They lept to failover their db when the proxy had issues, when it seems the failover can take multiple hours to successfully complete(??). - They did not have sufficient sampled trace/log data to form a hypothesis without this profiling. - New services like Packages and Codespaces rely on the `mysql1` primary db. Certainly I have much empathy for the responding teams – I have scars from responding to some very painful degradations myself – but as a Github customer this does leave me concerned.

thrdbndndn 5y ago
What is "$prevjob"? I thought it's meant to be a stock symbol (by Twitters' convention at least) but I can't find any company with this ticker.
- deleted 5y ago
  [deleted]
- vorpalhex 5y ago
  It's a placeholder for where the commentor last worked without actually using an employers name. It is a play on variable syntax.
- KMag 5y ago
  They're using shell variable replacement syntax to refer to their previous job, without naming their previous employer.
  - thrdbndndn 5y ago
    I feel so dumb now.. for some reason I didn't realize it's prev(ious) job.
    - KMag 5y ago
      To be fair, financial symbology is its own hellish rabbit hole. $prevjob is probably a symbol in someone's pet obscure symbology schema for some stock I've never heard of traded in a country I don't even know exists. Most NASDAQ tickers are 4 upper case letters. Most NYSE tickers are 3 upper case letters. I've never seen a 7 letter lower-case ticker. Tickers aren't unique across exchanges, so you need a pair of <ticker, exchange> to avoid ambiguity, but then you have to deal with shares being fungible across primary and secondary exchanges. CUSIPs aren't memorable and only apply to North American securities. ISINs are even less memorable. CUSIPs and ISINs include Luhn check-digits, which help catch typos. RICs are pretty good, except that they're copyrighted by Reuters, so if you build your systems around them, you may be obligating yourself to buy their market data services. Conversions between different types of symbols are sometimes many-to-one. (For instance composite RICs like AAPL.OQ and their primary exchange RICs like AAPL.O would have the same CUSIP and ISIN.) In most contexts, you can get away with treating tickers case-insensitively, except (last I checked) 1 pair of case-colliding tickers on the Toronto exchange and 3 colliding pairs of tickers on the Bangkok exchange. If you're dealing with financial symbols, implement them as an interface with methods to perform conversions, fungibility checks, etc. Don't pass around financial symbols as strings if you can help it. At my previous job, we had our own internal hierarchical symbology that covered rates, credit, currencies, commodities, equities, and more. Unfortunately, some systems pass these around as underscore-delimited case-insensitive strings. Externally, some symbology conventions distinguish day count conventions case-sensitively with an m or M suffix, which then had to be translated to ^M and M, respectively (and confusingly, since ^ looks a bit like an up arrow but marks the lower-case version). They've luckily replaced most usages of the hierarchical symbols with objects that pretty-print to the older textual representation and have constructors that take the older textual representation. (For more than a decade, I maintained/improved the functional reactive time series subset of a domain-specific financial programming language and integrated globally replicated distributed NoSQL DB for a large global financial firm. By default strings are treated case-insensitively. The case insensitivity helped lazy typists when the system was first used in New York and London, but is now illiquid technical debt. The ability to put spaces in variable names feels odd at first, but you quickly get used to it. The pain of case insensitivity by default never goes away, especially when someone from New York or London modifies symbology code. On the other hand, the lazily evaluated dataflow subset of the language with close database integration is really elegant, and it deals with collections much more elegantly and uniformly than Java/C++/Python/JavaScript.)
- lima 5y ago
  It's a common-ish way of saying "placeholder for an unnamed previous job".
- aeldidi 5y ago
  I believe it’s supposed to be like a shell variable, which are prefixed with “$”.
taywrobel 5y ago
Former tech lead on Packages here (no longer at GitHub). > New services like Packages and Codespaces rely on the `mysql1` primary db. A lot of this is due to that DB handling core data to GitHub (notably organization and repository data), which the integrated nature of the new feature offerings forces them to interact with (syncing permissions from repos for packages, publishing from actions, etc.). The link between repos and codespaces is even more unavoidable. We were careful with Packages to keep as few service dependencies as possible in the critical path, especially for things like the anonymous read path, so serving project packages to users for open source projects or package managers such as homebrew are as insulated as they can be (and I suspect were unaffected by this incident). But at the end of the day, there is some data that is central to most everything at the company, unfortunately.
- rattray 5y ago
  Thanks, that makes sense. I would have imagined that most of what you describe could hit the `mysql1` replica (not primary) but certainly it's imaginable that this would not be possible or wise in all cases. And of course it's worth applauding that I think most read traffic (whether to Packages or other services) worked just fine through these incidents, if I understand correctly.
- foolfoolz 5y ago
  this sounds backwards. i’ve worked at multiple places with the one big legacy mysql db. all of them had blanket rule no new features live in this db. whatever this complicates you would design around it
  - MrStonedOne 5y ago
  - taywrobel 5y ago
    Live in the DB and use data from the DB are two separate things. Packages has its own dedicated MySQL cluster for all package related info. Auth also lives in a separate database. But it still needs to access org or repo information in many use cases.
jasonhansel 5y ago
My guess is that part of the issue could be lock contention creating unexpected interactions between multiple simultaneous transactions. Issues like that can be really hard to pin down since there isn't a single transaction causing the problem; instead the load is the result of idiosyncratic interactions between multiple different transactions.
0xsql 5y ago
[Former GitHub Employee; I do not speak on behalf of the company] > The simple failover described on March 22nd (for example) took almost 3h to complete and > They lept to failover their db when the proxy had issues, when it seems the failover can take multiple hours to successfully complete Our failovers are very fast, take a couple seconds most, and work with near zero down time. I'm sure there were also other issues at play here.
- rattray 5y ago
  Got it – thank you, that makes a lot more sense. But fascinating, I wonder what those other issues were.

thrdbndndn 5y ago

What is "$prevjob"? I thought it's meant to be a stock symbol (by Twitters' convention at least) but I can't find any company with this ticker.

deleted 5y ago
[deleted]
vorpalhex 5y ago
It's a placeholder for where the commentor last worked without actually using an employers name. It is a play on variable syntax.
KMag 5y ago
They're using shell variable replacement syntax to refer to their previous job, without naming their previous employer.
- thrdbndndn 5y ago
  I feel so dumb now.. for some reason I didn't realize it's prev(ious) job.
  - KMag 5y ago
    To be fair, financial symbology is its own hellish rabbit hole. $prevjob is probably a symbol in someone's pet obscure symbology schema for some stock I've never heard of traded in a country I don't even know exists. Most NASDAQ tickers are 4 upper case letters. Most NYSE tickers are 3 upper case letters. I've never seen a 7 letter lower-case ticker. Tickers aren't unique across exchanges, so you need a pair of <ticker, exchange> to avoid ambiguity, but then you have to deal with shares being fungible across primary and secondary exchanges. CUSIPs aren't memorable and only apply to North American securities. ISINs are even less memorable. CUSIPs and ISINs include Luhn check-digits, which help catch typos. RICs are pretty good, except that they're copyrighted by Reuters, so if you build your systems around them, you may be obligating yourself to buy their market data services. Conversions between different types of symbols are sometimes many-to-one. (For instance composite RICs like AAPL.OQ and their primary exchange RICs like AAPL.O would have the same CUSIP and ISIN.) In most contexts, you can get away with treating tickers case-insensitively, except (last I checked) 1 pair of case-colliding tickers on the Toronto exchange and 3 colliding pairs of tickers on the Bangkok exchange. If you're dealing with financial symbols, implement them as an interface with methods to perform conversions, fungibility checks, etc. Don't pass around financial symbols as strings if you can help it. At my previous job, we had our own internal hierarchical symbology that covered rates, credit, currencies, commodities, equities, and more. Unfortunately, some systems pass these around as underscore-delimited case-insensitive strings. Externally, some symbology conventions distinguish day count conventions case-sensitively with an m or M suffix, which then had to be translated to ^M and M, respectively (and confusingly, since ^ looks a bit like an up arrow but marks the lower-case version). They've luckily replaced most usages of the hierarchical symbols with objects that pretty-print to the older textual representation and have constructors that take the older textual representation. (For more than a decade, I maintained/improved the functional reactive time series subset of a domain-specific financial programming language and integrated globally replicated distributed NoSQL DB for a large global financial firm. By default strings are treated case-insensitively. The case insensitivity helped lazy typists when the system was first used in New York and London, but is now illiquid technical debt. The ability to put spaces in variable names feels odd at first, but you quickly get used to it. The pain of case insensitivity by default never goes away, especially when someone from New York or London modifies symbology code. On the other hand, the lazily evaluated dataflow subset of the language with close database integration is really elegant, and it deals with collections much more elegantly and uniformly than Java/C++/Python/JavaScript.)
lima 5y ago
It's a common-ish way of saying "placeholder for an unnamed previous job".
aeldidi 5y ago
I believe it’s supposed to be like a shell variable, which are prefixed with “$”.

taywrobel 5y ago

Former tech lead on Packages here (no longer at GitHub). > New services like Packages and Codespaces rely on the `mysql1` primary db. A lot of this is due to that DB handling core data to GitHub (notably organization and repository data), which the integrated nature of the new feature offerings forces them to interact with (syncing permissions from repos for packages, publishing from actions, etc.). The link between repos and codespaces is even more unavoidable. We were careful with Packages to keep as few service dependencies as possible in the critical path, especially for things like the anonymous read path, so serving project packages to users for open source projects or package managers such as homebrew are as insulated as they can be (and I suspect were unaffected by this incident). But at the end of the day, there is some data that is central to most everything at the company, unfortunately.

rattray 5y ago
Thanks, that makes sense. I would have imagined that most of what you describe could hit the `mysql1` replica (not primary) but certainly it's imaginable that this would not be possible or wise in all cases. And of course it's worth applauding that I think most read traffic (whether to Packages or other services) worked just fine through these incidents, if I understand correctly.
foolfoolz 5y ago
this sounds backwards. i’ve worked at multiple places with the one big legacy mysql db. all of them had blanket rule no new features live in this db. whatever this complicates you would design around it
- MrStonedOne 5y ago
- taywrobel 5y ago
  Live in the DB and use data from the DB are two separate things. Packages has its own dedicated MySQL cluster for all package related info. Auth also lives in a separate database. But it still needs to access org or repo information in many use cases.

jasonhansel 5y ago

My guess is that part of the issue could be lock contention creating unexpected interactions between multiple simultaneous transactions. Issues like that can be really hard to pin down since there isn't a single transaction causing the problem; instead the load is the result of idiosyncratic interactions between multiple different transactions.

0xsql 5y ago

[Former GitHub Employee; I do not speak on behalf of the company] > The simple failover described on March 22nd (for example) took almost 3h to complete and > They lept to failover their db when the proxy had issues, when it seems the failover can take multiple hours to successfully complete Our failovers are very fast, take a couple seconds most, and work with near zero down time. I'm sure there were also other issues at play here.

rattray 5y ago
Got it – thank you, that makes a lot more sense. But fascinating, I wonder what those other issues were.