3 ms·

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike

by tyre 2mo ago

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

logicchains 1mo ago
How low would uptime have to get before you'd consider holding the engineers personally responsible for it?
- vrosas 1mo ago
  Agreed. At some point to have to stop hugging your ops team and start firing them.
  - ksenzee 1mo ago
    No, you hire more of them, you give them what they need to do their jobs, and you actually listen to their recommendations. It's not even complicated. It just takes time and money, neither of which companies want to spend on reliability until their customers scream very loud en masse.
  - Freedom2 1mo ago
    Agreed. In fact, keep firing ops until someone gets it done. There's no way your company will build a reputation for not supporting their teams.
- pixelatedindex 1mo ago
  Why are we holding the engineers accountable and not the CEO? The company isn’t one department. Responsibility should bubble up to the top.
  - pydry 1mo ago
    They dont have a CEO ever since their last CEO said that human programming "wasn't going anywhere".
    - pixelatedindex 1mo ago
      Is human programming going anywhere? I mean sure you’re not literally typing the actual lines of code, but we haven’t done assembly since compilers and transpilers exist. Personally, all AI has done is add another layer of abstraction where “pseudo code” can be the source input and out comes the compiled program. You still need to think about the design and maintenance.
hansvm 1mo ago
The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.
- linkregister 1mo ago
  For CI there are a number of drop-in commercially-available options. You can make it a staged migration, first trivially migrating the build runners, followed by the more complicated integration test and deployment runners. The CI harness itself can follow, and finally moving to a code hosting and visualization service is last. The final step is challenging; likely the most difficult part is changing all the code references and imports. Shadowing changes would be straightforward. Training your likely 25-50 engineers to use the new code review UX would likely not take that long. Considering the wasted engineering velocity during Github outages, it's worthwhile to do even a partial migration. Github's action runners have in my experience, been the most fragile part of the platform. Given the ease of moving build and merge queue runners to alternates, it's a no-brainer.
  - idkasam 1mo ago
    [dead]
- firesteelrain 1mo ago
  You are certainly free to attempt to migrate your workloads to your own or leased data center space in your area then buy your own on prem cloud like Dell Private Cloud and deal with all of the maintenance yourself. I don’t think it would amount to $100k/yr and you would be taking on all the risk.
  - dcow 1mo ago
    Self-hosted is not the only alternative...
atherton94027 1mo ago
Please, read between the lines. Github actions has been a mess since 2019 at least. None of the instability is new, before this unprecedented growth (for a service that's supposed to scale horizontally) the excuse du jour was the azure migration, before that it was the high rate of shipping post acquisition. Core parts of the product, like navigating to individual files in a code review, are broken
- misterderpie 1mo ago
  > Core parts of the product, like navigating to individual files in a code review, are broken I think this is a good argument to underline "It's not _just_ the scale". Adding to this, the Github Code Review experience is kind-of broken, the way comments/threads are stacked in the PR overview has not improved, pagination isn't really a thing, and these issues are age old. Hopefully, one day, Github will mature.
  - atherton94027 1mo ago
    It makes you wonder if people review code there given how broken the experience is, it's a core part of the product!
itemize123 1mo ago
not constructive, but also not personal. rude but based on observed facts.
mejutoco 1mo ago
Is it a personal attack if it is not directed to a person?
DANmode 1mo ago
Why do I feel like GitHub, without the Microsoft tax, would handle this huge spike in traffic fine? Can you comment on that?
deleted 1mo ago
[deleted]

by tyre 2mo ago

logicchains 1mo ago

How low would uptime have to get before you'd consider holding the engineers personally responsible for it?

vrosas 1mo ago
Agreed. At some point to have to stop hugging your ops team and start firing them.
- ksenzee 1mo ago
  No, you hire more of them, you give them what they need to do their jobs, and you actually listen to their recommendations. It's not even complicated. It just takes time and money, neither of which companies want to spend on reliability until their customers scream very loud en masse.
- Freedom2 1mo ago
  Agreed. In fact, keep firing ops until someone gets it done. There's no way your company will build a reputation for not supporting their teams.
pixelatedindex 1mo ago
Why are we holding the engineers accountable and not the CEO? The company isn’t one department. Responsibility should bubble up to the top.
- pydry 1mo ago
  They dont have a CEO ever since their last CEO said that human programming "wasn't going anywhere".
  - pixelatedindex 1mo ago
    Is human programming going anywhere? I mean sure you’re not literally typing the actual lines of code, but we haven’t done assembly since compilers and transpilers exist. Personally, all AI has done is add another layer of abstraction where “pseudo code” can be the source input and out comes the compiled program. You still need to think about the design and maintenance.

hansvm 1mo ago

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

linkregister 1mo ago
For CI there are a number of drop-in commercially-available options. You can make it a staged migration, first trivially migrating the build runners, followed by the more complicated integration test and deployment runners. The CI harness itself can follow, and finally moving to a code hosting and visualization service is last. The final step is challenging; likely the most difficult part is changing all the code references and imports. Shadowing changes would be straightforward. Training your likely 25-50 engineers to use the new code review UX would likely not take that long. Considering the wasted engineering velocity during Github outages, it's worthwhile to do even a partial migration. Github's action runners have in my experience, been the most fragile part of the platform. Given the ease of moving build and merge queue runners to alternates, it's a no-brainer.
- idkasam 1mo ago
  [dead]
firesteelrain 1mo ago
You are certainly free to attempt to migrate your workloads to your own or leased data center space in your area then buy your own on prem cloud like Dell Private Cloud and deal with all of the maintenance yourself. I don’t think it would amount to $100k/yr and you would be taking on all the risk.
- dcow 1mo ago
  Self-hosted is not the only alternative...

atherton94027 1mo ago

Please, read between the lines. Github actions has been a mess since 2019 at least. None of the instability is new, before this unprecedented growth (for a service that's supposed to scale horizontally) the excuse du jour was the azure migration, before that it was the high rate of shipping post acquisition. Core parts of the product, like navigating to individual files in a code review, are broken

misterderpie 1mo ago
> Core parts of the product, like navigating to individual files in a code review, are broken I think this is a good argument to underline "It's not _just_ the scale". Adding to this, the Github Code Review experience is kind-of broken, the way comments/threads are stacked in the PR overview has not improved, pagination isn't really a thing, and these issues are age old. Hopefully, one day, Github will mature.
- atherton94027 1mo ago
  It makes you wonder if people review code there given how broken the experience is, it's a core part of the product!

itemize123 1mo ago

not constructive, but also not personal. rude but based on observed facts.

mejutoco 1mo ago

Is it a personal attack if it is not directed to a person?

DANmode 1mo ago

Why do I feel like GitHub, without the Microsoft tax, would handle this huge spike in traffic fine? Can you comment on that?

deleted 1mo ago

[deleted]