4 ms·

Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a

Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it? It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact. What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

logicchains 6mo ago
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like? Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes? There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
- otterley 6mo ago
  You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
  - qotgalaxy 6mo ago
    [dead]
  - mattmanser 6mo ago
    Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what? The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day. Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
    - c-hendricks 6mo ago
      > Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders Genuinely could use a refresher here.
    - otterley 6mo ago
      I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.
  - dijit 6mo ago
    I have. You could argue the scales are different, but computers are also faster now. So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning? https://news.ycombinator.com/item?id=47947719 https://news.ycombinator.com/item?id=47947719 https://www.linkedin.com/in/jharasym/ https://www.linkedin.com/in/jharasym/
    - otterley 6mo ago
      > You could argue the scales are different, but computers are also faster now. Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex. Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.
      - dijit 6mo ago
        The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.
        otterley 6mo ago
        You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.
        deleted 6mo ago
        [deleted]
    - Aurornis 6mo ago
      > I have. I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow. GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak. You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.
VirusNewbie 6mo ago
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
- otterley 6mo ago
  Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
  - falcor84 6mo ago
    There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers. I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
    - vablings 6mo ago
      In a SWE job market like this, do you really want to be seen as the "conscientious objector"? There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
      - einsteinx2 6mo ago
        Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.
    - fragmede 6mo ago
      Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.
deleted 6mo ago
[deleted]
turtlebits 6mo ago
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable. Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
- otterley 6mo ago
  Who is “maintain[ing] the service”? The workers, of course!
StableAlkyne 6mo ago
When did OP blame the people involved personally? If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming. MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
ofjcihen 6mo ago
OP didn’t blame the staff. His focus is on the company. Invoking individual workers well-being to defend a billion dollar company is also very strange.
- otterley 6mo ago
  A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.
  - ofjcihen 6mo ago
    If anything your argument against MS’s uniqueness makes OPs case stronger.
    - otterley 6mo ago
      Tell us how.
ebiester 6mo ago
I have all the empathy for people in the world. A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
- otterley 6mo ago
  A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.
  - port11 6mo ago
    A corporation is made of people, but its ethos is the product of decision-making. If a corporation is consistently, say, unethical, is it because they hire only unethical individuals? Or because unethical people somewhere along the chain of command make unethical decisions?
    - otterley 6mo ago
      I'm not exactly sure what you're getting at with this question. It seems to still conflate corporate-level decisions with boots-on-the-ground work. Are you suggesting that whatever decisions their upper-level management makes that you consider unethical irreversibly and irrevocably taints all the difficult and honorable work that their engineers and operations people are performing?
      - port11 6mo ago
        I’m saying their lower-level employees are probably honest, hard-working people like everyone else. But the detachment that comes from a large corporate structure makes the higher-ups decide things that aren’t as honourable. “Corporations are made up of people” is a strange way to excuse the reality that the ‘bad’ things that corporations do are often decided by top management.
estimator7292 6mo ago
If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent. GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page. To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.
- otterley 6mo ago
  Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.
nimih 6mo ago
I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.
- otterley 6mo ago
  I agree 100%! But this important distinction and nuance seems to be lost here.

logicchains 6mo ago

>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like? Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes? There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.

otterley 6mo ago
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
- qotgalaxy 6mo ago
  [dead]
- mattmanser 6mo ago
  Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what? The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day. Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
  - c-hendricks 6mo ago
    > Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders Genuinely could use a refresher here.
  - otterley 6mo ago
    I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.
- dijit 6mo ago
  I have. You could argue the scales are different, but computers are also faster now. So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning? https://news.ycombinator.com/item?id=47947719 https://news.ycombinator.com/item?id=47947719 https://www.linkedin.com/in/jharasym/ https://www.linkedin.com/in/jharasym/
  - otterley 6mo ago
    > You could argue the scales are different, but computers are also faster now. Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex. Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.
    - dijit 6mo ago
      The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.
      - otterley 6mo ago
        You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.
        deleted 6mo ago
        [deleted]
  - Aurornis 6mo ago
    > I have. I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow. GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak. You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.

VirusNewbie 6mo ago

Executives have made a choice to not pay for top talent at Microsoft Azure and Github.

otterley 6mo ago
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
- falcor84 6mo ago
  There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers. I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
  - vablings 6mo ago
    In a SWE job market like this, do you really want to be seen as the "conscientious objector"? There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head
    - einsteinx2 6mo ago
      Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.
  - fragmede 6mo ago
    Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.

deleted 6mo ago

[deleted]

turtlebits 6mo ago

#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable. Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).

otterley 6mo ago
Who is “maintain[ing] the service”? The workers, of course!

StableAlkyne 6mo ago

When did OP blame the people involved personally? If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming. MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.

ofjcihen 6mo ago

OP didn’t blame the staff. His focus is on the company. Invoking individual workers well-being to defend a billion dollar company is also very strange.

otterley 6mo ago
A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.
- ofjcihen 6mo ago
  If anything your argument against MS’s uniqueness makes OPs case stronger.
  - otterley 6mo ago
    Tell us how.

ebiester 6mo ago

I have all the empathy for people in the world. A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.

otterley 6mo ago
A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.
- port11 6mo ago
  A corporation is made of people, but its ethos is the product of decision-making. If a corporation is consistently, say, unethical, is it because they hire only unethical individuals? Or because unethical people somewhere along the chain of command make unethical decisions?
  - otterley 6mo ago
    I'm not exactly sure what you're getting at with this question. It seems to still conflate corporate-level decisions with boots-on-the-ground work. Are you suggesting that whatever decisions their upper-level management makes that you consider unethical irreversibly and irrevocably taints all the difficult and honorable work that their engineers and operations people are performing?
    - port11 6mo ago
      I’m saying their lower-level employees are probably honest, hard-working people like everyone else. But the detachment that comes from a large corporate structure makes the higher-ups decide things that aren’t as honourable. “Corporations are made up of people” is a strange way to excuse the reality that the ‘bad’ things that corporations do are often decided by top management.

estimator7292 6mo ago

If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent. GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page. To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.

otterley 6mo ago
Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.

nimih 6mo ago

I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.

otterley 6mo ago
I agree 100%! But this important distinction and nuance seems to be lost here.