9 ms·

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

wrldos 4y ago
This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
- iancmceachern 4y ago
  Exactly, management shouldn't allow this kind of situation to occur by designing the xomoanies processes such that there are checks and balances
- thisarticle 4y ago
  The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
  - deleted 4y ago
    [deleted]
  - lotsofspots 4y ago
    Oh, they take full responsibility, it always says so in the mails they send out. It's just that taking responsibility doesn't appear to actually result in anything happening.
    - parsimo2010 4y ago
      A cyncical take might be that they are saying that they take responsibility (credit) for reducing the monthly payroll expenses. They may also have overhired in the past, but what's in the past was already paid for. The savings next month is how they justify a large paycheck.
    - shapefrog 4y ago
      Macroeconomic changes have made it impossible for me to want to pay you https://news.ycombinator.com/item?id=34515267 https://news.ycombinator.com/item?id=34515267
    - j33zusjuice 4y ago
      Their punishment is in bearing the shame of having been wrong. That’s the price of leadership.
      - randomdata 4y ago
        What shame is there in being wrong? Being wrong is the ideal state, paving a path to gaining an education, which is a source of pride and a benefit.
  - cc81 4y ago
    They only take responsibility for the profit margins. Over hiring affects those but often not significant enough and can be corrected with layoffs.
    - hgsgm 4y ago
      No, they only take responsibility for short-term market cap. Margins and profit don't matter. That's why they chase whatever fad hits the investor class.
  - duckmysick 4y ago
    There's plenty of companies replacing their CEOs. Just today Toyota announced theirs.
    - Octoth0rpe 4y ago
      The CEO of Toyoda is becoming the chairman of their board, that doesn't feel like a CEO being replaced as punishment for poor performance in the way that people are talking about in this thread. But even when CEOs are fully ousted over issues, the golden parachute makes it barely feel like a punishment anyway. I'm having trouble thinking of a case where a CEO actually seemed to be significantly financially impacted by such an event, though maybe FTX will provide an example shortly.
      - hgsgm 4y ago
        Are the golden parachutes bigger (as % of annual comp) than employee severance packages?
        Octoth0rpe 4y ago
        Do you think that honestly matters in the 10s of millions of dollars range? I certainly don't. The problematic parachutes in question are beyond enough for an excessively wealthy standard of living for the rest of their natural lives, even if it's proportionally smaller. Whether or not CEO comp should be as high as it currently is is another question entirely.
  - noisy_boy 4y ago
    I don't see why we reward scale-out/scale-in in the cloud but punish CEOs when they do the same with real people /s
    - hgsgm 4y ago
      How will those poor decommissioned computers get enough bytes to feed themselves?
  - oxfordmale 4y ago
    They are taking responsibility. They are just delegating the consequences to their staff. I suspect this will change soon. Activist investors are already surrounding companies like Salesforce and I can see CEOs being promoted sideways (board member only).
  - logifail 4y ago
    > The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring The list of managers stating that "they were taking responsibility" and then immediately stepping down was always fairly short.
  - itsoktocry 4y ago
    >See: not a single CEO stepping down for over hiring. Wait, what? You think a CEO should step down because their management over-hired a relatively small proportion of employees and had to do some layoffs?
    - hgsgm 4y ago
      It's not relatively small. All the companies are experiencing similar chaos to the NYSE because people in the middle of important operational work suddenly vanished. The people laid off weren't idle like H&R Block tax preparers in May or Target clerks in January. The people laid off and the people not needed were a different set of people, at the time of the layoff.
    - DrBazza 4y ago
      I'll make an extreme comparison: "Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror" If you make some idiotic financial decision near the bottom of the management tree, such as... over hiring, you'll likely lose your job or get demoted. Do it as a CEO, and get a huge bonus. [1] https://en.wikipedia.org/wiki/Jean_Rostand https://en.wikipedia.org/wiki/Jean_Rostand
      - zaroth 4y ago
        But it’s absurd. Companies are not supposed to only ever hire. Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certainly can be done well or poorly.
        DrBazza 4y ago
        It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.
  - wonderwonder 4y ago
    This is because the CEO's core job is to raise stock price. Nothing else. They hired in covid and profits & share price spiked due to the economic state at the time. Now the economic state has changed so they fire employees and the stock goes up. By that metric, the CEO will get a bonus at the end of the year. CEO does not get a bonus for not laying people off. Employees are not humans once you get to the csuite. An employee can be a person but multiple employees are just numbers on a ledger. They just send out "I'm sorry" emails to placate the masses and to get good media, no one really cares if the lower level people are upset. You only count once you get to a certain level.
- cm2187 4y ago
  If you have processes where there is nothing an employee can do to affect the outcome of the company you successfully built a legacy bureaucracy that is waiting to be disrupted.
  - yourapostasy 4y ago
    In this specific case, I don't think that's necessarily the outcome. Our industry has yet to accept a universally-acknowledged equivalent of a lockout/tagout (LOTO) interlock. There is no need for a bureaucracy if we have cryptographically-enforced multisig Shamir secret sharing keys where a LOTO prevents (in this case) a system from spinning up while another system (the backup system apparently in this case) is running. Allow it to be overridden by a sufficiently senior manager or say a sufficient number of lower-seniority managers, which leaves an audit trail. Integrate with a change management, notification, secrets storage infrastructures, and infrastructure as code, and it encodes these infrastructure dependencies into code, and can be queried to auto-construct change interlock sequences for a particular desired state. Of course, once you take advantage of such a representation at scale by deploying tremendously more complex infrastructures, you then have to deal with the dependency network meta challenge lest you inadvertently fall into dependency hell. While towards there lies NP-hard problems, they're still computable to a reasonable degree and I dare say a more robust situation than doing it all by hand like we do today. The real challenge is the vast majority of devops staff today would really dislike reasoning about such a representation when it blows up in their faces, and I can't blame them for that kind of reaction.
    - quantgenius 4y ago
      It's very easy to talk about completely automated systems and LOTO and you need these when you have under-skilled staff. The NYSE likely does NOT have under trained staff. If you have LOTO systems etc, what do you do when a sensor fails and you can't figure out why your method for checking whether the other system is running incorrectly thinks it is. Do you allow the stock market to simply not open? What if multiple sensors fail or it's an ambiguous situation like say you are deciding whether or not to fail over a power circuit and it's a brownout but not a complete power failure? What if there is a systemic problem and it's likely the backup power source is going to brown out too? At some point you need highly skilled individuals, like say trained airline pilots flying a plane who have the authority to override systems immediately without having to jump through hoops. This is especially true for mission critical systems. Many of the mission critical systems we rely on are NOT built on the cloud, i.e. other people's computers because you want to be really careful about what hardware you are using, precisely how your data center is setup and want to make sure things like a noisy neighbor do not impact you. Like it or not, these highly trained individuals are going to make mistakes every now and then. A failure like this once every decade or so really isn't so bad. The individual who made this error is likely not a "grunt". I suspect the individual in question will not necessarily suffer any major consequences as a result of this unless it wasn't a mistake but a flagrant disregard for the rules like say bringing a bottle of water into a data center that then spilled or something. Have you built a mission critical, distributed system that hasn't failed for 10 years? It's a lot harder than it looks. That's how often the NYSE has a problem like this, about once a decade. A lot of things that work in theory, don't work for the edge cases and things that lead to problems once a decade or so are extreme edge cases. In the grand scheme of things a mucked up opening auction is a minor problem and anyone who did not take the precaution of sending a limit order and sent a market on open order despite it being standard practice to essentially always use limits and go hurt badly will be made whole.
- gonzo41 4y ago
  Everyone makes mistakes. HOWEVER, Mordern leaders get to the heads of large organizations by never making mistakes, by blaming the little guy, or coworker and hustling up. They'll only fix this because they have too. There isn't a problem until it happens.
- JumpCrisscross 4y ago
  > individual's mistake can take out your business It didn’t.
NoboruWataya 4y ago
They're not mutually exclusive. A staffer leaving a backup system running may well have been the proximate cause of the issue but, if true, it was also likely a management/system issue as you say. The article is a bit strange in that it doesn't attribute the fact in the headline to any source. I don't see anything from the NYSE saying "it's all that guy's fault". On the contrary it says: > [NYSE execs] plan to examine the platform’s procedures and management, potentially reworking rules to be more flexible and provide further protections. Sounds like they know it's a management issue. The headline probably focuses on the staffer leaving the backup system running simply because it's a better headline.
- zmmmmm 4y ago
  yes ... if an organisation has a critical process where a single human making a mistake can cost it $millions then it has a management / process level issue not an issue with the human. Humans make mistakes. Apart from other obvious issues with it, creating a context where individual mistakes lead to horrific outcomes will create a toxic and horrifically stressful workplace - I would actively avoid working in a situation like that myself.
blippitybleep 4y ago
Sounds like many places I’ve worked. I think most devs have had a job like that.
bigpeopleareold 4y ago
I had a project that was really important once. I made a tiny mistake that had a big consequence - a couple of hours of potential lost revenues from our customers. I fixed my mistake with both my boss and CEO nearby. I said after I pushed the fix that I really need more resources around it. That little light of "yeah, this is important" that should have flickered didn't. :) I will not be surprised if nothing gets fixed with the issue at NYSE.
- smcin 4y ago
  But how much actual lost customer revenue? Also, did the customer even notice or not? You're reminding me of the difference between engineers and non-technical managers; to many of the latter something's only a problem if/when the customer or senior mgmt are on the phone complaining about it. Until then it's all naysaying engineers being too pessimistic about process and risk.
  - bigpeopleareold 4y ago
    No one gave me a figure, but the product going down does have a direct impact on customer revenues. So, while no one actually gave me direct numbers, it was felt quite a bit. The most notable thing from the incident was where certain account managers in Japan had to do formal mea culpas because of this mistake. So, in other words, they were on calls, but I got back just "you can't have this happen again - do something." Could someone else at least do code review? I was depressed that I got a lot of "well, we don't know the code base" - the issue was lost in the shuffle also with management. So, I just grew more pessimistic about process and risk. It's not healthy and I would not bottle it up now, but I did, and that was my mistake. I ended up hating that project, but at least I got to train someone else to work on it (who has a team around him.)
- 1980phipsi 4y ago
  If you were to ask what is the probability that this specific error happens again, I would think it would be pretty low. Probably lower than a week ago. If you were to ask what is the probability that some significant, costly error happens again, I don't think the probability is that much lower than a week ago.
dsfyu404ed 4y ago
> These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. At some point you need to strike a balance between freedom/flexibility and stupid proofing. HN goes real hard on the "people are idiots and we should design things that no matter what buttons get mashed it all works out fine" side of things but in the financial world the balance is struck a little further on the "train our employees to not be idiots" side of things. Furthermore, it's usually better optics to blame things on people because people can easily and cheaply alter their behavior cheaply (per incremental change). If you blame the outage on systems it raises questions of when it will be fixed and how much $$. As an aside, it was almost certainly not individual error. At places like NYSE you pretty much always have 2-3 people who should be in a position to catch a mistake like this.
- mcherm 4y ago
  > it was almost certainly not individual error. At places like NYSE you pretty much always have 2-3 people who should be in a position to catch a mistake like this. That's exactly the point that is being made here. Either the message being put out by the NYSE claiming this was an error by one individual is true -- in which case, NYSE leadership is to blame for setting up a process that allows catastrophic consequences for a single individual's error, OR the message being put out by the NYSE is a fabrication designed to redirect blame at some scapegoat, in which case NYSE leadership is to blame for putting out a false or misleading statement. [Edit: It seems I misunderstood -- attributing this to an individual was done by reporters and rumors, not by a formal statement from NYSE.]
credit_guy 4y ago
It does not sound like cover up to me. It was simply the explanation of what happened. I didn't get any hint that the said "staffer" will be fired or otherwise punished. Is there a problem with the system that did not have enough safeguards to let this happen. For sure, but then no system is perfect. This glitch does not happen every day. From memory, I remember a NASDAQ glitch at Facebook's IPO. Let's say there are 2 or 3 glitches like that for major exchanges in one decade. How can you design a system that prevents bugs that show up once a decade?
GuB-42 4y ago
We can always blame management, since they make decisions, including hiring, we can always trace back problems to management. But it is as unhelpful as blaming the grunts. Management has the job of making the company profitable, if they don't employees won't get paid, investors will lose money, and ultimately the company will fail and customers won't get service. And just like the "grunts", they are not perfect, sometimes, they make mistakes, sometimes, they have to take chances. In fact, blaming anyone is unhelpful unless baltent misconduct is the problem, and I don't think it is the case here. As always, shared responsibilities. I just wished a different wording, something like "NYSE Tuesday opening mayhem traced to a backup system not properly shut down". Leave the "staffer" part to the technical report. It is useful information for investing the problem and fixing what needs to be fixed, but it is inconsiderate for a press release.
- Waterluvian 4y ago
  The difference is that one asserts authority over the other.
reaperducer 4y ago
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. If you're going to move the blame up the food chain, might as well blame the shareholders for giving the company money and choosing to keep the upper management in place.
PragmaticPulp 4y ago
> These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. At the end of the day, the engineers are responsible for the engineering. Managers are responsible for managing. Shifting all responsibility for execution issues on to management can give warm fuzzies, but in reality managers aren’t all powerful in shaping execution by engineers. Companies that put all blame on managers when things fail are inevitably encumbered with excessive micromanagement, as the managers are effectively saddled with responsibility for execution as well. The article was purely anonymous. I don’t think it’s fair to assume they’re jumping to blame or fire individual engineers.
- willcipriano 4y ago
  Do engineers have authority over engineering? Can they overrule management on engineering issues? Whoever takes the authority gets the blame.
hinkley 4y ago
We have a Slack channel where we are expected to announce all of our production changes. Some updates are highly regimented, but a couple of the more operational teams have discretion to deploy things outside of that process, and most teams can flip feature toggles whenever they want. Point is that sometimes people will comment, or even veto changes. We have a major customer visiting today, or the sales team is at a conference. Don’t touch anything or you might break something.
baby 4y ago
It's also true that in systems like this there exist many single points of failure. There's a reason decentralized systems are seeing a rebirth.
NovemberWhiskey 4y ago
Look it's totally OK to recognize that a human action was the trigger for an incident - i.e. the causal chain for this specific incident started there. That's not the same thing as saying the human action was the root cause, and I hope by-and-large any kind of baseline competent engineering organization has gotten to that level of thinking by now.

wrldos 4y ago

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

iancmceachern 4y ago
Exactly, management shouldn't allow this kind of situation to occur by designing the xomoanies processes such that there are checks and balances
thisarticle 4y ago
The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
- deleted 4y ago
  [deleted]
- lotsofspots 4y ago
  Oh, they take full responsibility, it always says so in the mails they send out. It's just that taking responsibility doesn't appear to actually result in anything happening.
  - parsimo2010 4y ago
    A cyncical take might be that they are saying that they take responsibility (credit) for reducing the monthly payroll expenses. They may also have overhired in the past, but what's in the past was already paid for. The savings next month is how they justify a large paycheck.
  - shapefrog 4y ago
    Macroeconomic changes have made it impossible for me to want to pay you https://news.ycombinator.com/item?id=34515267 https://news.ycombinator.com/item?id=34515267
  - j33zusjuice 4y ago
    Their punishment is in bearing the shame of having been wrong. That’s the price of leadership.
    - randomdata 4y ago
      What shame is there in being wrong? Being wrong is the ideal state, paving a path to gaining an education, which is a source of pride and a benefit.
- cc81 4y ago
  They only take responsibility for the profit margins. Over hiring affects those but often not significant enough and can be corrected with layoffs.
  - hgsgm 4y ago
    No, they only take responsibility for short-term market cap. Margins and profit don't matter. That's why they chase whatever fad hits the investor class.
- duckmysick 4y ago
  There's plenty of companies replacing their CEOs. Just today Toyota announced theirs.
  - Octoth0rpe 4y ago
    The CEO of Toyoda is becoming the chairman of their board, that doesn't feel like a CEO being replaced as punishment for poor performance in the way that people are talking about in this thread. But even when CEOs are fully ousted over issues, the golden parachute makes it barely feel like a punishment anyway. I'm having trouble thinking of a case where a CEO actually seemed to be significantly financially impacted by such an event, though maybe FTX will provide an example shortly.
    - hgsgm 4y ago
      Are the golden parachutes bigger (as % of annual comp) than employee severance packages?
      - Octoth0rpe 4y ago
        Do you think that honestly matters in the 10s of millions of dollars range? I certainly don't. The problematic parachutes in question are beyond enough for an excessively wealthy standard of living for the rest of their natural lives, even if it's proportionally smaller. Whether or not CEO comp should be as high as it currently is is another question entirely.
- noisy_boy 4y ago
  I don't see why we reward scale-out/scale-in in the cloud but punish CEOs when they do the same with real people /s
  - hgsgm 4y ago
    How will those poor decommissioned computers get enough bytes to feed themselves?
- oxfordmale 4y ago
  They are taking responsibility. They are just delegating the consequences to their staff. I suspect this will change soon. Activist investors are already surrounding companies like Salesforce and I can see CEOs being promoted sideways (board member only).
- logifail 4y ago
  > The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring The list of managers stating that "they were taking responsibility" and then immediately stepping down was always fairly short.
- itsoktocry 4y ago
  >See: not a single CEO stepping down for over hiring. Wait, what? You think a CEO should step down because their management over-hired a relatively small proportion of employees and had to do some layoffs?
  - hgsgm 4y ago
    It's not relatively small. All the companies are experiencing similar chaos to the NYSE because people in the middle of important operational work suddenly vanished. The people laid off weren't idle like H&R Block tax preparers in May or Target clerks in January. The people laid off and the people not needed were a different set of people, at the time of the layoff.
  - DrBazza 4y ago
    I'll make an extreme comparison: "Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror" If you make some idiotic financial decision near the bottom of the management tree, such as... over hiring, you'll likely lose your job or get demoted. Do it as a CEO, and get a huge bonus. [1] https://en.wikipedia.org/wiki/Jean_Rostand https://en.wikipedia.org/wiki/Jean_Rostand
    - zaroth 4y ago
      But it’s absurd. Companies are not supposed to only ever hire. Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certainly can be done well or poorly.
      - DrBazza 4y ago
        It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.
- wonderwonder 4y ago
  This is because the CEO's core job is to raise stock price. Nothing else. They hired in covid and profits & share price spiked due to the economic state at the time. Now the economic state has changed so they fire employees and the stock goes up. By that metric, the CEO will get a bonus at the end of the year. CEO does not get a bonus for not laying people off. Employees are not humans once you get to the csuite. An employee can be a person but multiple employees are just numbers on a ledger. They just send out "I'm sorry" emails to placate the masses and to get good media, no one really cares if the lower level people are upset. You only count once you get to a certain level.
cm2187 4y ago
If you have processes where there is nothing an employee can do to affect the outcome of the company you successfully built a legacy bureaucracy that is waiting to be disrupted.
- yourapostasy 4y ago
  In this specific case, I don't think that's necessarily the outcome. Our industry has yet to accept a universally-acknowledged equivalent of a lockout/tagout (LOTO) interlock. There is no need for a bureaucracy if we have cryptographically-enforced multisig Shamir secret sharing keys where a LOTO prevents (in this case) a system from spinning up while another system (the backup system apparently in this case) is running. Allow it to be overridden by a sufficiently senior manager or say a sufficient number of lower-seniority managers, which leaves an audit trail. Integrate with a change management, notification, secrets storage infrastructures, and infrastructure as code, and it encodes these infrastructure dependencies into code, and can be queried to auto-construct change interlock sequences for a particular desired state. Of course, once you take advantage of such a representation at scale by deploying tremendously more complex infrastructures, you then have to deal with the dependency network meta challenge lest you inadvertently fall into dependency hell. While towards there lies NP-hard problems, they're still computable to a reasonable degree and I dare say a more robust situation than doing it all by hand like we do today. The real challenge is the vast majority of devops staff today would really dislike reasoning about such a representation when it blows up in their faces, and I can't blame them for that kind of reaction.
  - quantgenius 4y ago
    It's very easy to talk about completely automated systems and LOTO and you need these when you have under-skilled staff. The NYSE likely does NOT have under trained staff. If you have LOTO systems etc, what do you do when a sensor fails and you can't figure out why your method for checking whether the other system is running incorrectly thinks it is. Do you allow the stock market to simply not open? What if multiple sensors fail or it's an ambiguous situation like say you are deciding whether or not to fail over a power circuit and it's a brownout but not a complete power failure? What if there is a systemic problem and it's likely the backup power source is going to brown out too? At some point you need highly skilled individuals, like say trained airline pilots flying a plane who have the authority to override systems immediately without having to jump through hoops. This is especially true for mission critical systems. Many of the mission critical systems we rely on are NOT built on the cloud, i.e. other people's computers because you want to be really careful about what hardware you are using, precisely how your data center is setup and want to make sure things like a noisy neighbor do not impact you. Like it or not, these highly trained individuals are going to make mistakes every now and then. A failure like this once every decade or so really isn't so bad. The individual who made this error is likely not a "grunt". I suspect the individual in question will not necessarily suffer any major consequences as a result of this unless it wasn't a mistake but a flagrant disregard for the rules like say bringing a bottle of water into a data center that then spilled or something. Have you built a mission critical, distributed system that hasn't failed for 10 years? It's a lot harder than it looks. That's how often the NYSE has a problem like this, about once a decade. A lot of things that work in theory, don't work for the edge cases and things that lead to problems once a decade or so are extreme edge cases. In the grand scheme of things a mucked up opening auction is a minor problem and anyone who did not take the precaution of sending a limit order and sent a market on open order despite it being standard practice to essentially always use limits and go hurt badly will be made whole.
gonzo41 4y ago
Everyone makes mistakes. HOWEVER, Mordern leaders get to the heads of large organizations by never making mistakes, by blaming the little guy, or coworker and hustling up. They'll only fix this because they have too. There isn't a problem until it happens.
JumpCrisscross 4y ago
> individual's mistake can take out your business It didn’t.

NoboruWataya 4y ago

They're not mutually exclusive. A staffer leaving a backup system running may well have been the proximate cause of the issue but, if true, it was also likely a management/system issue as you say. The article is a bit strange in that it doesn't attribute the fact in the headline to any source. I don't see anything from the NYSE saying "it's all that guy's fault". On the contrary it says: > [NYSE execs] plan to examine the platform’s procedures and management, potentially reworking rules to be more flexible and provide further protections. Sounds like they know it's a management issue. The headline probably focuses on the staffer leaving the backup system running simply because it's a better headline.

zmmmmm 4y ago
yes ... if an organisation has a critical process where a single human making a mistake can cost it $millions then it has a management / process level issue not an issue with the human. Humans make mistakes. Apart from other obvious issues with it, creating a context where individual mistakes lead to horrific outcomes will create a toxic and horrifically stressful workplace - I would actively avoid working in a situation like that myself.

blippitybleep 4y ago

Sounds like many places I’ve worked. I think most devs have had a job like that.

bigpeopleareold 4y ago

I had a project that was really important once. I made a tiny mistake that had a big consequence - a couple of hours of potential lost revenues from our customers. I fixed my mistake with both my boss and CEO nearby. I said after I pushed the fix that I really need more resources around it. That little light of "yeah, this is important" that should have flickered didn't. :) I will not be surprised if nothing gets fixed with the issue at NYSE.

smcin 4y ago
But how much actual lost customer revenue? Also, did the customer even notice or not? You're reminding me of the difference between engineers and non-technical managers; to many of the latter something's only a problem if/when the customer or senior mgmt are on the phone complaining about it. Until then it's all naysaying engineers being too pessimistic about process and risk.
- bigpeopleareold 4y ago
  No one gave me a figure, but the product going down does have a direct impact on customer revenues. So, while no one actually gave me direct numbers, it was felt quite a bit. The most notable thing from the incident was where certain account managers in Japan had to do formal mea culpas because of this mistake. So, in other words, they were on calls, but I got back just "you can't have this happen again - do something." Could someone else at least do code review? I was depressed that I got a lot of "well, we don't know the code base" - the issue was lost in the shuffle also with management. So, I just grew more pessimistic about process and risk. It's not healthy and I would not bottle it up now, but I did, and that was my mistake. I ended up hating that project, but at least I got to train someone else to work on it (who has a team around him.)
1980phipsi 4y ago
If you were to ask what is the probability that this specific error happens again, I would think it would be pretty low. Probably lower than a week ago. If you were to ask what is the probability that some significant, costly error happens again, I don't think the probability is that much lower than a week ago.

dsfyu404ed 4y ago

> These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. At some point you need to strike a balance between freedom/flexibility and stupid proofing. HN goes real hard on the "people are idiots and we should design things that no matter what buttons get mashed it all works out fine" side of things but in the financial world the balance is struck a little further on the "train our employees to not be idiots" side of things. Furthermore, it's usually better optics to blame things on people because people can easily and cheaply alter their behavior cheaply (per incremental change). If you blame the outage on systems it raises questions of when it will be fixed and how much $$. As an aside, it was almost certainly not individual error. At places like NYSE you pretty much always have 2-3 people who should be in a position to catch a mistake like this.

mcherm 4y ago
> it was almost certainly not individual error. At places like NYSE you pretty much always have 2-3 people who should be in a position to catch a mistake like this. That's exactly the point that is being made here. Either the message being put out by the NYSE claiming this was an error by one individual is true -- in which case, NYSE leadership is to blame for setting up a process that allows catastrophic consequences for a single individual's error, OR the message being put out by the NYSE is a fabrication designed to redirect blame at some scapegoat, in which case NYSE leadership is to blame for putting out a false or misleading statement. [Edit: It seems I misunderstood -- attributing this to an individual was done by reporters and rumors, not by a formal statement from NYSE.]

credit_guy 4y ago

It does not sound like cover up to me. It was simply the explanation of what happened. I didn't get any hint that the said "staffer" will be fired or otherwise punished. Is there a problem with the system that did not have enough safeguards to let this happen. For sure, but then no system is perfect. This glitch does not happen every day. From memory, I remember a NASDAQ glitch at Facebook's IPO. Let's say there are 2 or 3 glitches like that for major exchanges in one decade. How can you design a system that prevents bugs that show up once a decade?

GuB-42 4y ago

We can always blame management, since they make decisions, including hiring, we can always trace back problems to management. But it is as unhelpful as blaming the grunts. Management has the job of making the company profitable, if they don't employees won't get paid, investors will lose money, and ultimately the company will fail and customers won't get service. And just like the "grunts", they are not perfect, sometimes, they make mistakes, sometimes, they have to take chances. In fact, blaming anyone is unhelpful unless baltent misconduct is the problem, and I don't think it is the case here. As always, shared responsibilities. I just wished a different wording, something like "NYSE Tuesday opening mayhem traced to a backup system not properly shut down". Leave the "staffer" part to the technical report. It is useful information for investing the problem and fixing what needs to be fixed, but it is inconsiderate for a press release.

Waterluvian 4y ago
The difference is that one asserts authority over the other.

reaperducer 4y ago

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. If you're going to move the blame up the food chain, might as well blame the shareholders for giving the company money and choosing to keep the upper management in place.

PragmaticPulp 4y ago

> These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. At the end of the day, the engineers are responsible for the engineering. Managers are responsible for managing. Shifting all responsibility for execution issues on to management can give warm fuzzies, but in reality managers aren’t all powerful in shaping execution by engineers. Companies that put all blame on managers when things fail are inevitably encumbered with excessive micromanagement, as the managers are effectively saddled with responsibility for execution as well. The article was purely anonymous. I don’t think it’s fair to assume they’re jumping to blame or fire individual engineers.

willcipriano 4y ago
Do engineers have authority over engineering? Can they overrule management on engineering issues? Whoever takes the authority gets the blame.

hinkley 4y ago

We have a Slack channel where we are expected to announce all of our production changes. Some updates are highly regimented, but a couple of the more operational teams have discretion to deploy things outside of that process, and most teams can flip feature toggles whenever they want. Point is that sometimes people will comment, or even veto changes. We have a major customer visiting today, or the sales team is at a conference. Don’t touch anything or you might break something.

baby 4y ago

It's also true that in systems like this there exist many single points of failure. There's a reason decentralized systems are seeing a rebirth.

NovemberWhiskey 4y ago

Look it's totally OK to recognize that a human action was the trigger for an incident - i.e. the causal chain for this specific incident started there. That's not the same thing as saying the human action was the root cause, and I hope by-and-large any kind of baseline competent engineering organization has gotten to that level of thinking by now.