3 ms·

I think engineers are partially to blame for these type of situations. I can just imagine the internal scenario, the rep escalated, there's some support ticket

I think engineers are partially to blame for these type of situations. I can just imagine the internal scenario, the rep escalated, there's some support ticket sitting on some engineering teams backlog somewhere. Rep can't do anything since it's all automated, have to wait for the devs to do something about it. The systems probably lack manual overrides, or other such measure. Maybe they were flagged for fraud, but somehow the payload got stuck in some error state somewhere, and now there's an engineer that needs to figure out how to get things moving again. To the engineering team, that's 1 error out of hundreds of thousands of requests per day. This is low severity. I guess my point is that, as an engineer myself, I often feel very disconnected from the users themselves. We see errors and issues as percentage of total traffic. Low number of errors, not important. We don't always stop to think, wait is there an actual person or business that relies on this for their livelihood? This one error, what's the impact to the user that depends on my systems? The same way that big corporations start to see users and employees as just a number, I think so do we often fall into this trap. We're always focused on the scale out, never the quality of service of individual requests. It always seems more important to launch in one more region, than to fix the hand full of errors and corrupted requests in the existing one.

hippich 7y ago
You can't blame engineers implementing the business logic defined by management. Current business logic appears to be something like "remove humans making decisions, implement algos doing the job, remove the ability to question that logic, in any not clear case pick safest for amazon option".
londons_explore 7y ago
A better way to look at this is to consider a webmail service. There are 20 engineers on the core dev team. There are 1 Billion customers. 99.9% of customers love the service. 0.1% fall into some bug, and need customer support to sort it out. 0.1% of those complaints are unusual and need an engineer to sort it by writing some code. Thats 1000 issues that require one of those 20 engineers. 50 each. It might take a day per issue, since these are the in-depth, real corner cases. Some will require re-architecting things to fix. You are the project manager of this webmail system. Do you spend 2 months of the entire teams productivity to help 0.0001% of the customer base, or do you ignore them and build some features for the other 99.9999%?
- hellotomyrars 7y ago
  How many of the users pay the company $5,000/mo for an account manager? That's my sticking point. If I pay a company to have a human advocate for me at the company, I should actually have a seat at the table instead of being subject to the machinations of a black-box system that nobody can seem to touch or do anything about unless someone high enough sees people on social media shaming them.
- didibus 7y ago
  I know that this is how the common thinking goes, but this is also the same thinking that those "big bad corporations" use, and that the "big bad government" uses, etc. My point is that we're no better. Do we stop and think... Hum, what's happening to that 0.0001% Did they just miss some important email from the USCIS about some mandatory thing they needed to do? Will that get them deported? What if they missed a memo from their boss? Etc. And in the case of the article here, does it mean that a few dozens of people haven't been paid in a few weeks? By reducing everything to some number, I feel it disconnects us and trivializes things. I don't know that I have an answer. But I find that it is interesting how I think everyone in a company can somehow wash their hands on issues the company is causing by adopting this kind of mentality. Well, given the constraints we had, we went for the big fish only. It's almost some kind of oppression by the majority. Majority of users get their new features while a minority suffers. I think it's easy to start scape goating left and right: well our team isn't given enough resources, well we missed out on profit targets so had to cut back, well investors are pulling out, well government imposes all these regulations on us, well voters wants us too, etc. So I guess we're all a victim of the system and no one is to blame or could have done anything about it. Maybe I'm being a bit cynical, but it's something I've been asking myself more. Could I actually do something about it at my work? At least when it comes to engineering systems without such error, maybe just thinking about impact one such failure could have, I can push a little harder, I can consider it a little more.
ImaCake 7y ago
>To the engineering team, that's 1 error out of hundreds of thousands of requests per day. This is low severity. I think ignoring these edge cases is a mistake. In statistics, we are often most interested in the tails of our curves. That is where the interesting stuff happens. P-values disprove null hypothesises, things fall outside confidence intervals, medium importance individuals suffer a failure and cause a PR disaster that diverts millions of dollars in revenue from your company. I wonder if big (or even small) companies could benefit from this "attention to the tails" thinking. It would be the kind of long term thinking and planning that keeps them on top for a long time instead of crumpling at the first sign of disruption. Of course this subscribes too the idea that anyone in the money-side of the company even cares when mostly they are just fighting for power in a zero sum game. I am worried this doesn't make sense, but I hope it provides insight :)
- didibus 7y ago
  Hum, I don't know much about statistics, so I'm not totally following, but this got me interested to read more about this "tails of our curve". I feel common metrics applied to software systems arn't very elaborate. Like I said, you might look at percentage of errors over total. Then you might take a goal like ok, let's try and have it less then 1%. The number chosen is arbitrary and has no rational, could have been 2%, or 0.5 but 1 sounded good right now. Every metric seem mostly linear and arbitrary, nothing fancy. I wonder if applying something more thorough like what is done in research and leveraging better statistic could make things better.
  - ImaCake 7y ago
    Hmm I definitely think using some basic statistical theory could improve your choice of numbers. A bit of stats provides a lot more information from which you can draw more informed decisions. And of course there are lots of results that are highly unintuitive which have real world applications. I think this would fall under "risk engineering" and it is most relevant there. For an example, say you are interested in how frequently a particular function fails. You are especially worried that a first time user will have this function fail on the first attempt and then disengage entirely from your product. So you want to know how likely it is that your function will fail on the first try. You can run an experiment where you call the function repeatedly until it fails. You find it fails on the 10th attempt. You can use knowledge of probability distributions and a trick for calculating the most likely probability given the experiment you conducted to conclude the obvious: that it has a 1/10 chance of failing on the first go. What the long way gets you, is the chance for it failing after x attempts. So a bit of knowledge gets you a lot more information to make informed decisions with. You could always simulate this experiment in a script to estimate these numbers as well, which I think could be useful to someone with a software engineering background but without the time or resources to learn statistics. 0.https://en.wikipedia.org/wiki/Geometric_distribution https://en.wikipedia.org/wiki/Geometric_distribution 1.Probability dist for the example: https://www.wolframalpha.com/input/?i=geometric+distribution+p%3D0.1 https://www.wolframalpha.com/input/?i=geometric+distribution...
woutr_be 7y ago
> I guess my point is that, as an engineer myself, I often feel very disconnected from the users themselves. We see errors and issues as percentage of total traffic. Low number of errors, not important. We don't always stop to think, wait is there an actual person or business that relies on this for their livelihood? This one error, what's the impact to the user that depends on my systems? I feel the same way, in fact, I won't see any production errors until a ticket is assigned to our team. So if the business decides to ignore certain errors or issues, I simply won't know about it. And if I know about it, I won't get a chance to fix them. And even if I fix them, they most likely won't be pushed to production. It's one thing I truly liked when working at startups, you're much closers to users, and it makes you care more. At that time, I had no problem staying late or working weekends to fix an issue a user had experienced. Because when a user experienced a problem, it would most likely also be a problem for me.

hippich 7y ago

You can't blame engineers implementing the business logic defined by management. Current business logic appears to be something like "remove humans making decisions, implement algos doing the job, remove the ability to question that logic, in any not clear case pick safest for amazon option".

londons_explore 7y ago

A better way to look at this is to consider a webmail service. There are 20 engineers on the core dev team. There are 1 Billion customers. 99.9% of customers love the service. 0.1% fall into some bug, and need customer support to sort it out. 0.1% of those complaints are unusual and need an engineer to sort it by writing some code. Thats 1000 issues that require one of those 20 engineers. 50 each. It might take a day per issue, since these are the in-depth, real corner cases. Some will require re-architecting things to fix. You are the project manager of this webmail system. Do you spend 2 months of the entire teams productivity to help 0.0001% of the customer base, or do you ignore them and build some features for the other 99.9999%?

hellotomyrars 7y ago
How many of the users pay the company $5,000/mo for an account manager? That's my sticking point. If I pay a company to have a human advocate for me at the company, I should actually have a seat at the table instead of being subject to the machinations of a black-box system that nobody can seem to touch or do anything about unless someone high enough sees people on social media shaming them.
didibus 7y ago
I know that this is how the common thinking goes, but this is also the same thinking that those "big bad corporations" use, and that the "big bad government" uses, etc. My point is that we're no better. Do we stop and think... Hum, what's happening to that 0.0001% Did they just miss some important email from the USCIS about some mandatory thing they needed to do? Will that get them deported? What if they missed a memo from their boss? Etc. And in the case of the article here, does it mean that a few dozens of people haven't been paid in a few weeks? By reducing everything to some number, I feel it disconnects us and trivializes things. I don't know that I have an answer. But I find that it is interesting how I think everyone in a company can somehow wash their hands on issues the company is causing by adopting this kind of mentality. Well, given the constraints we had, we went for the big fish only. It's almost some kind of oppression by the majority. Majority of users get their new features while a minority suffers. I think it's easy to start scape goating left and right: well our team isn't given enough resources, well we missed out on profit targets so had to cut back, well investors are pulling out, well government imposes all these regulations on us, well voters wants us too, etc. So I guess we're all a victim of the system and no one is to blame or could have done anything about it. Maybe I'm being a bit cynical, but it's something I've been asking myself more. Could I actually do something about it at my work? At least when it comes to engineering systems without such error, maybe just thinking about impact one such failure could have, I can push a little harder, I can consider it a little more.

ImaCake 7y ago

>To the engineering team, that's 1 error out of hundreds of thousands of requests per day. This is low severity. I think ignoring these edge cases is a mistake. In statistics, we are often most interested in the tails of our curves. That is where the interesting stuff happens. P-values disprove null hypothesises, things fall outside confidence intervals, medium importance individuals suffer a failure and cause a PR disaster that diverts millions of dollars in revenue from your company. I wonder if big (or even small) companies could benefit from this "attention to the tails" thinking. It would be the kind of long term thinking and planning that keeps them on top for a long time instead of crumpling at the first sign of disruption. Of course this subscribes too the idea that anyone in the money-side of the company even cares when mostly they are just fighting for power in a zero sum game. I am worried this doesn't make sense, but I hope it provides insight :)

didibus 7y ago
Hum, I don't know much about statistics, so I'm not totally following, but this got me interested to read more about this "tails of our curve". I feel common metrics applied to software systems arn't very elaborate. Like I said, you might look at percentage of errors over total. Then you might take a goal like ok, let's try and have it less then 1%. The number chosen is arbitrary and has no rational, could have been 2%, or 0.5 but 1 sounded good right now. Every metric seem mostly linear and arbitrary, nothing fancy. I wonder if applying something more thorough like what is done in research and leveraging better statistic could make things better.
- ImaCake 7y ago
  Hmm I definitely think using some basic statistical theory could improve your choice of numbers. A bit of stats provides a lot more information from which you can draw more informed decisions. And of course there are lots of results that are highly unintuitive which have real world applications. I think this would fall under "risk engineering" and it is most relevant there. For an example, say you are interested in how frequently a particular function fails. You are especially worried that a first time user will have this function fail on the first attempt and then disengage entirely from your product. So you want to know how likely it is that your function will fail on the first try. You can run an experiment where you call the function repeatedly until it fails. You find it fails on the 10th attempt. You can use knowledge of probability distributions and a trick for calculating the most likely probability given the experiment you conducted to conclude the obvious: that it has a 1/10 chance of failing on the first go. What the long way gets you, is the chance for it failing after x attempts. So a bit of knowledge gets you a lot more information to make informed decisions with. You could always simulate this experiment in a script to estimate these numbers as well, which I think could be useful to someone with a software engineering background but without the time or resources to learn statistics. 0.https://en.wikipedia.org/wiki/Geometric_distribution https://en.wikipedia.org/wiki/Geometric_distribution 1.Probability dist for the example: https://www.wolframalpha.com/input/?i=geometric+distribution+p%3D0.1 https://www.wolframalpha.com/input/?i=geometric+distribution...

woutr_be 7y ago

> I guess my point is that, as an engineer myself, I often feel very disconnected from the users themselves. We see errors and issues as percentage of total traffic. Low number of errors, not important. We don't always stop to think, wait is there an actual person or business that relies on this for their livelihood? This one error, what's the impact to the user that depends on my systems? I feel the same way, in fact, I won't see any production errors until a ticket is assigned to our team. So if the business decides to ignore certain errors or issues, I simply won't know about it. And if I know about it, I won't get a chance to fix them. And even if I fix them, they most likely won't be pushed to production. It's one thing I truly liked when working at startups, you're much closers to users, and it makes you care more. At that time, I had no problem staying late or working weekends to fix an issue a user had experienced. Because when a user experienced a problem, it would most likely also be a problem for me.