5 ms·

I disagree. For reference, I work in a big bank and all the high level people are always on call when fecal matter hits the rotating turbines. Often times, ther

I disagree. For reference, I work in a big bank and all the high level people are always on call when fecal matter hits the rotating turbines. Often times, theres an issue with this or that feature in production or we are having an issue with a release and we need an emergency signoff. The process stipulates that a high level person is required to give this emergency signoff, and they often want to know what happened, what are our options, which one we chose so there's usually a lot of time spent chasing down answers or people who can answer those questions.

Jach 7y ago
This sounds like a disorganized workplace to me. The standard thought experiment is: what happens if <supposedly important person> gets hit by a bus and dies? What if they're just in a coma for a few weeks? The response to the first satisfies a sudden "I'm quitting" and the response to the second satisfies a sudden "I'm going on vacation". Business is more forgiving of course, since both quitting and vacation planning typically have a time delay. The simplest response for both is to delegate: "<person y> now has temporary or permanent authority in the decisions <person x> used to have". That means you need to have a person y for each person x and there needs to be enough trust that person y will make decisions that x would approve of (even if sometimes different). If you're a single-person startup, there is no person y, but then these questions aren't really relevant. As the Pinboard FAQ says, the answer to what happens is the bus will be fine.
- bostik 7y ago
  I am going to hazard a guess: you have not been exposed to some of the nuttier regulations within this space. The regulatory demands in financial technology, and especially in those industries where both the regulators as well as majority of the players may not be technically literate, are effectively the anti-thesis of robust and reliable engineering. When the industry regulations demand that for every potentially disruptive action (recovering from an outage is very much potentially disruptive) there has to be a named individual - not even a role, but an individual - to sign off on every single release step... This is why an outage can last for hours. An engineer, or even a lead, is with high probability not allowed to sign off on an out-of-hours deployment. The fix might take 20 minutes to identify, 5 minutes to implement, 15 minutes to test, and 10 minutes to roll out. But until the organisation can get the named [supposedly responsible] person online, the fix must wait, unreleased. And the irony? The wider dysfunction may be imposed by the regulations, but it has been requested by the industry at large. It is the result of the industry players, majority of whom are incapable (read: technologically illiterate) of interpreting prescriptive regulations. So instead of figuring out how to meet expectations and adapt their processes, they have been lobbying for the regulators to come up with highly descriptive playbooks and step-by-step instructions. These sequences then specify rules and requirements that allow technologically incompetent players to comply, even if they do not understand why they are doing the things. And then these regulatory playbooks become the One True Way[tm], effectively forcing dysfunctional practices, but also actively preventing any process improvements. It's cargo-culting taken to the extreme: "do this and you will not be found to be non-compliant, even if the quality is utter shite". The worst part of this all is that the descriptive regulations make every effort to strip engineers of their agency or decision power. Every "modern" best practice is explicitly ruled as non-compliant and hence effectively illegal. I use scare quotes around modern to highlight that these practices are by no means modern and have been known since ancient times. There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward. The finance industry regulations explicitly reject them because these studies assume that engineers can be trusted to know what they are doing. So... to anyone wondering why finance industry jobs are generally thought of as soul-sucking: that's why. They are subject to regulations, expressly designed to deprive individuals of their agency.
  - ddelt 7y ago
    Everyone who has not worked in the heavily regulated finance industry should read the above post, because this person speaks the truth.
    - perl4ever 7y ago
      As someone who has never worked in the finance industry, I feel like one shouldn't lose sight of the fact that the point of it all is to not have millions of dollars just vanish at random as often happens with Bitcoin. Making software engineers happy is not the point.
      - bostik 7y ago
        I agree with your sentiment. However, I believe you are missing the forest from the trees. The desired result of the regulations is, first and foremost, accountability. This includes, but is not limited to, audit trails and provenance. But because the regulations have been written to specify exactly One True Way[tm] of reaching an adequate level of accountability, they end up causing nasty second-order effects. As sibling comment by PeterStuer mentions, the unfortunate end result is inventive interpretation to avoid the worst of the disruptions. So from my point of view, the rules put in place to _enforce_ accountability have created perverse incentives to _reduce_ proper accountability. That can not be a healthy situation either in short or long term.
  - mehrdadn 7y ago
    > There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward. Interested in this... any links?
    - bostik 7y ago
      I'll dig out the archive when back at the office, next week. There is plenty of research into iterative/agile development, slightly less into quality, and surprisingly little on accountability. The most valuable sources have been just 5 (yes, five) research papers / reports. Luckily they sport citations, but sadly I haven't poured over more than a fraction of them so far.
0xEFF 7y ago
Emergency sign off is correlated with a broken process in my experience.
watwut 7y ago
Here is original statement I reacted to: "once you're senior enough, you are never 100% on vacation. If enough goes wrong, you will get called back to work." If fecal matter hits the rotating turbines often that you are never 100% on vacation, then you have fecal hitting around all the time. It is not like seniors would be on vacation 50% time of year. > Often times, theres an issue with this or that feature in production You should not have often issues with features in production. You should have them once in a while, rarely. The situation that seniors are always desperately needed to fix crisis during their vacation requires workplace that is having crisis pretty much all the time. Capable management does not generate projects that are in crisis all the time.

Jach 7y ago

This sounds like a disorganized workplace to me. The standard thought experiment is: what happens if <supposedly important person> gets hit by a bus and dies? What if they're just in a coma for a few weeks? The response to the first satisfies a sudden "I'm quitting" and the response to the second satisfies a sudden "I'm going on vacation". Business is more forgiving of course, since both quitting and vacation planning typically have a time delay. The simplest response for both is to delegate: "<person y> now has temporary or permanent authority in the decisions <person x> used to have". That means you need to have a person y for each person x and there needs to be enough trust that person y will make decisions that x would approve of (even if sometimes different). If you're a single-person startup, there is no person y, but then these questions aren't really relevant. As the Pinboard FAQ says, the answer to what happens is the bus will be fine.

bostik 7y ago
I am going to hazard a guess: you have not been exposed to some of the nuttier regulations within this space. The regulatory demands in financial technology, and especially in those industries where both the regulators as well as majority of the players may not be technically literate, are effectively the anti-thesis of robust and reliable engineering. When the industry regulations demand that for every potentially disruptive action (recovering from an outage is very much potentially disruptive) there has to be a named individual - not even a role, but an individual - to sign off on every single release step... This is why an outage can last for hours. An engineer, or even a lead, is with high probability not allowed to sign off on an out-of-hours deployment. The fix might take 20 minutes to identify, 5 minutes to implement, 15 minutes to test, and 10 minutes to roll out. But until the organisation can get the named [supposedly responsible] person online, the fix must wait, unreleased. And the irony? The wider dysfunction may be imposed by the regulations, but it has been requested by the industry at large. It is the result of the industry players, majority of whom are incapable (read: technologically illiterate) of interpreting prescriptive regulations. So instead of figuring out how to meet expectations and adapt their processes, they have been lobbying for the regulators to come up with highly descriptive playbooks and step-by-step instructions. These sequences then specify rules and requirements that allow technologically incompetent players to comply, even if they do not understand why they are doing the things. And then these regulatory playbooks become the One True Way[tm], effectively forcing dysfunctional practices, but also actively preventing any process improvements. It's cargo-culting taken to the extreme: "do this and you will not be found to be non-compliant, even if the quality is utter shite". The worst part of this all is that the descriptive regulations make every effort to strip engineers of their agency or decision power. Every "modern" best practice is explicitly ruled as non-compliant and hence effectively illegal. I use scare quotes around modern to highlight that these practices are by no means modern and have been known since ancient times. There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward. The finance industry regulations explicitly reject them because these studies assume that engineers can be trusted to know what they are doing. So... to anyone wondering why finance industry jobs are generally thought of as soul-sucking: that's why. They are subject to regulations, expressly designed to deprive individuals of their agency.
- ddelt 7y ago
  Everyone who has not worked in the heavily regulated finance industry should read the above post, because this person speaks the truth.
  - perl4ever 7y ago
    As someone who has never worked in the finance industry, I feel like one shouldn't lose sight of the fact that the point of it all is to not have millions of dollars just vanish at random as often happens with Bitcoin. Making software engineers happy is not the point.
    - bostik 7y ago
      I agree with your sentiment. However, I believe you are missing the forest from the trees. The desired result of the regulations is, first and foremost, accountability. This includes, but is not limited to, audit trails and provenance. But because the regulations have been written to specify exactly One True Way[tm] of reaching an adequate level of accountability, they end up causing nasty second-order effects. As sibling comment by PeterStuer mentions, the unfortunate end result is inventive interpretation to avoid the worst of the disruptions. So from my point of view, the rules put in place to _enforce_ accountability have created perverse incentives to _reduce_ proper accountability. That can not be a healthy situation either in short or long term.
- mehrdadn 7y ago
  > There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward. Interested in this... any links?
  - bostik 7y ago
    I'll dig out the archive when back at the office, next week. There is plenty of research into iterative/agile development, slightly less into quality, and surprisingly little on accountability. The most valuable sources have been just 5 (yes, five) research papers / reports. Luckily they sport citations, but sadly I haven't poured over more than a fraction of them so far.

0xEFF 7y ago

Emergency sign off is correlated with a broken process in my experience.

watwut 7y ago

Here is original statement I reacted to: "once you're senior enough, you are never 100% on vacation. If enough goes wrong, you will get called back to work." If fecal matter hits the rotating turbines often that you are never 100% on vacation, then you have fecal hitting around all the time. It is not like seniors would be on vacation 50% time of year. > Often times, theres an issue with this or that feature in production You should not have often issues with features in production. You should have them once in a while, rarely. The situation that seniors are always desperately needed to fix crisis during their vacation requires workplace that is having crisis pretty much all the time. Capable management does not generate projects that are in crisis all the time.