18 ms·

The danger of hidden functional roles

168 points by henrik_w 5y ago

lrem 5y ago
I wonder what kind of an exciting outage provoked this post :)
- withinboredom 5y ago
  Probably the one where the kid didn't make it to school on time and everyone's day was out of whack.
eru 5y ago
> Now imagine declaring an extended period of time where, in the interest of reducing risk(!), no new code is deployed, and Chaos Monkey is disabled. I remember reading about similar experiment in biology: If you have a petri dish full of bacteria, and you gradually introduce antibiotics, not only will your bacteria population grow resistant to the antibiotics over time, but there metabolism even might become dependent on it. You see, biology is a system as least as complicated as any computing infrastructure we have. (Compare also most organisms these days being reliant on oxygen. Or how humans lost the ability to make vitamin C in their bodies.)
- lucumo 5y ago
  > You see, biology is a system as least as complicated as any computing infrastructure we have. One of my first bosses describes the opposite relationship. We tend to think technology is very rationalized. He once told me that we should think about the system we were building as an organism. Build it, but also play with it and get to know it, because it will have its own personality.
- reuben364 5y ago
  Here is a video on the topic of bacteria in an experiment evolving to use a second source of food: https://www.youtube.com/watch?v=w4sLAQvEH-M https://www.youtube.com/watch?v=w4sLAQvEH-M. I'm not sure if they depend on it being present though.
Okkef 5y ago
When a person leaves a company who has been there for a long time, a similar effect occurs. These people often do a host of small tasks to that noone is really aware of (themselves included; they don't see how it's special or that nobody else knows about it). It can take the organization a long time to recover and realign.
- xyz11 5y ago
  Yes, very true!
- quickthrower2 5y ago
  Why do they keep the tasks secret? I bet because they know they are important but they also know they are fragile in the sense no one else will care enough to do those things. And worse someone else will say don’t do them because they waste time.
  - lucumo 5y ago
    It's probably not so much keeping it a secret, but just not broadcasting everything they do.
  - watwut 5y ago
    They don't do them in secret. Typically they just do them as part of their routine.
  - magicalhippo 5y ago
    During a summer not long ago, a client called with an urgent issue. Some critical messages were not being passed on, leading to a full stop on their end. However I couldn't make heads or tails of what systems this was or how it related to us. After a bit of back and forth, the customer said "ask Glenn, he always fixes this for us". Glenn was of course on vacation and didn't respond to messages... A few hours pass while we make zero progress on this issue and customer increasingly annoyed, when Glenn finally responds to my messages. Turns out that on a different server from where our software is installed, there was a Java-based message broker service running that needed occasional kick in the pants (IIRC stopping, deleting a file, restarting). We're not a Java shop, we don't use message brokers. For years, Glenn had been keeping this service running and updated for our customer, as the customer didn't have any internal resources that was capable of handling this. And nobody else at our shop knew this. Why? Well, he just did what had to be done and moved on to the next issue at hand, as he always did...
moksly 5y ago
This is why documentation matters, and why it can’t just live in a drawer. Who would set up a virtual machine backup system that offers no alerts when it recycles something? I mean, I might do it myself, so I sort of know, but you shouldn’t do it. Similarly I’m currently having the “pleasure” of documenting every function I’m responsible for as I’m leaving the public sector for the private sector, and man oh man, would the excel sheet of functions and directions to where they happen have saved myself from hundreds of hours “searching” or “refreshing” my knowledge on things when I had to revisit them after years of not working on them. It’s also why MBA business process types are valuable in any enterprise sized organisation. Because they tend to both discover and document these hidden functions when they do their work. Of course, 9 out of 10 times nothing comes of their discoveries, because managers are notoriously poor at long term benefits realisation.
- brobdingnagians 5y ago
  I've been thinking about this in regards to things like Idris vs TLA+ as well. I prefer using Idris than TLA+ simply because Idris is continually with you, type checking and part of the overall design. TLA+ can be used to verify a design, but then it is not really a living, breathing part of the end product. Having deduction built into the safety of the language is preferable to only verifying it externally then having it mutate under the next engineer. It is the same with self-documenting code; if the function and variables names are clear with a sensible architecture in place, then it is easier to keep everything in sync and readable even as different engineers take over; rather than having external documentation which must be sought for, possibly discarded, and therefore not used as much as it ought. One of the things I really like about Elixir is being able to put executable tests and examples inside the module documentation. If you run the tests and it fails, you know you need to update the examples in the documentation.
  - simiones 5y ago
    As far as I understand, Idris and dependent typing in general are only useable for small systems today - the largest systems successfully proved using dependent typing are the SEL4 kernel (which is ~10k lines of C) and the CompCert C compilers (which compiles a subset of the C language, and has only a minimalistic optimizer, if I remeber correctly). And these two have been 5-10 year PhD projects.
a_c 5y ago
https://xkcd.com/1172/ https://xkcd.com/1172/
withinboredom 5y ago
heh, my boss went on vacation. We were scrambling to do the technical part of their job. We never knew the boss did so much behind the scenes and we had no idea how difficult or time consuming it was. When they came back, we had so much mad respect for the things they kept out of our day-to-day job.
- orangepanda 5y ago
  Similar with _good_ project managers - you get shielded from so many useless meetings and drama
sbierwagen 5y ago
>I didn’t know that I was functioning as an alarm clock for her! That’s why I was careful to be quiet, and why I didn’t even think to mention to her about the new videochat time. >I suspect this is failure mode is more common than we realize: there is a process inside a system, and over time the process comes to fulfill some unintended, ancillary functional role, and there are people who participate in this process that aren’t even aware of this function. Ah, Hyrum's Law showing up in an unexpected context, yet again. https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
- deleted 5y ago
  [deleted]
- bestouff 5y ago
  Oblig XKCD: https://xkcd.com/1172/ https://xkcd.com/1172/
hsn915 5y ago
It's not so much of a danger. Occasionally disrupting the system can reveal such weaknesses. Once you identify such "fragilities" you can compensate by adding redundancies. For example, add a second alarm clock at 6:30 and 7:00. When it comes to programming, this is one of the good reasons why simplification matters. Systems that are complicated to the point where you can't understand how the intended functionality "arises" out of the interaction of its parts, is a system that is very difficult to debug and evolve.
k__ 5y ago
I call that simply lazy management. That's one reason why so many managers hate remote work. Before remote, things would sort them out on their own most of the time. Now they have to think about their job and define processes etc.
deleted 5y ago
[deleted]
iLemming 5y ago
A meeting. At 7AM? I dunno, man, I think I'd demand to be paid a million a year if someone forced me to attend a meeting at 7AM. Even with everyone working remotely can we please try not to make this kind of BS a norm in the industry?
- shikoba 5y ago
  Some people like that, because they can leave earlier.
- bklaasen 5y ago
  The first sentence of the article is "There’s a collection of friends that I have a standing videochat with every couple of weeks."
  - iLemming 5y ago
    Well, I read it off my phone at 7AM. My brain interpreted that as "a daily standup at 7AM". I guess it illustrates precisely why it's a bad idea to have meetings very early in the morning.
- billsmithaustin 5y ago
  The author meets a collection of friends at 7am because a later time conflicts with taking his kids to school. Friends, not co-workers.
- toomanybeersies 5y ago
  All my mates who are builders, or work in related trades, start work at 7 AM. They most certainly don't get paid a million bucks to do so.
  - iLemming 5y ago
    Most my mates who are artists, designers, writers and scientists love working late at night... See? The argument can easily go the other way around. Builders, bus drivers, farmers, etc., all have to start early, and that's understandable. Our field of work (I suppose most readers here are IT professionals) never had to normalize fluctuating schedules until relatively recently - when our profession has become fashionably decentralized. And I think we need to be careful and make sure that harmful practices like having too many meetings or mandatory meetings at odd hours have not become a norm. Programmers don't create things and solve problems at meetings. Most of the time, the meetings are there for other people in the business.
_v7gu 5y ago
Tangential rant, I recently learned it was illegal in some parts of North America that letting your child go to school by themselves is illegal. As in CPS gets involved if they get a call from a "concerned" citizen. For example in Ontario anyone under 16 has to be constantly under adult supervision. What the fuck, America? What the fuck, Canada?
- nuclearnice1 5y ago
  I think the Ontario story was misreported. Here is a long article trying to correct the record. https://www.google.com/amp/s/www.thestar.com/amp/news/gta/2017/09/07/until-what-age-do-kids-need-direct-supervision-well-it-depends-keenan.html https://www.google.com/amp/s/www.thestar.com/amp/news/gta/20...
- burnished 5y ago
  Might just be anger-bait? I grew up in Washington (both in America and near Canada!) and I remember walking to school on my own at around eight years old or so I think, maybe a bit older (started somewhere in grades 1-6).
mpls 5y ago
> their service might exhibit problems with long-lived instances that they never notice because Chaos Monkey tends to terminate instances before they hit those problems Tangential to the main point of the article, but I've seen this exact failure before! I worked at a retailer that would reboot all point-of-sale devices daily, after stores closed. The day before biggest sales day of the year, the team decided to leave the devices online overnight - sometimes a device would fail to come back online after the reboot, so leaving them on would prevent that sort of issue. Of course, one bug or another reared its head after 24 hours of uptime. We ended up rebooting all of the devices anyway.
ChrisMarshallNY 5y ago
I like it when folks map experiences in their lives to software development. It doesn’t always work, but this time, it did.
klodolph 5y ago
We've had discussions about this at work. New versions of our service got deployed once per week, barring delays and hotfixes. Sometimes we'd miss a release, and a version would stay in production for two weeks. Maybe there would be a code freeze and we'd miss a release, and a version would stay in production for three weeks. We occasionally saw problems. Some cache would fill up over two weeks but wouldn't empty, or something like that. Something that you'd like to test, but which doesn't show up unless you run something in production for a month. I advocated a simple solution: either check that the process runs for multiple weeks in testing, or restart the process every week in production. In other words, don't test it in production.
pxc 5y ago
There's a notion in cybernetics of something like ‘the real org chart’ going all the way back to Stafford Beer's seminal work The Brain of the Firm. In any enterprise, org charts are fundamentally nominal, and you should always expect there to be gaps between that model and the way roles are actually structured in the organization.

168 points by henrik_w 5y ago

lrem 5y ago

I wonder what kind of an exciting outage provoked this post :)

withinboredom 5y ago
Probably the one where the kid didn't make it to school on time and everyone's day was out of whack.

eru 5y ago

> Now imagine declaring an extended period of time where, in the interest of reducing risk(!), no new code is deployed, and Chaos Monkey is disabled. I remember reading about similar experiment in biology: If you have a petri dish full of bacteria, and you gradually introduce antibiotics, not only will your bacteria population grow resistant to the antibiotics over time, but there metabolism even might become dependent on it. You see, biology is a system as least as complicated as any computing infrastructure we have. (Compare also most organisms these days being reliant on oxygen. Or how humans lost the ability to make vitamin C in their bodies.)

lucumo 5y ago
> You see, biology is a system as least as complicated as any computing infrastructure we have. One of my first bosses describes the opposite relationship. We tend to think technology is very rationalized. He once told me that we should think about the system we were building as an organism. Build it, but also play with it and get to know it, because it will have its own personality.
reuben364 5y ago
Here is a video on the topic of bacteria in an experiment evolving to use a second source of food: https://www.youtube.com/watch?v=w4sLAQvEH-M https://www.youtube.com/watch?v=w4sLAQvEH-M. I'm not sure if they depend on it being present though.

Okkef 5y ago

When a person leaves a company who has been there for a long time, a similar effect occurs. These people often do a host of small tasks to that noone is really aware of (themselves included; they don't see how it's special or that nobody else knows about it). It can take the organization a long time to recover and realign.

xyz11 5y ago
Yes, very true!
quickthrower2 5y ago
Why do they keep the tasks secret? I bet because they know they are important but they also know they are fragile in the sense no one else will care enough to do those things. And worse someone else will say don’t do them because they waste time.
- lucumo 5y ago
  It's probably not so much keeping it a secret, but just not broadcasting everything they do.
- watwut 5y ago
  They don't do them in secret. Typically they just do them as part of their routine.
- magicalhippo 5y ago
  During a summer not long ago, a client called with an urgent issue. Some critical messages were not being passed on, leading to a full stop on their end. However I couldn't make heads or tails of what systems this was or how it related to us. After a bit of back and forth, the customer said "ask Glenn, he always fixes this for us". Glenn was of course on vacation and didn't respond to messages... A few hours pass while we make zero progress on this issue and customer increasingly annoyed, when Glenn finally responds to my messages. Turns out that on a different server from where our software is installed, there was a Java-based message broker service running that needed occasional kick in the pants (IIRC stopping, deleting a file, restarting). We're not a Java shop, we don't use message brokers. For years, Glenn had been keeping this service running and updated for our customer, as the customer didn't have any internal resources that was capable of handling this. And nobody else at our shop knew this. Why? Well, he just did what had to be done and moved on to the next issue at hand, as he always did...

moksly 5y ago

This is why documentation matters, and why it can’t just live in a drawer. Who would set up a virtual machine backup system that offers no alerts when it recycles something? I mean, I might do it myself, so I sort of know, but you shouldn’t do it. Similarly I’m currently having the “pleasure” of documenting every function I’m responsible for as I’m leaving the public sector for the private sector, and man oh man, would the excel sheet of functions and directions to where they happen have saved myself from hundreds of hours “searching” or “refreshing” my knowledge on things when I had to revisit them after years of not working on them. It’s also why MBA business process types are valuable in any enterprise sized organisation. Because they tend to both discover and document these hidden functions when they do their work. Of course, 9 out of 10 times nothing comes of their discoveries, because managers are notoriously poor at long term benefits realisation.

brobdingnagians 5y ago
I've been thinking about this in regards to things like Idris vs TLA+ as well. I prefer using Idris than TLA+ simply because Idris is continually with you, type checking and part of the overall design. TLA+ can be used to verify a design, but then it is not really a living, breathing part of the end product. Having deduction built into the safety of the language is preferable to only verifying it externally then having it mutate under the next engineer. It is the same with self-documenting code; if the function and variables names are clear with a sensible architecture in place, then it is easier to keep everything in sync and readable even as different engineers take over; rather than having external documentation which must be sought for, possibly discarded, and therefore not used as much as it ought. One of the things I really like about Elixir is being able to put executable tests and examples inside the module documentation. If you run the tests and it fails, you know you need to update the examples in the documentation.
- simiones 5y ago
  As far as I understand, Idris and dependent typing in general are only useable for small systems today - the largest systems successfully proved using dependent typing are the SEL4 kernel (which is ~10k lines of C) and the CompCert C compilers (which compiles a subset of the C language, and has only a minimalistic optimizer, if I remeber correctly). And these two have been 5-10 year PhD projects.

a_c 5y ago

https://xkcd.com/1172/ https://xkcd.com/1172/

withinboredom 5y ago

heh, my boss went on vacation. We were scrambling to do the technical part of their job. We never knew the boss did so much behind the scenes and we had no idea how difficult or time consuming it was. When they came back, we had so much mad respect for the things they kept out of our day-to-day job.

orangepanda 5y ago
Similar with _good_ project managers - you get shielded from so many useless meetings and drama

sbierwagen 5y ago

>I didn’t know that I was functioning as an alarm clock for her! That’s why I was careful to be quiet, and why I didn’t even think to mention to her about the new videochat time. >I suspect this is failure mode is more common than we realize: there is a process inside a system, and over time the process comes to fulfill some unintended, ancillary functional role, and there are people who participate in this process that aren’t even aware of this function. Ah, Hyrum's Law showing up in an unexpected context, yet again. https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

deleted 5y ago
[deleted]
bestouff 5y ago
Oblig XKCD: https://xkcd.com/1172/ https://xkcd.com/1172/

hsn915 5y ago

It's not so much of a danger. Occasionally disrupting the system can reveal such weaknesses. Once you identify such "fragilities" you can compensate by adding redundancies. For example, add a second alarm clock at 6:30 and 7:00. When it comes to programming, this is one of the good reasons why simplification matters. Systems that are complicated to the point where you can't understand how the intended functionality "arises" out of the interaction of its parts, is a system that is very difficult to debug and evolve.

k__ 5y ago

I call that simply lazy management. That's one reason why so many managers hate remote work. Before remote, things would sort them out on their own most of the time. Now they have to think about their job and define processes etc.

deleted 5y ago

[deleted]

iLemming 5y ago

A meeting. At 7AM? I dunno, man, I think I'd demand to be paid a million a year if someone forced me to attend a meeting at 7AM. Even with everyone working remotely can we please try not to make this kind of BS a norm in the industry?

shikoba 5y ago
Some people like that, because they can leave earlier.
bklaasen 5y ago
The first sentence of the article is "There’s a collection of friends that I have a standing videochat with every couple of weeks."
- iLemming 5y ago
  Well, I read it off my phone at 7AM. My brain interpreted that as "a daily standup at 7AM". I guess it illustrates precisely why it's a bad idea to have meetings very early in the morning.
billsmithaustin 5y ago
The author meets a collection of friends at 7am because a later time conflicts with taking his kids to school. Friends, not co-workers.
toomanybeersies 5y ago
All my mates who are builders, or work in related trades, start work at 7 AM. They most certainly don't get paid a million bucks to do so.
- iLemming 5y ago
  Most my mates who are artists, designers, writers and scientists love working late at night... See? The argument can easily go the other way around. Builders, bus drivers, farmers, etc., all have to start early, and that's understandable. Our field of work (I suppose most readers here are IT professionals) never had to normalize fluctuating schedules until relatively recently - when our profession has become fashionably decentralized. And I think we need to be careful and make sure that harmful practices like having too many meetings or mandatory meetings at odd hours have not become a norm. Programmers don't create things and solve problems at meetings. Most of the time, the meetings are there for other people in the business.

_v7gu 5y ago

Tangential rant, I recently learned it was illegal in some parts of North America that letting your child go to school by themselves is illegal. As in CPS gets involved if they get a call from a "concerned" citizen. For example in Ontario anyone under 16 has to be constantly under adult supervision. What the fuck, America? What the fuck, Canada?

nuclearnice1 5y ago
I think the Ontario story was misreported. Here is a long article trying to correct the record. https://www.google.com/amp/s/www.thestar.com/amp/news/gta/2017/09/07/until-what-age-do-kids-need-direct-supervision-well-it-depends-keenan.html https://www.google.com/amp/s/www.thestar.com/amp/news/gta/20...
burnished 5y ago
Might just be anger-bait? I grew up in Washington (both in America and near Canada!) and I remember walking to school on my own at around eight years old or so I think, maybe a bit older (started somewhere in grades 1-6).

mpls 5y ago

> their service might exhibit problems with long-lived instances that they never notice because Chaos Monkey tends to terminate instances before they hit those problems Tangential to the main point of the article, but I've seen this exact failure before! I worked at a retailer that would reboot all point-of-sale devices daily, after stores closed. The day before biggest sales day of the year, the team decided to leave the devices online overnight - sometimes a device would fail to come back online after the reboot, so leaving them on would prevent that sort of issue. Of course, one bug or another reared its head after 24 hours of uptime. We ended up rebooting all of the devices anyway.

ChrisMarshallNY 5y ago

I like it when folks map experiences in their lives to software development. It doesn’t always work, but this time, it did.

klodolph 5y ago

We've had discussions about this at work. New versions of our service got deployed once per week, barring delays and hotfixes. Sometimes we'd miss a release, and a version would stay in production for two weeks. Maybe there would be a code freeze and we'd miss a release, and a version would stay in production for three weeks. We occasionally saw problems. Some cache would fill up over two weeks but wouldn't empty, or something like that. Something that you'd like to test, but which doesn't show up unless you run something in production for a month. I advocated a simple solution: either check that the process runs for multiple weeks in testing, or restart the process every week in production. In other words, don't test it in production.

pxc 5y ago

There's a notion in cybernetics of something like ‘the real org chart’ going all the way back to Stafford Beer's seminal work The Brain of the Firm. In any enterprise, org charts are fundamentally nominal, and you should always expect there to be gaps between that model and the way roles are actually structured in the organization.