6 ms·

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

blantonl 6y ago
Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.
- staunch 6y ago
  And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
  - nemosaltat 6y ago
    Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.
    - AtlasBarfed 6y ago
      I've noticed from pair programming that the person navigating with a mouse is far less able to read and interpret their surroundings or pick up typos while typing, than an observer that simply has to watch what the other person is doing. Like when clicking on a file in a directory you just entered and looking for the file, the observer can literally locate and point to the file for the mouse user 5-10x faster than the mouse operator. The observer seems to interpret the information that results from the directory listing faster than the person who just did the double-click to enter the directory because they don't have the muscle coordination context switch and can immediately move to interpreting the results. It's probably because mouse manipulation uses brain infrastructure that is more recently evolved, but observe-react is a lot earlier in the brain processing pipeline evolutionarily, and a lot more refined/involved.
      - yobert 6y ago
        I think this is because writing software is so much more than operating switches and controls. I really hate pair programming for this reason, but I love industrial-style controls and protocols involving multiple people.
        robaato 6y ago
        Back in the '80s I worked on a financial system (SWIFT interface) for an Italian bank. It went operational and we observed 2 operators effectively doing "pair operating". We just thought it was weird Italian style socialising - one had the keyboard and the other was chattering away with a commentary. But they were surprisingly effective! I accidentally learned when teaching a course at a site with too many people for the available machines, that pair exercises was very effective - I got lots more questions and overall learning went way up. If the pair discussed it and couldn't find an answer they would have the confidence to ask. On their own, neither would probably bother and just wait for me to go through things.
      - Fr0styMatt88 6y ago
        Since I have a vision impairment, I'm sure the effect is amplified very much for me, but using the mouse is such a massive break in flow: - First you have to lift one hand up off the keyboard and put it down on the mouse. This may or may not mean taking your eyes off the screen. - Then you need to find the mouse pointer on the screen - Then you need to aim for what is usually a relatively small target and move the pointer there. - If you're right-clicking, the right-click menu usually presents more small targets you need to aim for. - If you need to use the keyboard, again you have to move your hand over to the keyboard from the mouse. For finding the pointer, I developed this unconscious habit of slamming the mouse pointer to the very top-left of the screen. It's difficult though when on someone else's machine, where your brain isn't used to the pointer velocity or where multi-monitor means that slamming the mouse to the top-left actually puts the pointer on another monitor. People look at me in awe when I'm using a two-pane file manager but honestly not having to take your hands off the keyboard and not having to move your eyes off the screen gives so much better flow. It's also why I like the UI of Blender - one hand on the keyboard and one hand on the mouse at most times.
  - quercusa 6y ago
    And then there's the interactions with the Air Traffic Control system: flight plans, Standard Approaches, charts, etc. It's very impressive.
  - Sharlin 6y ago
    And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.
- morty_s 6y ago
  Came here for this. A: “Passing control” B: “Taking control” A: “You have control” B: “I have control” This is how I remember it (6174, UH-1Y).
  - cortesoft 6y ago
    The TCP handshake IRL
  - BalinKing 6y ago
    As a fun fact, I was taught a shorter version (during private pilot instruction): "You have the control." "I have the control." IDK if it changes between aircraft types, commercial/private/military cultures, or if it's just coincidence.
    - regularfry 6y ago
      Yep, I had this version in a military trainer.
    - pacaro 6y ago
      Or as we learned from Sully, when you're the captain, and you take charge and responsibility quickly, a simple "My plane" suffices
  - rurp 6y ago
    Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like A: "Belay on?" B: "Belay on" A: "Climbing" B: "Climb on" Then the climber begins. It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.
    - Talanes 6y ago
      For the ultimate low stakes version of this, when I played World of Warcraft in my younger days, tank swaps would communicate the same way.
- waterhouse 6y ago
  I searched Youtube for examples of this. This is a little bit staged, but it seems to be a real checklist they're going through: https://www.youtube.com/watch?v=JG7SkOQDDt0 https://www.youtube.com/watch?v=JG7SkOQDDt0 Though they're not perfect. They said that one pilot is supposed to read the item, the other pilot say the answer, and the first pilot visually confirm it; but at 1:42, I noticed the first pilot say "emergency exit lights", hear the confirmation, and move to the next item without her eyes moving away from the list. I'm not sure which of several possible conclusions to draw from that. ("Humans suck", "it is indeed staged", "the procedure has enough redundancy that the chance they're both careless on a given step is small", "the pilots feel that the emergency exit lights aren't particularly important", ...)
  - andi999 6y ago
    Routine is the killer. Have a look at the fatal maglev train accident in Germany. Service car was on the track. Presence of service car in service bay (and not on track) can be seen visually by operator (driver) in the control centre when turning head. (If I remember correctly)
js2 6y ago
I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Technically he had no beard and if he had, it wouldn't have been gray.
- brundolf 6y ago
  Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement
  - MaxBarraclough 6y ago
    Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear. Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.
    - brundolf 6y ago
      That's nifty, but sounds like more effort than changing a single color in one's .bash_profile
      - MaxBarraclough 6y ago
        That's true, but depending on the configuration it may benefit all future users of the server.
      - gknoy 6y ago
        Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this: ib production && ssh production-machine ib demo && ssh demo-machine It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.
morelisp 6y ago
I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on. It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different opinions.) Unfortunately if you do P&C and still make the mistake due to the aforementioned tooling, you look even stupider.
- myself248 6y ago
  Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day. When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point. Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance. There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)

blantonl 6y ago

Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.

staunch 6y ago
And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
- nemosaltat 6y ago
  Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.
  - AtlasBarfed 6y ago
    I've noticed from pair programming that the person navigating with a mouse is far less able to read and interpret their surroundings or pick up typos while typing, than an observer that simply has to watch what the other person is doing. Like when clicking on a file in a directory you just entered and looking for the file, the observer can literally locate and point to the file for the mouse user 5-10x faster than the mouse operator. The observer seems to interpret the information that results from the directory listing faster than the person who just did the double-click to enter the directory because they don't have the muscle coordination context switch and can immediately move to interpreting the results. It's probably because mouse manipulation uses brain infrastructure that is more recently evolved, but observe-react is a lot earlier in the brain processing pipeline evolutionarily, and a lot more refined/involved.
    - yobert 6y ago
      I think this is because writing software is so much more than operating switches and controls. I really hate pair programming for this reason, but I love industrial-style controls and protocols involving multiple people.
      - robaato 6y ago
        Back in the '80s I worked on a financial system (SWIFT interface) for an Italian bank. It went operational and we observed 2 operators effectively doing "pair operating". We just thought it was weird Italian style socialising - one had the keyboard and the other was chattering away with a commentary. But they were surprisingly effective! I accidentally learned when teaching a course at a site with too many people for the available machines, that pair exercises was very effective - I got lots more questions and overall learning went way up. If the pair discussed it and couldn't find an answer they would have the confidence to ask. On their own, neither would probably bother and just wait for me to go through things.
    - Fr0styMatt88 6y ago
      Since I have a vision impairment, I'm sure the effect is amplified very much for me, but using the mouse is such a massive break in flow: - First you have to lift one hand up off the keyboard and put it down on the mouse. This may or may not mean taking your eyes off the screen. - Then you need to find the mouse pointer on the screen - Then you need to aim for what is usually a relatively small target and move the pointer there. - If you're right-clicking, the right-click menu usually presents more small targets you need to aim for. - If you need to use the keyboard, again you have to move your hand over to the keyboard from the mouse. For finding the pointer, I developed this unconscious habit of slamming the mouse pointer to the very top-left of the screen. It's difficult though when on someone else's machine, where your brain isn't used to the pointer velocity or where multi-monitor means that slamming the mouse to the top-left actually puts the pointer on another monitor. People look at me in awe when I'm using a two-pane file manager but honestly not having to take your hands off the keyboard and not having to move your eyes off the screen gives so much better flow. It's also why I like the UI of Blender - one hand on the keyboard and one hand on the mouse at most times.
- quercusa 6y ago
  And then there's the interactions with the Air Traffic Control system: flight plans, Standard Approaches, charts, etc. It's very impressive.
- Sharlin 6y ago
  And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.
morty_s 6y ago
Came here for this. A: “Passing control” B: “Taking control” A: “You have control” B: “I have control” This is how I remember it (6174, UH-1Y).
- cortesoft 6y ago
  The TCP handshake IRL
- BalinKing 6y ago
  As a fun fact, I was taught a shorter version (during private pilot instruction): "You have the control." "I have the control." IDK if it changes between aircraft types, commercial/private/military cultures, or if it's just coincidence.
  - regularfry 6y ago
    Yep, I had this version in a military trainer.
  - pacaro 6y ago
    Or as we learned from Sully, when you're the captain, and you take charge and responsibility quickly, a simple "My plane" suffices
- rurp 6y ago
  Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like A: "Belay on?" B: "Belay on" A: "Climbing" B: "Climb on" Then the climber begins. It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.
  - Talanes 6y ago
    For the ultimate low stakes version of this, when I played World of Warcraft in my younger days, tank swaps would communicate the same way.
waterhouse 6y ago
I searched Youtube for examples of this. This is a little bit staged, but it seems to be a real checklist they're going through: https://www.youtube.com/watch?v=JG7SkOQDDt0 https://www.youtube.com/watch?v=JG7SkOQDDt0 Though they're not perfect. They said that one pilot is supposed to read the item, the other pilot say the answer, and the first pilot visually confirm it; but at 1:42, I noticed the first pilot say "emergency exit lights", hear the confirmation, and move to the next item without her eyes moving away from the list. I'm not sure which of several possible conclusions to draw from that. ("Humans suck", "it is indeed staged", "the procedure has enough redundancy that the chance they're both careless on a given step is small", "the pilots feel that the emergency exit lights aren't particularly important", ...)
- andi999 6y ago
  Routine is the killer. Have a look at the fatal maglev train accident in Germany. Service car was on the track. Presence of service car in service bay (and not on track) can be seen visually by operator (driver) in the control centre when turning head. (If I remember correctly)

js2 6y ago

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Technically he had no beard and if he had, it wouldn't have been gray.

brundolf 6y ago
Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement
- MaxBarraclough 6y ago
  Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear. Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.
  - brundolf 6y ago
    That's nifty, but sounds like more effort than changing a single color in one's .bash_profile
    - MaxBarraclough 6y ago
      That's true, but depending on the configuration it may benefit all future users of the server.
    - gknoy 6y ago
      Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this: ib production && ssh production-machine ib demo && ssh demo-machine It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.

morelisp 6y ago

I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on. It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different opinions.) Unfortunately if you do P&C and still make the mistake due to the aforementioned tooling, you look even stupider.

myself248 6y ago
Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day. When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point. Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance. There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)