27 ms·

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling https://en.wikipedia.org/wiki/Pointing_and_calling

MaxBarraclough 6y ago
Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.
- blantonl 6y ago
  Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.
  - staunch 6y ago
    And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
    - nemosaltat 6y ago
      Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.
      - AtlasBarfed 6y ago
        I've noticed from pair programming that the person navigating with a mouse is far less able to read and interpret their surroundings or pick up typos while typing, than an observer that simply has to watch what the other person is doing. Like when clicking on a file in a directory you just entered and looking for the file, the observer can literally locate and point to the file for the mouse user 5-10x faster than the mouse operator. The observer seems to interpret the information that results from the directory listing faster than the person who just did the double-click to enter the directory because they don't have the muscle coordination context switch and can immediately move to interpreting the results. It's probably because mouse manipulation uses brain infrastructure that is more recently evolved, but observe-react is a lot earlier in the brain processing pipeline evolutionarily, and a lot more refined/involved.
        yobert 6y ago
        I think this is because writing software is so much more than operating switches and controls. I really hate pair programming for this reason, but I love industrial-style controls and protocols involving multiple people.
        robaato 6y ago
        Back in the '80s I worked on a financial system (SWIFT interface) for an Italian bank. It went operational and we observed 2 operators effectively doing "pair operating". We just thought it was weird Italian style socialising - one had the keyboard and the other was chattering away with a commentary. But they were surprisingly effective! I accidentally learned when teaching a course at a site with too many people for the available machines, that pair exercises was very effective - I got lots more questions and overall learning went way up. If the pair discussed it and couldn't find an answer they would have the confidence to ask. On their own, neither would probably bother and just wait for me to go through things.
        Fr0styMatt88 6y ago
        Since I have a vision impairment, I'm sure the effect is amplified very much for me, but using the mouse is such a massive break in flow: - First you have to lift one hand up off the keyboard and put it down on the mouse. This may or may not mean taking your eyes off the screen. - Then you need to find the mouse pointer on the screen - Then you need to aim for what is usually a relatively small target and move the pointer there. - If you're right-clicking, the right-click menu usually presents more small targets you need to aim for. - If you need to use the keyboard, again you have to move your hand over to the keyboard from the mouse. For finding the pointer, I developed this unconscious habit of slamming the mouse pointer to the very top-left of the screen. It's difficult though when on someone else's machine, where your brain isn't used to the pointer velocity or where multi-monitor means that slamming the mouse to the top-left actually puts the pointer on another monitor. People look at me in awe when I'm using a two-pane file manager but honestly not having to take your hands off the keyboard and not having to move your eyes off the screen gives so much better flow. It's also why I like the UI of Blender - one hand on the keyboard and one hand on the mouse at most times.
    - quercusa 6y ago
      And then there's the interactions with the Air Traffic Control system: flight plans, Standard Approaches, charts, etc. It's very impressive.
    - Sharlin 6y ago
      And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.
  - morty_s 6y ago
    Came here for this. A: “Passing control” B: “Taking control” A: “You have control” B: “I have control” This is how I remember it (6174, UH-1Y).
    - cortesoft 6y ago
      The TCP handshake IRL
    - BalinKing 6y ago
      As a fun fact, I was taught a shorter version (during private pilot instruction): "You have the control." "I have the control." IDK if it changes between aircraft types, commercial/private/military cultures, or if it's just coincidence.
      - regularfry 6y ago
        Yep, I had this version in a military trainer.
      - pacaro 6y ago
        Or as we learned from Sully, when you're the captain, and you take charge and responsibility quickly, a simple "My plane" suffices
    - rurp 6y ago
      Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like A: "Belay on?" B: "Belay on" A: "Climbing" B: "Climb on" Then the climber begins. It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.
      - Talanes 6y ago
        For the ultimate low stakes version of this, when I played World of Warcraft in my younger days, tank swaps would communicate the same way.
  - waterhouse 6y ago
    I searched Youtube for examples of this. This is a little bit staged, but it seems to be a real checklist they're going through: https://www.youtube.com/watch?v=JG7SkOQDDt0 https://www.youtube.com/watch?v=JG7SkOQDDt0 Though they're not perfect. They said that one pilot is supposed to read the item, the other pilot say the answer, and the first pilot visually confirm it; but at 1:42, I noticed the first pilot say "emergency exit lights", hear the confirmation, and move to the next item without her eyes moving away from the list. I'm not sure which of several possible conclusions to draw from that. ("Humans suck", "it is indeed staged", "the procedure has enough redundancy that the chance they're both careless on a given step is small", "the pilots feel that the emergency exit lights aren't particularly important", ...)
    - andi999 6y ago
      Routine is the killer. Have a look at the fatal maglev train accident in Germany. Service car was on the track. Presence of service car in service bay (and not on track) can be seen visually by operator (driver) in the control centre when turning head. (If I remember correctly)
- js2 6y ago
  I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Technically he had no beard and if he had, it wouldn't have been gray.
  - brundolf 6y ago
    Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement
    - MaxBarraclough 6y ago
      Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear. Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.
      - brundolf 6y ago
        That's nifty, but sounds like more effort than changing a single color in one's .bash_profile
        MaxBarraclough 6y ago
        That's true, but depending on the configuration it may benefit all future users of the server.
        gknoy 6y ago
        Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this: ib production && ssh production-machine ib demo && ssh demo-machine It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.
- morelisp 6y ago
  I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on. It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different opinions.) Unfortunately if you do P&C and still make the mistake due to the aforementioned tooling, you look even stupider.
  - myself248 6y ago
    Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day. When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point. Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance. There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)
dheera 6y ago
> I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. The concept makes sense, though I don't quite fully get how to translate it to other contexts besides train driving where unexpected and unpredictable events come up all the time. Let's say you're driving a car and the traffic light turns red. Do you point at the traffic light, say "red", point at your brake pedal, say "brakes", and then hit the brakes?
- Timpy 6y ago
  Whenever I have something in my hand that I'm about to put down for a second in the exact absent minded kind of way that would leave me searching all over the house for it 5 minutes later, I say it out loud. "Headphones on the table by front door."
  - roland35 6y ago
    Embarrassingly I once lost a hamburger while still holding it.. I had my arm propped up on a the back of the chair and it was just out of my peripheral vision. Not my smartest moment.
    - adiM 6y ago
      I lost my sunglasses when I was wearing them! We were going to a state park for a hike. It was a 2 hr ride for which I was wearing my sunglasses but forgot. As we came out of the car to start the hike, I spent 5 minutes searching for my sunglasses in my backpack until my friend asked what I was searching for .... Maybe I should be saying "sunglasses on" from now on
      - mamon 6y ago
        Funny, there is a Polish rhyme [1] for children based on the same concept: a person searching the whole house looking for glasses which they were wearing all the time :) [1] https://blogs.transparent.com/polish/okulary-by-julian-tuwim/ https://blogs.transparent.com/polish/okulary-by-julian-tuwim... (scroll down for english version)
YeGoblynQueenne 6y ago
>> I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. Yeah, ouch. More ouch if it's the other way around- you delete the test database and it's not the test database. (long story)
- throwaway894345 6y ago
  I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big deal).
  - txutxu 6y ago
    I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc). He had been all the morning launching and deleting the dev environment. Team mates were joking loud around his table before the moment it did happen. Sadly, he got fired (the company was proud of it's cost savy choices, didn't have other backups than a few days of snapshots, probably CTO choice).
    - Gene_Parmesan 6y ago
      Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massive losses via redundancy & the sort of measures we're talking about here & in the OP. Firing somebody in these circumstances just acts to severely reduce morale, since we all secretly know in our hearts that it very easily could have been us. Firing in this case just seems retributive. It's not going to bring the lost data back, and you've just eliminated the very person who could have told you most about the chain of events leading to the incident in question to help you guard against it in the future. These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. A lack of team focus on reliability/quality, a lack of communication or trust about decisions made (or not made) by higher ups, or so on. And they are probably the single least likely person to cause a similar incident again -- that person will now likely be double and triple checking their commands for eternity.
rachelbythebay 6y ago
I do that when I drive around. Car on the side street. Kid over there... with a ball. Hidden left turner in 3...2...1... yep. I love finding out that this stuff works.
uyt 6y ago
This is true for NYC subways too! https://www.youtube.com/watch?v=i9jIsxQNz0M https://www.youtube.com/watch?v=i9jIsxQNz0M
- greenyoda 6y ago
  The video doesn't really explain why conductors point at the signs - it just says "to prove they're paying attention". Paying attention to what? The answer is that they are verifying that the train is correctly positioned in the station so that all of the doors will open on the platform. Explained here: https://www.nydailynews.com/new-york/mta-conductors-point-stopping-subway-station-article-1.3502286 https://www.nydailynews.com/new-york/mta-conductors-point-st...
  - tialaramex 6y ago
    This comes up every few weeks on HN but nobody has ever offered any statistics that would suggest this is as good let alone better than just having the trains handle alignment automatically. It's a task humans are bad at and machines are good at, so just giving it to machines makes more sense, modulo unions. London Underground hasn't had guards for decades at this point, and the Docklands Light Railway hasn't even had drivers (there is a member of staff who is trained to be able to drive it on every train, but they are usually doing other things) since its creation. If they're misaligning often enough for it to be possible for New York to be statistically better I haven't seen anything about it after repeatedly asking.
    - jpcooper 6y ago
      Actually what exactly is the member of staff doing on the DLR that is necessary, other than answering tourists' questions and putting a triangular key into a receptacle at every stop and then turning it? I have not been able to figure this out. In the Netherlands, the NS has two types of trains that go between towns. Intercity and Sprinter. Sprinters have someone who will walk onto the platform at every stop, or failing that, lean out of the carriage, verify that no one is getting in, and then step into the train again to put the key into the receptacle and then turn it. Following that, the doors close. In contrast, there is no such person on Intercity trains; they do fine without. There may be a conductor who checks tickets. In comparison to the DLR, both Sprinter and Intercity trains have drivers. Is there some requirement or function that I am missing that requires a dedicated member of staff to perform this key-turning ritual at every stop on the DLR and Sprinter, or is this simply to appease the unions? It could be that Sprinters are meant to be more lenient towards people running to get on than Intercities, which might have a stricter schedule.
brundolf 6y ago
I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.
- brlewis 6y ago
  >very experienced developers in their 40s I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.
  - BalinKing 6y ago
    This may not have been their first time, though :-P
    - __d 6y ago
      Some time ago, it was common in Unix sites to have an NFS filesystem mounted on all machines that contained locally-built binaries to augment those provided by the operating system. At this site, we used a bunch of different platforms: OSF/1, Solaris, Linux, HP/UX, etc. So we had a large filesystem containing the source code, and built binaries for all the different platforms, and this included heaps of things, from Bash upwards. A colleague of mine accidentally ran rm -rf on this filesystem. It was taking a loooong time, so he realised and killed it, but not before it had removed a heap of stuff. Because this was something that could be rebuilt, it wasn't backed up, so we had to go through the process of downloading the tarballs, and recompiling everything for all the different platforms. It took a few days to recover most of it, and weeks to completely restore things. The day after the incident, when he arrived at work, he found his keyboard was missing a few keycaps. It took him a while to realise that there were four gone: 'R', 'M', '-', and 'F' ... Good times.
  - PopeDotNinja 6y ago
    I once cloned a directory for standing up an environment via Terraform. I modified all of the environment variables and config and ran it. It worked perfectly. Except I’d forgotten to wipe out the Terraform state, which meant that in the process of creating a new environment, it completely deleted the environment I had cloned. That was my initiation into very experienced :)
saberdancer 6y ago
OpenShift does this by forcing you to write the name of the project you are about to delete. It was something that used to annoy me but reading this I understand it is a good call from their side.
nailer 6y ago
I do things like const HARD_CODE_TEST_DATABASE_FOR_SAFETY = 'unit-testing' destroyDatabase(HARD_CODE_TEST_DATABASE_FOR_SAFETY) 1. Avoid silly terms our industry should have ditched years ago, like 'drop' 2. Making sure that nobody will ever change HARD_CODE_TEST_DATABASE_FOR_SAFETY because they thought it should 'always be the active database' or whatever.
myself248 6y ago
Ever since hearing about point-and-call, I've started using it in the kitchen when turning on the stove. I used to destroy one or two pans a year by turning on the wrong burner, but it's now been about a year and a half and I haven't screwed it up yet. The knobs are labeled with a terrible little glyph meant to indicate which is which, and I've supplemented this with plain-english Brady labels "front left", "front right", etc. Now I speak the words above the knob, and point to the burner. It felt goofy at first, but now it feels normal, and like I'm tempting fate if I skip it.
- giantDinosaur 6y ago
  I'm curious how exactly you managed to destroy pans. I've never destroyed a pan in my life, and take no particular precautions - is this a common thing? Is this more common with non-stick stuff or something?
  - ajb 6y ago
    Not the op, but non stick pans will burn if the pan is heated while empty.
    - pantalaimon 6y ago
      I think non-stick pans are a fad. A well greased iron or steel pan works much better and is impossible to destroy.
      - rpeden 6y ago
        Can non-stick pans even be a fad when Teflon coated cookware has been popular for 60+ years?
        542354234235 6y ago
        Totally a fad. I bet they won't be popular for more than a century or two.
  - 6y ago
viraptor 6y ago
I try to do that during incidents. I'm not 100% there since it's no a company rule, but it helps me at the time and later when writing up details: "I see <behaviour X>", "<Y> should fix it because <Z>", "I'm starting to do <Z> now and seeing ...", etc. It also helps when Z results in a total meltdown and you need to pull in more people to help out, so they have context of what happened.
- Qu3tzal 6y ago
  French firefighters do this when arriving at a scene. The first messages sent over the radio will say: - I am... (who you are and where you are) - I see... (describe what you see in simple non-ambiguous terms) - I do... (what action you are taking now) - I ask... (ask for reinforcements if necessary, you may be asked to justify yourself more)
stjohnswarts 6y ago
I always called it a "that can't be right" interrogative.
xvf22 6y ago
Killed just under 1k access points when they all upgraded on one go. They had no problem erasing the firmware but when they all tried to download the new one at once it killer the service and we ended up with a lot of blank APs. The conformation message for 1 or 1000 APs is unhelpfully "This will overwrite all existing system images. Are you sure Y/N"
acdha 6y ago
Back when I shelled into servers more, I really liked having my deployment put the environment in the prompt and set a red background on production for similar reasons. It only takes a small change to jar you out of habit.
ekanes 6y ago
I do this with my kids, gesturing (not pointing) as it helps my mind remain focused on truly listening to them amid everything else going on. I probably look ridiculous, but I'm a better father for it so ¯\_(ツ)_/¯
justinlloyd 6y ago
I have had many disasters in my software career because I jut wantonly hit "Y" without thinking about it. I have noticed, since learning to cook at a professional level in the kitchen, that I point and call out a lot more in my other activities too. "From hot behind" and "knife" and "oven is over temp" to "Saw blade is live" and "circuit is live" in the workshop to "production server" and "erasing records" in database maintenance. Some days I feel like Sigourney "I have one job damnit" Weaver in Galaxyquest. It's a useful stop-think-go sanity check.
m463 6y ago
> forcing a cache miss in the brain That is an interesting way of looking at it. I think a router analogy might be more precise - more like fast path / slow path - where when most packets come in they hit the fast path in hardware, and slow path exception packets get handled by the cpu. :)

MaxBarraclough 6y ago

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

blantonl 6y ago
Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.
- staunch 6y ago
  And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.
  - nemosaltat 6y ago
    Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action. For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step. Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye. People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.
    - AtlasBarfed 6y ago
      I've noticed from pair programming that the person navigating with a mouse is far less able to read and interpret their surroundings or pick up typos while typing, than an observer that simply has to watch what the other person is doing. Like when clicking on a file in a directory you just entered and looking for the file, the observer can literally locate and point to the file for the mouse user 5-10x faster than the mouse operator. The observer seems to interpret the information that results from the directory listing faster than the person who just did the double-click to enter the directory because they don't have the muscle coordination context switch and can immediately move to interpreting the results. It's probably because mouse manipulation uses brain infrastructure that is more recently evolved, but observe-react is a lot earlier in the brain processing pipeline evolutionarily, and a lot more refined/involved.
      - yobert 6y ago
        I think this is because writing software is so much more than operating switches and controls. I really hate pair programming for this reason, but I love industrial-style controls and protocols involving multiple people.
        robaato 6y ago
        Back in the '80s I worked on a financial system (SWIFT interface) for an Italian bank. It went operational and we observed 2 operators effectively doing "pair operating". We just thought it was weird Italian style socialising - one had the keyboard and the other was chattering away with a commentary. But they were surprisingly effective! I accidentally learned when teaching a course at a site with too many people for the available machines, that pair exercises was very effective - I got lots more questions and overall learning went way up. If the pair discussed it and couldn't find an answer they would have the confidence to ask. On their own, neither would probably bother and just wait for me to go through things.
      - Fr0styMatt88 6y ago
        Since I have a vision impairment, I'm sure the effect is amplified very much for me, but using the mouse is such a massive break in flow: - First you have to lift one hand up off the keyboard and put it down on the mouse. This may or may not mean taking your eyes off the screen. - Then you need to find the mouse pointer on the screen - Then you need to aim for what is usually a relatively small target and move the pointer there. - If you're right-clicking, the right-click menu usually presents more small targets you need to aim for. - If you need to use the keyboard, again you have to move your hand over to the keyboard from the mouse. For finding the pointer, I developed this unconscious habit of slamming the mouse pointer to the very top-left of the screen. It's difficult though when on someone else's machine, where your brain isn't used to the pointer velocity or where multi-monitor means that slamming the mouse to the top-left actually puts the pointer on another monitor. People look at me in awe when I'm using a two-pane file manager but honestly not having to take your hands off the keyboard and not having to move your eyes off the screen gives so much better flow. It's also why I like the UI of Blender - one hand on the keyboard and one hand on the mouse at most times.
  - quercusa 6y ago
    And then there's the interactions with the Air Traffic Control system: flight plans, Standard Approaches, charts, etc. It's very impressive.
  - Sharlin 6y ago
    And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.
- morty_s 6y ago
  Came here for this. A: “Passing control” B: “Taking control” A: “You have control” B: “I have control” This is how I remember it (6174, UH-1Y).
  - cortesoft 6y ago
    The TCP handshake IRL
  - BalinKing 6y ago
    As a fun fact, I was taught a shorter version (during private pilot instruction): "You have the control." "I have the control." IDK if it changes between aircraft types, commercial/private/military cultures, or if it's just coincidence.
    - regularfry 6y ago
      Yep, I had this version in a military trainer.
    - pacaro 6y ago
      Or as we learned from Sully, when you're the captain, and you take charge and responsibility quickly, a simple "My plane" suffices
  - rurp 6y ago
    Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like A: "Belay on?" B: "Belay on" A: "Climbing" B: "Climb on" Then the climber begins. It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.
    - Talanes 6y ago
      For the ultimate low stakes version of this, when I played World of Warcraft in my younger days, tank swaps would communicate the same way.
- waterhouse 6y ago
  I searched Youtube for examples of this. This is a little bit staged, but it seems to be a real checklist they're going through: https://www.youtube.com/watch?v=JG7SkOQDDt0 https://www.youtube.com/watch?v=JG7SkOQDDt0 Though they're not perfect. They said that one pilot is supposed to read the item, the other pilot say the answer, and the first pilot visually confirm it; but at 1:42, I noticed the first pilot say "emergency exit lights", hear the confirmation, and move to the next item without her eyes moving away from the list. I'm not sure which of several possible conclusions to draw from that. ("Humans suck", "it is indeed staged", "the procedure has enough redundancy that the chance they're both careless on a given step is small", "the pilots feel that the emergency exit lights aren't particularly important", ...)
  - andi999 6y ago
    Routine is the killer. Have a look at the fatal maglev train accident in Germany. Service car was on the track. Presence of service car in service bay (and not on track) can be seen visually by operator (driver) in the control centre when turning head. (If I remember correctly)
js2 6y ago
I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Technically he had no beard and if he had, it wouldn't have been gray.
- brundolf 6y ago
  Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement
  - MaxBarraclough 6y ago
    Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear. Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.
    - brundolf 6y ago
      That's nifty, but sounds like more effort than changing a single color in one's .bash_profile
      - MaxBarraclough 6y ago
        That's true, but depending on the configuration it may benefit all future users of the server.
      - gknoy 6y ago
        Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this: ib production && ssh production-machine ib demo && ssh demo-machine It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.
morelisp 6y ago
I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on. It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different opinions.) Unfortunately if you do P&C and still make the mistake due to the aforementioned tooling, you look even stupider.
- myself248 6y ago
  Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day. When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point. Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance. There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)

dheera 6y ago

> I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. The concept makes sense, though I don't quite fully get how to translate it to other contexts besides train driving where unexpected and unpredictable events come up all the time. Let's say you're driving a car and the traffic light turns red. Do you point at the traffic light, say "red", point at your brake pedal, say "brakes", and then hit the brakes?

Timpy 6y ago
Whenever I have something in my hand that I'm about to put down for a second in the exact absent minded kind of way that would leave me searching all over the house for it 5 minutes later, I say it out loud. "Headphones on the table by front door."
- roland35 6y ago
  Embarrassingly I once lost a hamburger while still holding it.. I had my arm propped up on a the back of the chair and it was just out of my peripheral vision. Not my smartest moment.
  - adiM 6y ago
    I lost my sunglasses when I was wearing them! We were going to a state park for a hike. It was a 2 hr ride for which I was wearing my sunglasses but forgot. As we came out of the car to start the hike, I spent 5 minutes searching for my sunglasses in my backpack until my friend asked what I was searching for .... Maybe I should be saying "sunglasses on" from now on
    - mamon 6y ago
      Funny, there is a Polish rhyme [1] for children based on the same concept: a person searching the whole house looking for glasses which they were wearing all the time :) [1] https://blogs.transparent.com/polish/okulary-by-julian-tuwim/ https://blogs.transparent.com/polish/okulary-by-julian-tuwim... (scroll down for english version)

YeGoblynQueenne 6y ago

>> I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. Yeah, ouch. More ouch if it's the other way around- you delete the test database and it's not the test database. (long story)

throwaway894345 6y ago
I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big deal).
- txutxu 6y ago
  I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc). He had been all the morning launching and deleting the dev environment. Team mates were joking loud around his table before the moment it did happen. Sadly, he got fired (the company was proud of it's cost savy choices, didn't have other backups than a few days of snapshots, probably CTO choice).
  - Gene_Parmesan 6y ago
    Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massive losses via redundancy & the sort of measures we're talking about here & in the OP. Firing somebody in these circumstances just acts to severely reduce morale, since we all secretly know in our hearts that it very easily could have been us. Firing in this case just seems retributive. It's not going to bring the lost data back, and you've just eliminated the very person who could have told you most about the chain of events leading to the incident in question to help you guard against it in the future. These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. A lack of team focus on reliability/quality, a lack of communication or trust about decisions made (or not made) by higher ups, or so on. And they are probably the single least likely person to cause a similar incident again -- that person will now likely be double and triple checking their commands for eternity.

rachelbythebay 6y ago

I do that when I drive around. Car on the side street. Kid over there... with a ball. Hidden left turner in 3...2...1... yep. I love finding out that this stuff works.

uyt 6y ago

This is true for NYC subways too! https://www.youtube.com/watch?v=i9jIsxQNz0M https://www.youtube.com/watch?v=i9jIsxQNz0M

greenyoda 6y ago
The video doesn't really explain why conductors point at the signs - it just says "to prove they're paying attention". Paying attention to what? The answer is that they are verifying that the train is correctly positioned in the station so that all of the doors will open on the platform. Explained here: https://www.nydailynews.com/new-york/mta-conductors-point-stopping-subway-station-article-1.3502286 https://www.nydailynews.com/new-york/mta-conductors-point-st...
- tialaramex 6y ago
  This comes up every few weeks on HN but nobody has ever offered any statistics that would suggest this is as good let alone better than just having the trains handle alignment automatically. It's a task humans are bad at and machines are good at, so just giving it to machines makes more sense, modulo unions. London Underground hasn't had guards for decades at this point, and the Docklands Light Railway hasn't even had drivers (there is a member of staff who is trained to be able to drive it on every train, but they are usually doing other things) since its creation. If they're misaligning often enough for it to be possible for New York to be statistically better I haven't seen anything about it after repeatedly asking.
  - jpcooper 6y ago
    Actually what exactly is the member of staff doing on the DLR that is necessary, other than answering tourists' questions and putting a triangular key into a receptacle at every stop and then turning it? I have not been able to figure this out. In the Netherlands, the NS has two types of trains that go between towns. Intercity and Sprinter. Sprinters have someone who will walk onto the platform at every stop, or failing that, lean out of the carriage, verify that no one is getting in, and then step into the train again to put the key into the receptacle and then turn it. Following that, the doors close. In contrast, there is no such person on Intercity trains; they do fine without. There may be a conductor who checks tickets. In comparison to the DLR, both Sprinter and Intercity trains have drivers. Is there some requirement or function that I am missing that requires a dedicated member of staff to perform this key-turning ritual at every stop on the DLR and Sprinter, or is this simply to appease the unions? It could be that Sprinters are meant to be more lenient towards people running to get on than Intercities, which might have a stricter schedule.

brundolf 6y ago

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

brlewis 6y ago
>very experienced developers in their 40s I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.
- BalinKing 6y ago
  This may not have been their first time, though :-P
  - __d 6y ago
    Some time ago, it was common in Unix sites to have an NFS filesystem mounted on all machines that contained locally-built binaries to augment those provided by the operating system. At this site, we used a bunch of different platforms: OSF/1, Solaris, Linux, HP/UX, etc. So we had a large filesystem containing the source code, and built binaries for all the different platforms, and this included heaps of things, from Bash upwards. A colleague of mine accidentally ran rm -rf on this filesystem. It was taking a loooong time, so he realised and killed it, but not before it had removed a heap of stuff. Because this was something that could be rebuilt, it wasn't backed up, so we had to go through the process of downloading the tarballs, and recompiling everything for all the different platforms. It took a few days to recover most of it, and weeks to completely restore things. The day after the incident, when he arrived at work, he found his keyboard was missing a few keycaps. It took him a while to realise that there were four gone: 'R', 'M', '-', and 'F' ... Good times.
- PopeDotNinja 6y ago
  I once cloned a directory for standing up an environment via Terraform. I modified all of the environment variables and config and ran it. It worked perfectly. Except I’d forgotten to wipe out the Terraform state, which meant that in the process of creating a new environment, it completely deleted the environment I had cloned. That was my initiation into very experienced :)

saberdancer 6y ago

OpenShift does this by forcing you to write the name of the project you are about to delete. It was something that used to annoy me but reading this I understand it is a good call from their side.

nailer 6y ago

I do things like const HARD_CODE_TEST_DATABASE_FOR_SAFETY = 'unit-testing' destroyDatabase(HARD_CODE_TEST_DATABASE_FOR_SAFETY) 1. Avoid silly terms our industry should have ditched years ago, like 'drop' 2. Making sure that nobody will ever change HARD_CODE_TEST_DATABASE_FOR_SAFETY because they thought it should 'always be the active database' or whatever.

myself248 6y ago

Ever since hearing about point-and-call, I've started using it in the kitchen when turning on the stove. I used to destroy one or two pans a year by turning on the wrong burner, but it's now been about a year and a half and I haven't screwed it up yet. The knobs are labeled with a terrible little glyph meant to indicate which is which, and I've supplemented this with plain-english Brady labels "front left", "front right", etc. Now I speak the words above the knob, and point to the burner. It felt goofy at first, but now it feels normal, and like I'm tempting fate if I skip it.

giantDinosaur 6y ago
I'm curious how exactly you managed to destroy pans. I've never destroyed a pan in my life, and take no particular precautions - is this a common thing? Is this more common with non-stick stuff or something?
- ajb 6y ago
  Not the op, but non stick pans will burn if the pan is heated while empty.
  - pantalaimon 6y ago
    I think non-stick pans are a fad. A well greased iron or steel pan works much better and is impossible to destroy.
    - rpeden 6y ago
      Can non-stick pans even be a fad when Teflon coated cookware has been popular for 60+ years?
      - 542354234235 6y ago
        Totally a fad. I bet they won't be popular for more than a century or two.
- 6y ago

viraptor 6y ago

I try to do that during incidents. I'm not 100% there since it's no a company rule, but it helps me at the time and later when writing up details: "I see <behaviour X>", "<Y> should fix it because <Z>", "I'm starting to do <Z> now and seeing ...", etc. It also helps when Z results in a total meltdown and you need to pull in more people to help out, so they have context of what happened.

Qu3tzal 6y ago
French firefighters do this when arriving at a scene. The first messages sent over the radio will say: - I am... (who you are and where you are) - I see... (describe what you see in simple non-ambiguous terms) - I do... (what action you are taking now) - I ask... (ask for reinforcements if necessary, you may be asked to justify yourself more)

stjohnswarts 6y ago

I always called it a "that can't be right" interrogative.

xvf22 6y ago

Killed just under 1k access points when they all upgraded on one go. They had no problem erasing the firmware but when they all tried to download the new one at once it killer the service and we ended up with a lot of blank APs. The conformation message for 1 or 1000 APs is unhelpfully "This will overwrite all existing system images. Are you sure Y/N"

acdha 6y ago

Back when I shelled into servers more, I really liked having my deployment put the environment in the prompt and set a red background on production for similar reasons. It only takes a small change to jar you out of habit.

ekanes 6y ago

I do this with my kids, gesturing (not pointing) as it helps my mind remain focused on truly listening to them amid everything else going on. I probably look ridiculous, but I'm a better father for it so ¯\_(ツ)_/¯

justinlloyd 6y ago

I have had many disasters in my software career because I jut wantonly hit "Y" without thinking about it. I have noticed, since learning to cook at a professional level in the kitchen, that I point and call out a lot more in my other activities too. "From hot behind" and "knife" and "oven is over temp" to "Saw blade is live" and "circuit is live" in the workshop to "production server" and "erasing records" in database maintenance. Some days I feel like Sigourney "I have one job damnit" Weaver in Galaxyquest. It's a useful stop-think-go sanity check.

m463 6y ago

> forcing a cache miss in the brain That is an interesting way of looking at it. I think a router analogy might be more precise - more like fast path / slow path - where when most packets come in they hit the fast path in hardware, and slow path exception packets get handled by the cpu. :)