7 ms·

The strings used for debugging at Google were supposed to include “DO NOT SUBMIT” which, a bot to check for this was added after a debug strings made it to prod

The strings used for debugging at Google were supposed to include “DO NOT SUBMIT” which, a bot to check for this was added after a debug strings made it to production (It was before my time but I believe it was after the legendary WD incident- “western digital” was remapped to “woodly doodly” in test code that accidentally shipped for those curious)

riffraff 2y ago
Was the debug string that made it to production embarrassing? I remember a swearword appearing in a very big national company login form once, and I'd have felt sorry for the dev if it wasn't so hilarious.
- PetahNZ 2y ago
  Yea, I have seen a very inappropriate debug statement being left in the led to the dev being promptly fired.
  - arethuza 2y ago
    I was leading a startup development team in the late 90's and one of our developers told me that some very inappropriate test data had been sent to IBM to be included in a product demo and burned onto tens of thousands of discs that they would distribute. Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless. Talk about a moment when I thought my career had crashed and burned...
    - MonkeyClub 2y ago
      > Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless. Please, do contextualize it?
  - HaZeust 2y ago
    Likewise; they tried to pin it on me, and I had to go to the Git commit history to show it was another employee. My administrative leave was promptly revoked.
- mattnewton 2y ago
  “Western Digital” was replaced with “Woodly Doodly” on the srp. Not really thag embarassing but I am sure WD didn’t appreciate it
raverbashing 2y ago
Pro tip: never add a "funny comment" especially an error message that's "never supposed to happen" unless you're happy with that message showing up in a board meeting, in front of your boss or worse (and it being traced back to you)
- mackman 2y ago
  Listen. If it’s never supposed to happen then you definitely want a customer to tell you if it happens. Best way to make sure that happens is to insult their mum. It’s just good engineering discipline.
  - bartread 2y ago
    In my younger and more foolish years I worked for a company called Redgate and, whilst there, I had to do some licensing jiggery pokery that stopped people using our tools if the licensing code had been tampered with or removed. I didn't want to make it too easy for people trying to crack our tools so, if such a problem were detected, the error message spat out to the console bore no relation to what had gone wrong. By "bore no relation" I mean the error message would be one of a number of lines of dialogue from a scene in a House episode where House is trying to figure out who Wilson's girlfriend is... which led to this: https://gist.github.com/gregoryyoung/871736 https://gist.github.com/gregoryyoung/871736 Yes, Greg Young (the event sourcing guy) ran into an error that said "Because I wanna ask you about your girlfriend. I must know who she is, or you would've told me her name," because he'd moved Smartassembly minus the licensing DLLs. Of course, he didn't realise what he'd done "wrong" so he tweeted it to his considerable following and it went viral. Because I'm exactly the sort of walking cliche you'd expect I was blissfully unaware of the unfolding drama, being away on sabbatical snowboarding at the time. I hadn't taken my laptop with me, I'd disabled data roaming on my phone to avoid a whopper of a bill after a couple of months away, and had only a gen 1 iPad to access email and internet. So of course I came back in the evening to discover that it had all kicked off, with this ludicrous chain of emails where people were trying to figure out what was going on, until somebody had the sense to go and look in our source control system (which at the time was still svn for most things), and figure out that it was my fault. People saw the funny side and it wasn't that big a deal but the reason I bring this up is that the last line of dialogue in that scene is, "Your mama," delivered by Wilson to House, and it was also the error message for the final failure scenario I was looking for. I'm glad that one didn't get printed.
smitelli 2y ago
> "DO NOT SUBMIT" Kinda verbose, ain't it? Just speaking from my own personal experience, usually when I resort to print-debugging I'm already pretty punchy and more likely to use a quick "ASDFASD" or similar.
- bipson 2y ago
  Yes, but you can check for "DO NOT SUBMIT" with automation. You can't automate checking for random strings, right?
  - neutronicus 2y ago
    Perhaps an abbreviation would be the best of both worlds, and debug strings should be prefixed with "DNS" You won't need to submit that particular string working at Google, right?
    - tetha 2y ago
      We could also have an acronym for the more severe "DO NOT SUBMIT - SECURITY EXPECTATIONS COMPROMISE".
      - pixl97 2y ago
        DNS-SEC... Confused sysadmins wondering if this is SOX code...
  - Tyr42 2y ago
    The automation which can check for do not submit itself is hard to submit. Or at least updates to it are hard to submit.
    - Dylan16807 2y ago
      forbidden_string = "DO NOT " + "SUBMIT" Seems easy enough?
    - organsnyder 2y ago
veunes 2y ago
It's amazing how many processes evolve because of incidents like these.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
indrora 2y ago
When I worked as a technical writer earlier in my career, we had a debug/prod check: If you used a series of words it would fail you in prod (refused to build). These were usually codenames of projects and such. One afternoon, I added the phrase "[Ww]ombo\s[Cc]ombo" (it supported regular expressions) and an expansion set (&prodbomb; evaluated to &wca; &wcb; to create "womb combo") but only in prod. In debug builds, it would produce "Wombat Combat"in bright red text. This was a useful way to mark pages as "Don't build this in prod yet" - a habit some teams had of just yolo'ing their development branches right into production instead of doing a rebase to main. More than one build was saved by it within a few months.
thephyber 2y ago
All of the comments seem to talk about automation for detecting sloppy logging. Is nobody reviewing code before merging it upstream?
- mattnewton 2y ago
  Belt and suspenders. Every line was reviewed but that doesn’t mean it would be caught every time. And if it’s automatable to catch things like this it should be.
  - thephyber 2y ago
    Right, but I’m pointing out that everybody is using a belt and nobody suspenders.
CPLX 2y ago
> "DO NOT SUBMIT" That's pretty good advice if you find yourself dealing with a large tech company.

riffraff 2y ago

Was the debug string that made it to production embarrassing? I remember a swearword appearing in a very big national company login form once, and I'd have felt sorry for the dev if it wasn't so hilarious.

PetahNZ 2y ago
Yea, I have seen a very inappropriate debug statement being left in the led to the dev being promptly fired.
- arethuza 2y ago
  I was leading a startup development team in the late 90's and one of our developers told me that some very inappropriate test data had been sent to IBM to be included in a product demo and burned onto tens of thousands of discs that they would distribute. Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless. Talk about a moment when I thought my career had crashed and burned...
  - MonkeyClub 2y ago
    > Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless. Please, do contextualize it?
- HaZeust 2y ago
  Likewise; they tried to pin it on me, and I had to go to the Git commit history to show it was another employee. My administrative leave was promptly revoked.
mattnewton 2y ago
“Western Digital” was replaced with “Woodly Doodly” on the srp. Not really thag embarassing but I am sure WD didn’t appreciate it

raverbashing 2y ago

Pro tip: never add a "funny comment" especially an error message that's "never supposed to happen" unless you're happy with that message showing up in a board meeting, in front of your boss or worse (and it being traced back to you)

mackman 2y ago
Listen. If it’s never supposed to happen then you definitely want a customer to tell you if it happens. Best way to make sure that happens is to insult their mum. It’s just good engineering discipline.
- bartread 2y ago
  In my younger and more foolish years I worked for a company called Redgate and, whilst there, I had to do some licensing jiggery pokery that stopped people using our tools if the licensing code had been tampered with or removed. I didn't want to make it too easy for people trying to crack our tools so, if such a problem were detected, the error message spat out to the console bore no relation to what had gone wrong. By "bore no relation" I mean the error message would be one of a number of lines of dialogue from a scene in a House episode where House is trying to figure out who Wilson's girlfriend is... which led to this: https://gist.github.com/gregoryyoung/871736 https://gist.github.com/gregoryyoung/871736 Yes, Greg Young (the event sourcing guy) ran into an error that said "Because I wanna ask you about your girlfriend. I must know who she is, or you would've told me her name," because he'd moved Smartassembly minus the licensing DLLs. Of course, he didn't realise what he'd done "wrong" so he tweeted it to his considerable following and it went viral. Because I'm exactly the sort of walking cliche you'd expect I was blissfully unaware of the unfolding drama, being away on sabbatical snowboarding at the time. I hadn't taken my laptop with me, I'd disabled data roaming on my phone to avoid a whopper of a bill after a couple of months away, and had only a gen 1 iPad to access email and internet. So of course I came back in the evening to discover that it had all kicked off, with this ludicrous chain of emails where people were trying to figure out what was going on, until somebody had the sense to go and look in our source control system (which at the time was still svn for most things), and figure out that it was my fault. People saw the funny side and it wasn't that big a deal but the reason I bring this up is that the last line of dialogue in that scene is, "Your mama," delivered by Wilson to House, and it was also the error message for the final failure scenario I was looking for. I'm glad that one didn't get printed.

smitelli 2y ago

> "DO NOT SUBMIT" Kinda verbose, ain't it? Just speaking from my own personal experience, usually when I resort to print-debugging I'm already pretty punchy and more likely to use a quick "ASDFASD" or similar.

bipson 2y ago
Yes, but you can check for "DO NOT SUBMIT" with automation. You can't automate checking for random strings, right?
- neutronicus 2y ago
  Perhaps an abbreviation would be the best of both worlds, and debug strings should be prefixed with "DNS" You won't need to submit that particular string working at Google, right?
  - tetha 2y ago
    We could also have an acronym for the more severe "DO NOT SUBMIT - SECURITY EXPECTATIONS COMPROMISE".
    - pixl97 2y ago
      DNS-SEC... Confused sysadmins wondering if this is SOX code...
- Tyr42 2y ago
  The automation which can check for do not submit itself is hard to submit. Or at least updates to it are hard to submit.
  - Dylan16807 2y ago
    forbidden_string = "DO NOT " + "SUBMIT" Seems easy enough?
  - organsnyder 2y ago

veunes 2y ago

It's amazing how many processes evolve because of incidents like these.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

indrora 2y ago

When I worked as a technical writer earlier in my career, we had a debug/prod check: If you used a series of words it would fail you in prod (refused to build). These were usually codenames of projects and such. One afternoon, I added the phrase "[Ww]ombo\s[Cc]ombo" (it supported regular expressions) and an expansion set (&prodbomb; evaluated to &wca; &wcb; to create "womb combo") but only in prod. In debug builds, it would produce "Wombat Combat"in bright red text. This was a useful way to mark pages as "Don't build this in prod yet" - a habit some teams had of just yolo'ing their development branches right into production instead of doing a rebase to main. More than one build was saved by it within a few months.

thephyber 2y ago

All of the comments seem to talk about automation for detecting sloppy logging. Is nobody reviewing code before merging it upstream?

mattnewton 2y ago
Belt and suspenders. Every line was reviewed but that doesn’t mean it would be caught every time. And if it’s automatable to catch things like this it should be.
- thephyber 2y ago
  Right, but I’m pointing out that everybody is using a belt and nobody suspenders.

CPLX 2y ago

> "DO NOT SUBMIT" That's pretty good advice if you find yourself dealing with a large tech company.