5 ms·

I'd be interested if anyone has any anecdotes that compare from software engineering. Sending a car out with suspension thingamies still attached seems like the

I'd be interested if anyone has any anecdotes that compare from software engineering. Sending a car out with suspension thingamies still attached seems like the equivalent of pushing a build to production with `DEBUG = True`, and I'm not sure much useful can be learned by you in that situation.

pjc50 9y ago
If you don't learn something from pushing a debug build to production, your debug build isn't instrumented enough. Maybe all you learn is what the effect of building with/without optimisation is on the real workload. There's practically an entire genre of "here's what I learned from my expensive mistake" stories. Usually they teach humility and caution. This one is interesting because it's a variant of YAGNI. Imagine that you've carefully tuned your (software/car) for a specific set of parameters, then throw one out of whack by mistake - does this get caught by monitoring? Do you even notice? If you hillclimb from the new status to an optimal one, do you get back to the same local optimum? Then of course there's the famous RISKS digest, which can be funny or depressing depending on your mood and dependence on complex systems. (My own personal one of these? "killall" behaves differently on Solaris than it does on Linux ..)
- lathiat 9y ago
  I'd also strongly suggest not typing "exit" into kdb, the Solaris debugger when it is in write mode. It sets the global pointer for exit (the libc function) to NULL and the whole system will crash a few seconds to minutes later when the next process exits. (It's not uncommon to use this tool to reconfigure some configuration values at runtime) At least not on Solaris 10 or any of the open source derivatives. Seems it was special cased at some point in Solaris 11. Had to make that mistake 2-3 times before I finally realised what I was doing to crash it given the slight delay.
  - oblio 9y ago
    Solaris, the system were ease of use is an enemy.
    - ch4ck 9y ago
      Unlike other Unixen...
      - oblio 9y ago
        GNU CLI stuff is in terms of ease of use leagues ahead of most Unixes and even the BSDs. The BSDs at least make it up by orderly docs. Tiny example, GNU ls: ls -lh bla : ok ls bla -lh : ok Most non-GNU lses, second form : oh-oh! :)
lr4444lr 9y ago
You could learn from 'DEBUG=True' that you haven't automated your production deploy process carefully enough, including but not limited to testing failure cases. Raising the priority on that and making it part of routine work can save you from innumerable costly boneheaded mistaken deploys in the future.
- fredley 9y ago
  I understand this, I guess what I'm trying to get at is that the learning from the article was more than just procedural (I.e how can we make sure this doesn't happen again), and more about the system itself.
friendzis 9y ago
Particularities elided, I was working on a POC of pretty critical (possible damage) closed loop (tightly controlled by operator at the time) system - basically read inputs, react according to algorithm, rinse and repeat. Inputs, however, came from multiple unsynchronised sources and we could arrive at situations where output depended on multiple inputs. I guess we took pretty natural decision to use latest available value of inputs when needed. During one test run, one input did not update, we were happily feeding stale input to algorithm, which in turn took control paths that could have caused severe damages in unsupervised environment. Lessons learned: if you rely on invariants about inputs in code (update interval in this case), guard against violations of those invariants. This helped me many times later.
- mooreds 9y ago
  So basically treat even invariants as possibly varying.
  - jeremiep 9y ago
    Invariants are still invariants. You just want some guarantee that they actually hold. Every time you use assert() somewhere you're doing just that.
  - friendzis 9y ago
    Not necessarily. Very often we extract some invariants from data and embed them in code because it makes code much cleaner, easier to write, etc.. The problem is that some things that we assume (sometimes explicitly, sometimes by mistake) to be invariants are not necessarily invariants after all. Very, very unlikely to change, but not invariants nevertheless. I guess rather well-known real world example is systems starting to show weird behaviour when database resultset ordering changes (e.g. changes in indexing) and someone relies on correct entry being first/last. Assumed invariant here is resultset ordering and code is written with this particular invariant in mind. If not enough ordering clauses are used, resultset ordering is not invariant, even though it worked for years already :)
rconti 9y ago
Google "VW shipping puck" and you'll see this actually happens in the real world all the time, too!
lmm 9y ago
We broke one API for 2 months, and thereby discovered that no-one was actually using it and so removed it. We left some database healthchecks running on every request for a while, making load times a factor of 10 longer than they needed to be. Learnt our stuff was really fast and we didn't need to worry about performance much, I guess.
groby_b 9y ago
We occasionally ship a build with the equivalent of 'DEBUG=true' to (parts of) production. Deliberately. For an extremely short time period. What that gains us is a run with real data and a build that's instrumented with assertions - which flushes out a lot of previously unknown issues. So yes, you learn useful things. If you ever did it accidentally, the useful thing you learn is that shipping ideally is not an on/off switch, but a carefully monitored process :)
- fredley 9y ago
  That's fascinating. Is there any reason to do this in production vs. a staging environment (assuming you've got one prod-like enough with replayed traffic)? It seems a little risky to me!
  - groby_b 9y ago
    It's sort-of a staging, sort-of production environment :) I'm working on desktop software, and this happens to builds that are full production builds, but that don't go to stable. So a very small subset of our users - who has agreed to being guinea pigs - occasionally gets these. It also helps that we have a clear policy that assertions can only fire in cases that truly would be a bug in the wild - you can't assert and handle the issue. Either you think it might happen, then you handle it - or you state an invariant, in which case there's no reason to handle it being false. In other words, the only price you pay is that your build is slower - any assertion that fires would've been a silent bug before anyways. And, to be as close to non-debug as possible, any assertion firing merely creates a crash dump, it doesn't actually crash the executable. And finally, we simply need to do it in the wild because many of our bugs are simply due to the staggering number of environments desktop software encounters.

pjc50 9y ago

If you don't learn something from pushing a debug build to production, your debug build isn't instrumented enough. Maybe all you learn is what the effect of building with/without optimisation is on the real workload. There's practically an entire genre of "here's what I learned from my expensive mistake" stories. Usually they teach humility and caution. This one is interesting because it's a variant of YAGNI. Imagine that you've carefully tuned your (software/car) for a specific set of parameters, then throw one out of whack by mistake - does this get caught by monitoring? Do you even notice? If you hillclimb from the new status to an optimal one, do you get back to the same local optimum? Then of course there's the famous RISKS digest, which can be funny or depressing depending on your mood and dependence on complex systems. (My own personal one of these? "killall" behaves differently on Solaris than it does on Linux ..)

lathiat 9y ago
I'd also strongly suggest not typing "exit" into kdb, the Solaris debugger when it is in write mode. It sets the global pointer for exit (the libc function) to NULL and the whole system will crash a few seconds to minutes later when the next process exits. (It's not uncommon to use this tool to reconfigure some configuration values at runtime) At least not on Solaris 10 or any of the open source derivatives. Seems it was special cased at some point in Solaris 11. Had to make that mistake 2-3 times before I finally realised what I was doing to crash it given the slight delay.
- oblio 9y ago
  Solaris, the system were ease of use is an enemy.
  - ch4ck 9y ago
    Unlike other Unixen...
    - oblio 9y ago
      GNU CLI stuff is in terms of ease of use leagues ahead of most Unixes and even the BSDs. The BSDs at least make it up by orderly docs. Tiny example, GNU ls: ls -lh bla : ok ls bla -lh : ok Most non-GNU lses, second form : oh-oh! :)

lr4444lr 9y ago

You could learn from 'DEBUG=True' that you haven't automated your production deploy process carefully enough, including but not limited to testing failure cases. Raising the priority on that and making it part of routine work can save you from innumerable costly boneheaded mistaken deploys in the future.

fredley 9y ago
I understand this, I guess what I'm trying to get at is that the learning from the article was more than just procedural (I.e how can we make sure this doesn't happen again), and more about the system itself.

friendzis 9y ago

Particularities elided, I was working on a POC of pretty critical (possible damage) closed loop (tightly controlled by operator at the time) system - basically read inputs, react according to algorithm, rinse and repeat. Inputs, however, came from multiple unsynchronised sources and we could arrive at situations where output depended on multiple inputs. I guess we took pretty natural decision to use latest available value of inputs when needed. During one test run, one input did not update, we were happily feeding stale input to algorithm, which in turn took control paths that could have caused severe damages in unsupervised environment. Lessons learned: if you rely on invariants about inputs in code (update interval in this case), guard against violations of those invariants. This helped me many times later.

mooreds 9y ago
So basically treat even invariants as possibly varying.
- jeremiep 9y ago
  Invariants are still invariants. You just want some guarantee that they actually hold. Every time you use assert() somewhere you're doing just that.
- friendzis 9y ago
  Not necessarily. Very often we extract some invariants from data and embed them in code because it makes code much cleaner, easier to write, etc.. The problem is that some things that we assume (sometimes explicitly, sometimes by mistake) to be invariants are not necessarily invariants after all. Very, very unlikely to change, but not invariants nevertheless. I guess rather well-known real world example is systems starting to show weird behaviour when database resultset ordering changes (e.g. changes in indexing) and someone relies on correct entry being first/last. Assumed invariant here is resultset ordering and code is written with this particular invariant in mind. If not enough ordering clauses are used, resultset ordering is not invariant, even though it worked for years already :)

rconti 9y ago

Google "VW shipping puck" and you'll see this actually happens in the real world all the time, too!

lmm 9y ago

We broke one API for 2 months, and thereby discovered that no-one was actually using it and so removed it. We left some database healthchecks running on every request for a while, making load times a factor of 10 longer than they needed to be. Learnt our stuff was really fast and we didn't need to worry about performance much, I guess.

groby_b 9y ago

We occasionally ship a build with the equivalent of 'DEBUG=true' to (parts of) production. Deliberately. For an extremely short time period. What that gains us is a run with real data and a build that's instrumented with assertions - which flushes out a lot of previously unknown issues. So yes, you learn useful things. If you ever did it accidentally, the useful thing you learn is that shipping ideally is not an on/off switch, but a carefully monitored process :)

fredley 9y ago
That's fascinating. Is there any reason to do this in production vs. a staging environment (assuming you've got one prod-like enough with replayed traffic)? It seems a little risky to me!
- groby_b 9y ago
  It's sort-of a staging, sort-of production environment :) I'm working on desktop software, and this happens to builds that are full production builds, but that don't go to stable. So a very small subset of our users - who has agreed to being guinea pigs - occasionally gets these. It also helps that we have a clear policy that assertions can only fire in cases that truly would be a bug in the wild - you can't assert and handle the issue. Either you think it might happen, then you handle it - or you state an invariant, in which case there's no reason to handle it being false. In other words, the only price you pay is that your build is slower - any assertion that fires would've been a silent bug before anyways. And, to be as close to non-debug as possible, any assertion firing merely creates a crash dump, it doesn't actually crash the executable. And finally, we simply need to do it in the wild because many of our bugs are simply due to the staggering number of environments desktop software encounters.