7 ms·

Ask HN: Please help me recall a certain programmer horror story

155 points by kuba-orlik 8y ago

I remember having read a story about a programmer that encountered a weird error - his program was spitting out racial slurs in its output. The issue turned out to be a malicious version of the compiler, but somehow after removing it it got back to the system, and what should take only a few minutes turned into days and days of trying to find the culprit. I've been searching long and hard but cannot seem to find it. I hope that what little I remember from it is enough and some kind soul will help me find the original article online... :)

pliny 8y ago
https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute https://www.quora.com/What-is-a-coders-worst-nightmare/answe...
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- pmichaud 8y ago
  Jesus, that is intense. I wonder if the guy was personally out to get the Doc, or if he was just a smart kid with too much time and not enough superego?
  - kazinator 8y ago
    He did make it super easy to find this by having the compiler modify source code. If you run "cc file.c" and file.c is overwritten, you should know that it's the "cc" doing it. It would have been more evil to do it the original way, as described in Thompson's paper: recognize the program sources and emit altered object code only; no mucking with the source.
  - rexpop 8y ago
    Interesting that the first adjective you use was "smart" when, in any decent scope of consideration, he has actually wasted many hours of others' time, and caused only suffering. When I do that at my company, they do not call me "smart."
    - hhmc 8y ago
      Cleverness and morality can be orthogonal.
      - jmmcd 8y ago
        Bostrom's orthogonality thesis!
      - rexpop 8y ago
        Of course they can, but when I see an act of ingenious evil, my first response is not circumscribed admiration.
        breakingcups 8y ago
        Concluding someone is smart doesn't always equal admiring them.
    - webmaven 8y ago
      If you do it to your company's competitor, you may qualify for a massive bonus (modulo your employer's moral compass and appetite for risk).
- MrEldritch 8y ago
  Thank you! I saw Ken Thomson's classic Reflections on Trusting Trust linked on HN a couple weeks ago, and was looking forward to rereading this very same story when I was surprised to discover it wasn't and had never been part of that essay at all. I had no idea where to find the actual story I'd been looking for! (Indeed, I was no longer sure this story had actually existed, and wasn't just something my brain had made up from cobbling together fragmentary memories) Thank you for solving this mystery. :)
- stefs 8y ago
  ah yes i remember this. a practical implementation of the http://wiki.c2.com/?TheKenThompsonHack http://wiki.c2.com/?TheKenThompsonHack ?
- TheArcane 8y ago
  > Because the compiler was poisoned with other source code that we didn't have. And that source code, that now existed only in the executable compiler, put those changes back into the compiler source before it compiled it. I don't quite understand this part. How would this work?
  - antsar 8y ago
    If I read it correctly, they used the binary compiler which existed on the machine to compile a new copy of the compiler (from clean source). This existing binary compiler was "poisoned" (modified) such that it would not only poison the psychiatrist's program, but also poison any attempts to compile a new copy of the compiler. So when they recompiled the compiler, the new copy was already poisoned (because they used the old, poisoned compiler to do it).
  - ChristianBundy 8y ago
    You need a compiler to compile anything, including a compiler, so if your compiler was tampered with then anything you compile can be tampered with as well. The only solution is be to download a binary compiler from someone you trust.
    - stefs 8y ago
      yes, but ken thompsons evil compiler theory states, that a trusted compiler is technically impossible (in the paranoid world of theoretical security where all compiler binaries are possibly tainted already) except if you write your own bootstrap compiler chain in machine code. still, all other running software (OS/drivers/the intel management engine) might be able to modify the binary in memory on the fly. the rabbit hole goes down to the hardware level.
      - emilfihlman 8y ago
        It doesn't even stop at hardware exactly (mcu/cpu), it only stops when you assemble the mcu/cpu yourself.
- kuba-orlik 8y ago
  Thank you, that's it!
- tim333 8y ago
  Some HN discussion on that (224 comments) https://news.ycombinator.com/item?id=10606226 https://news.ycombinator.com/item?id=10606226
jlebrech 8y ago
that would be a pretty evil way to enforce payment, aka a DRM that turns racist if you don't pay up.
- jackhack 8y ago
  It's an equally effective way to end a career (perhaps before it's even begun, as this is a graduate student's work we're talking of). Regardless, it's a stunningly bold lack of judgement. But then, bad decisions often make for interesting stories.
  - rexpop 8y ago
    Is it "bad judgement"? Aren't there, perhaps, better words for this phenomenon? Is all hate speech mere "bad judgement"? Is KKK membership, cross-burning, lynching, etc. all circumscribed by the label "bad judgement"?
    - turc1656 8y ago
      That's an extraordinarily disingenuous take on that comment. They were clearly referring to the decision to mess with the code in this way at all, not the actual content of what was being displayed on screen. The point was obviously that deciding to use this method to cause this amount of effort/trouble and mess with the computer system belonging to someone else is the act of bad judgment. The fact they chose to do it with white supremacist messages is irrelevant to the original comment.
      - AnimalMuppet 8y ago
        I don't think I agree with you. jackhack's comment is in reply to jlebrech's comment, which explicitly mentioned racism.
      - jackhack 8y ago
        You summarized my thoughts pretty well. I would have thought it obvious but I forget sometimes what a hypersensitive world we live in, and how quickly some want to rush to judgement.
dwheeler 8y ago
I mentioned this "malicious version of compiler" on my web page about my approach for countering the trusting trust attack. See: https://www.dwheeler.com/trusting-trust/ https://www.dwheeler.com/trusting-trust/
dzdt 8y ago
This story strikes me as likely to be exaggerated or invented. It is exactly the scenario described in Ken Thomson's Reflections on Trusting Trust. But unlike Thomson, who was developing the compiper, there is no reason a random grad student would have access to recompile the compiler. And if the student gained root access to replace system binaries, why would they bother making the compiler reinfect itself on recompilation?
- microtherion 8y ago
  The hardware mentioned was an AT&T 3B2, so this probably would have taken place in the early 1980s. Access policies for such things could be quite different, back then. Even when I was a graduate student a decade later, in a lab with professional administrators, grad students routinely had pretty much all the permissions they would ask for, because we worked odd hours and performed exotic experiments. Policing was done after the fact. To have a grad student in Psychology with that level of programming skill would be rather unusual, though.
  - TomVDB 8y ago
    In the eighties, all engineering students at my university had access to an IBM 3090 mainframe. There were some CS kids who were way better than the admins at exploiting the system. (I don't remember the details, but the student newspaper once wrote about an exploit that required a coordinated attack with the mainframe of a different university.) The admins were smart enough to befriend those kids and let them help manage the system. Better to have them to your side than to have them as an adversary.
swarnie_ 8y ago
I didn't recognize the story from your description but after the first paragraph its unmistakable. A good, fun read. Thank you!
marcodave 8y ago
Related to the question, another nice short story posted some time ago on HN about hacked compiler code https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
pressurefree 8y ago
never herd of it

155 points by kuba-orlik 8y ago

pliny 8y ago

https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute https://www.quora.com/What-is-a-coders-worst-nightmare/answe...

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
pmichaud 8y ago
Jesus, that is intense. I wonder if the guy was personally out to get the Doc, or if he was just a smart kid with too much time and not enough superego?
- kazinator 8y ago
  He did make it super easy to find this by having the compiler modify source code. If you run "cc file.c" and file.c is overwritten, you should know that it's the "cc" doing it. It would have been more evil to do it the original way, as described in Thompson's paper: recognize the program sources and emit altered object code only; no mucking with the source.
- rexpop 8y ago
  Interesting that the first adjective you use was "smart" when, in any decent scope of consideration, he has actually wasted many hours of others' time, and caused only suffering. When I do that at my company, they do not call me "smart."
  - hhmc 8y ago
    Cleverness and morality can be orthogonal.
    - jmmcd 8y ago
      Bostrom's orthogonality thesis!
    - rexpop 8y ago
      Of course they can, but when I see an act of ingenious evil, my first response is not circumscribed admiration.
      - breakingcups 8y ago
        Concluding someone is smart doesn't always equal admiring them.
  - webmaven 8y ago
    If you do it to your company's competitor, you may qualify for a massive bonus (modulo your employer's moral compass and appetite for risk).
MrEldritch 8y ago
Thank you! I saw Ken Thomson's classic Reflections on Trusting Trust linked on HN a couple weeks ago, and was looking forward to rereading this very same story when I was surprised to discover it wasn't and had never been part of that essay at all. I had no idea where to find the actual story I'd been looking for! (Indeed, I was no longer sure this story had actually existed, and wasn't just something my brain had made up from cobbling together fragmentary memories) Thank you for solving this mystery. :)
stefs 8y ago
ah yes i remember this. a practical implementation of the http://wiki.c2.com/?TheKenThompsonHack http://wiki.c2.com/?TheKenThompsonHack ?
TheArcane 8y ago
> Because the compiler was poisoned with other source code that we didn't have. And that source code, that now existed only in the executable compiler, put those changes back into the compiler source before it compiled it. I don't quite understand this part. How would this work?
- antsar 8y ago
  If I read it correctly, they used the binary compiler which existed on the machine to compile a new copy of the compiler (from clean source). This existing binary compiler was "poisoned" (modified) such that it would not only poison the psychiatrist's program, but also poison any attempts to compile a new copy of the compiler. So when they recompiled the compiler, the new copy was already poisoned (because they used the old, poisoned compiler to do it).
- ChristianBundy 8y ago
  You need a compiler to compile anything, including a compiler, so if your compiler was tampered with then anything you compile can be tampered with as well. The only solution is be to download a binary compiler from someone you trust.
  - stefs 8y ago
    yes, but ken thompsons evil compiler theory states, that a trusted compiler is technically impossible (in the paranoid world of theoretical security where all compiler binaries are possibly tainted already) except if you write your own bootstrap compiler chain in machine code. still, all other running software (OS/drivers/the intel management engine) might be able to modify the binary in memory on the fly. the rabbit hole goes down to the hardware level.
    - emilfihlman 8y ago
      It doesn't even stop at hardware exactly (mcu/cpu), it only stops when you assemble the mcu/cpu yourself.
kuba-orlik 8y ago
Thank you, that's it!
tim333 8y ago
Some HN discussion on that (224 comments) https://news.ycombinator.com/item?id=10606226 https://news.ycombinator.com/item?id=10606226

jlebrech 8y ago

that would be a pretty evil way to enforce payment, aka a DRM that turns racist if you don't pay up.

jackhack 8y ago
It's an equally effective way to end a career (perhaps before it's even begun, as this is a graduate student's work we're talking of). Regardless, it's a stunningly bold lack of judgement. But then, bad decisions often make for interesting stories.
- rexpop 8y ago
  Is it "bad judgement"? Aren't there, perhaps, better words for this phenomenon? Is all hate speech mere "bad judgement"? Is KKK membership, cross-burning, lynching, etc. all circumscribed by the label "bad judgement"?
  - turc1656 8y ago
    That's an extraordinarily disingenuous take on that comment. They were clearly referring to the decision to mess with the code in this way at all, not the actual content of what was being displayed on screen. The point was obviously that deciding to use this method to cause this amount of effort/trouble and mess with the computer system belonging to someone else is the act of bad judgment. The fact they chose to do it with white supremacist messages is irrelevant to the original comment.
    - AnimalMuppet 8y ago
      I don't think I agree with you. jackhack's comment is in reply to jlebrech's comment, which explicitly mentioned racism.
    - jackhack 8y ago
      You summarized my thoughts pretty well. I would have thought it obvious but I forget sometimes what a hypersensitive world we live in, and how quickly some want to rush to judgement.

dwheeler 8y ago

I mentioned this "malicious version of compiler" on my web page about my approach for countering the trusting trust attack. See: https://www.dwheeler.com/trusting-trust/ https://www.dwheeler.com/trusting-trust/

dzdt 8y ago

This story strikes me as likely to be exaggerated or invented. It is exactly the scenario described in Ken Thomson's Reflections on Trusting Trust. But unlike Thomson, who was developing the compiper, there is no reason a random grad student would have access to recompile the compiler. And if the student gained root access to replace system binaries, why would they bother making the compiler reinfect itself on recompilation?

microtherion 8y ago
The hardware mentioned was an AT&T 3B2, so this probably would have taken place in the early 1980s. Access policies for such things could be quite different, back then. Even when I was a graduate student a decade later, in a lab with professional administrators, grad students routinely had pretty much all the permissions they would ask for, because we worked odd hours and performed exotic experiments. Policing was done after the fact. To have a grad student in Psychology with that level of programming skill would be rather unusual, though.
- TomVDB 8y ago
  In the eighties, all engineering students at my university had access to an IBM 3090 mainframe. There were some CS kids who were way better than the admins at exploiting the system. (I don't remember the details, but the student newspaper once wrote about an exploit that required a coordinated attack with the mainframe of a different university.) The admins were smart enough to befriend those kids and let them help manage the system. Better to have them to your side than to have them as an adversary.

swarnie_ 8y ago

I didn't recognize the story from your description but after the first paragraph its unmistakable. A good, fun read. Thank you!

marcodave 8y ago

Related to the question, another nice short story posted some time ago on HN about hacked compiler code https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/

pressurefree 8y ago

never herd of it