8 ms·

The coder's worst nightmare (2016)

76 points by otagekki 6y ago

wccrawford 6y ago
Holy crap. That went way deeper than I expected it to. About halfway through, I was ready to copy the source to another computer and compile it there instead, and then suggest they nuke the original computer. Turns out I was right. I just didn't know how right I was, which could be important for whatever they plan to do to the person who hacked that computer.
- hbrav 6y ago
  Given the computer model involved, this is probably in the late 80s. If you've read The Cuckoo's Egg, it suggests that getting law enforcement to take computer intrusion seriously was very hard back then. I suppose the university could impose some kind of sanctions such as revoking the student's degree.
SideburnsOfDoom 6y ago
This is modelled on "Reflections on Trusting Trust" by Ken Thompson, 1984 https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- isaacimagine 6y ago
  That's what I was thinking while reading this article, +1! In short, the premise of the talk is: if you add a self replicating change to any self-hosted compiler: - You can 'teach' the compiler to do things that are never explicitly stated in their source code, like, in this article, making the compiler modify programs that meet specific predicates. - You can not trust that your compiler hasn't been modified without bootstrapping it yourself. You can't trust a program without trusting its compiler; and since compilers are programs, well it's turtles all the way down. To build a trusted compiler, you have to bootstrap one yourself from some trusted manner, usually from hand-written assembly. In C, this would be akin to writing a small C compiler in assembly, using that to compile tcc, using that to compile a slightly larger C compiler, and that to compile clang or gcc. But in other more complex languages, such as Rust, bootstrapping is not as clear cut. There's the Reproducible Builds project, but afaict there exists no minimal small Rust compiler you can bootstrap the latest one from: You have to go back the the original OCaml one and compile up from there, or just trust that the publicly available build is secure (which it most likely is). I guess the lesson is that it's possible to inject hidden self-replicating behavior into any significantly complex system.
  - steveklabnik 6y ago
    You can use mrustc to not have to go back to the original OCaml.
    - isaacimagine 6y ago
      Hmm, that's an interesting project! You'd still have to bootstrap a cpp compiler first, though
      - steveklabnik 6y ago
        Sure, you always have to bootstrap something.
  - sandinmyjoints 6y ago
    I get the theory here, but never having written a compiler myself, I’m curious how easy/difficult it would be to come up with those predicates in practice?
    - isaacimagine 6y ago
      See 'Reflections on Rusting Trust' by Manish Goregaokar (In Pursuit of Laziness). https://manishearth.github.io/blog/2016/12/02/reflections-on-rusting-trust/ https://manishearth.github.io/blog/2016/12/02/reflections-on... He modifies rustc with a predicate that replaces the string 'hello world' with something else. The method used to replicate the change is basically a self-modifying quine.
  - sandinmyjoints 6y ago
    https://github.com/certik/bcompiler/ https://github.com/certik/bcompiler/ is an effort to do this very bootstrapping.
  - SAI_Peregrinus 6y ago
    An assembler can also do this same attack. You have to hand-assemble your basic assembler to machine code. In theory a sophisticated-enough hex editor could also be subverted, silently adding in code when it's used to edit executable-looking files. The system dynamic linker can be compromised to subvert any executable. Etc. If a machine is compromised, it can't be recovered without access to a known-good machine.
- wantoncl 6y ago
  Coincidentally, Computerphile just posted this today: https://www.youtube.com/watch?v=SJ7lOus1FzQ https://www.youtube.com/watch?v=SJ7lOus1FzQ
Zealotux 6y ago
>I was hired by a psychologist to fix a program that seemed to have "strange output" written by one of his ex-grad students. >printed some white supremacy message So I'm missing some details here maybe, but isn't that easier to ask the ex-student? He's probably responsible for all of this right? I don't see how anyone could get away with such thing.
- lifeisstillgood 6y ago
  That seems unlikely. From my reading there was a program (a simple research tool to ask people psychology questions), that was written by an "ex-Grad Student". Presumably it worked fine for some time, and then started exhibiting strange behaviour (after all if it was written to do this from the get go who would ever have used it?). If we are going to do detective guess-work (#) get the list of students who took the test. This level of effort to poison a research lab suggests to me the questions raised some uncomfortable issues with one of the respondents, who happened to have compiler abilities and the ability to read the root password on the post-it note on professors desk. (#) which frankly is how most detectives dive in - that one seems suspicious, look for evidence.
- nicklecompte 6y ago
  Considering this was malicious sabotage, I doubt the ex-student was even in contact with his former advisor, and regardless a conversation probably wouldn't be very helpful.
Larrikin 6y ago
This level of dedication to white supremacy is unsurprising to me after spending years on the internet, yet there are so many people who refuse to believe it's a real problem.
- cgist 6y ago
  It is easy to mistake 14-16 year olds with a lot of time on their hands for dedicated real supremacists. The best example was hordes of teenagers teaching a Microsoft chat bot to be a supremacist. Clearly they were having fun. It's their way of rebelling against the establishment by trolling. Since a teenager can no longer troll by assuming a leftist position, they must do so by pretending to be far right.
  - Larrikin 6y ago
    I don't think teenagers being racist is a normal activity that is whimsical fun. There are actual consequences to people. Do you think the black 12 to 16 year old with an interest in technology continues that interest when the chat bot starts yelling racial slurs at him and telling him his life his worthless. It's very easy for people who have never experienced racism to say he should just brush it off. "It was just a joke bro" is also one of the favorite tools of neo nazis and other groups to gauge interest from other people while being able to claim they aren't actually racist. The story is also about a former grad student so he is well into adult hood but probably started off as one of those teenagers just "trolling".
    - bayindirh 6y ago
      > I don't think teenagers being racist is a normal activity that is whimsical fun. Bullies think bullying is fun. Spectators think the event is fun. The bullied one pays the price (don't ask me how I know). Some bullies never grow. They think it's fun, it's a way of expressing themselves. Yes, neither racism nor bullying is normal, but exhibitor of these behaviors normalize it.
    - skocznymroczny 6y ago
      I think most black teenagers have a much thicker skin than you think. Which wouldn't surprise me because most of the outrage about racism seems to be coming from preemptive actions from progressive whites rather than the ones who are supposedly affected by it.
isp 6y ago
Reminds me of the "Robin Hood and Friar Tuck" hack. Excerpts: > They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called 'Robin Hood' and 'Friar Tuck'. Robin Hood and Friar Tuck were designed to run as 'ghost jobs' (daemons, in Unix terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them. > They found the bandit ghost jobs running, and killed them... and were once again surprised. When Robin Hood was gunned, the following sequence of events took place: id1: Friar Tuck... I am under attack! Pray save me! id1: Off (aborted) id2: Fear not, friend Robin! I shall rout the Sheriff of Nottingham's men! id1: Thank you, my good fellow! > Each ghost-job would detect the fact that the other had been killed, and would start a new copy of the recently slain program within a few milliseconds. The only way to kill both ghosts was to kill them simultaneously (very difficult) or to deliberately crash the system. > Finally, the system programmers did the latter - only to find that the bandits appeared once again when the system rebooted! Quoted in the Jargon File: http://www.catb.org/jargon/html/meaning-of-hack.html http://www.catb.org/jargon/html/meaning-of-hack.html Original story: https://www.cs.utah.edu/~elb/folklore/xerox.txt https://www.cs.utah.edu/~elb/folklore/xerox.txt
- dale_glass 6y ago
  A couple things I wonder there: How does Robin (id1) manage to print the "I am under attack" message? Shouldn't a kill be immediate? Couldn't they just do something like sending a SIGSTOP to processes, or use killall in a tight loop? I get it's not Unix, but it seems reasonable similar concepts should exist. Also, this is more in line with Ken Thompson's "Reflections on Trusting Trust"
  - hectorhector 6y ago
    You can put a signal handler in your program to catch the SIGSTOP and run some code before it gets killed.
    - 6y ago
runawaybottle 6y ago
Did this sort of read like a writing-prompt? By day 4 he could have unraveled a larger ploy involving lizard people. Putting the ‘Code’ into DaVinci Code.
midjji 6y ago
Every time I read these stories I wonder if they arent, just that, stories, then I realize it would be kinda fun to create one of these lowlevel compiler infecting things, and realize nah there are atleast a few out there written for shits and giggles. If nothing else, just because of the stories.
- exikyut 6y ago
  The best way to do a modern rendition would be to couple it with some sort of infrastructure vulnerability like a package-squatting attack or similar, so it has a chance to infiltrate for at least a little while before getting killed. The point would be to have it simply persist - and ideally for a little while too - but do nothing useful other than for example allocate an additional string on a local stack somewhere or something. Absolute best case scenario would be to announce its existence one year after successful infiltration. :D
deleted 6y ago
[deleted]
Justsignedup 6y ago
So tl;dr - the original creator created a self-replicating virus that would modify this one program, making it unmodifiable. - actually it was in the compiler. woop. can't fix it without external fixes - creates a backdoor into the system, could have fucked with the whole campus' data - he would have gotten away with it too, if not for the fact that he made this supremacist text So basically, dang.
kazinator 6y ago
> Several days later, an AT&T tech shows up with a disk and loads the proper compile and linker source and we recompile the compiler from the source. Because he would bill for less time if he just showed up with the compiler binaries and copied them over the rogue ones? "Let's take pristine source across the air-gap, and manipulate it on the infected computer where there is nonzero risk that the wrong compiler could somehow compile it." What if the kernel, C library or utilities in /bin and /usr/bin had been involved too?

76 points by otagekki 6y ago

Holy crap. That went way deeper than I expected it to. About halfway through, I was ready to copy the source to another computer and compile it there instead, and then suggest they nuke the original computer. Turns out I was right. I just didn't know how right I was, which could be important for whatever they plan to do to the person who hacked that computer.

hbrav 6y ago
Given the computer model involved, this is probably in the late 80s. If you've read The Cuckoo's Egg, it suggests that getting law enforcement to take computer intrusion seriously was very hard back then. I suppose the university could impose some kind of sanctions such as revoking the student's degree.

SideburnsOfDoom 6y ago

This is modelled on "Reflections on Trusting Trust" by Ken Thompson, 1984 https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

isaacimagine 6y ago
That's what I was thinking while reading this article, +1! In short, the premise of the talk is: if you add a self replicating change to any self-hosted compiler: - You can 'teach' the compiler to do things that are never explicitly stated in their source code, like, in this article, making the compiler modify programs that meet specific predicates. - You can not trust that your compiler hasn't been modified without bootstrapping it yourself. You can't trust a program without trusting its compiler; and since compilers are programs, well it's turtles all the way down. To build a trusted compiler, you have to bootstrap one yourself from some trusted manner, usually from hand-written assembly. In C, this would be akin to writing a small C compiler in assembly, using that to compile tcc, using that to compile a slightly larger C compiler, and that to compile clang or gcc. But in other more complex languages, such as Rust, bootstrapping is not as clear cut. There's the Reproducible Builds project, but afaict there exists no minimal small Rust compiler you can bootstrap the latest one from: You have to go back the the original OCaml one and compile up from there, or just trust that the publicly available build is secure (which it most likely is). I guess the lesson is that it's possible to inject hidden self-replicating behavior into any significantly complex system.
- steveklabnik 6y ago
  You can use mrustc to not have to go back to the original OCaml.
  - isaacimagine 6y ago
    Hmm, that's an interesting project! You'd still have to bootstrap a cpp compiler first, though
    - steveklabnik 6y ago
      Sure, you always have to bootstrap something.
- sandinmyjoints 6y ago
  I get the theory here, but never having written a compiler myself, I’m curious how easy/difficult it would be to come up with those predicates in practice?
  - isaacimagine 6y ago
    See 'Reflections on Rusting Trust' by Manish Goregaokar (In Pursuit of Laziness). https://manishearth.github.io/blog/2016/12/02/reflections-on-rusting-trust/ https://manishearth.github.io/blog/2016/12/02/reflections-on... He modifies rustc with a predicate that replaces the string 'hello world' with something else. The method used to replicate the change is basically a self-modifying quine.
- sandinmyjoints 6y ago
  https://github.com/certik/bcompiler/ https://github.com/certik/bcompiler/ is an effort to do this very bootstrapping.
- SAI_Peregrinus 6y ago
  An assembler can also do this same attack. You have to hand-assemble your basic assembler to machine code. In theory a sophisticated-enough hex editor could also be subverted, silently adding in code when it's used to edit executable-looking files. The system dynamic linker can be compromised to subvert any executable. Etc. If a machine is compromised, it can't be recovered without access to a known-good machine.
wantoncl 6y ago
Coincidentally, Computerphile just posted this today: https://www.youtube.com/watch?v=SJ7lOus1FzQ https://www.youtube.com/watch?v=SJ7lOus1FzQ

Zealotux 6y ago

>I was hired by a psychologist to fix a program that seemed to have "strange output" written by one of his ex-grad students. >printed some white supremacy message So I'm missing some details here maybe, but isn't that easier to ask the ex-student? He's probably responsible for all of this right? I don't see how anyone could get away with such thing.

lifeisstillgood 6y ago
That seems unlikely. From my reading there was a program (a simple research tool to ask people psychology questions), that was written by an "ex-Grad Student". Presumably it worked fine for some time, and then started exhibiting strange behaviour (after all if it was written to do this from the get go who would ever have used it?). If we are going to do detective guess-work (#) get the list of students who took the test. This level of effort to poison a research lab suggests to me the questions raised some uncomfortable issues with one of the respondents, who happened to have compiler abilities and the ability to read the root password on the post-it note on professors desk. (#) which frankly is how most detectives dive in - that one seems suspicious, look for evidence.
nicklecompte 6y ago
Considering this was malicious sabotage, I doubt the ex-student was even in contact with his former advisor, and regardless a conversation probably wouldn't be very helpful.

Larrikin 6y ago

This level of dedication to white supremacy is unsurprising to me after spending years on the internet, yet there are so many people who refuse to believe it's a real problem.

cgist 6y ago
It is easy to mistake 14-16 year olds with a lot of time on their hands for dedicated real supremacists. The best example was hordes of teenagers teaching a Microsoft chat bot to be a supremacist. Clearly they were having fun. It's their way of rebelling against the establishment by trolling. Since a teenager can no longer troll by assuming a leftist position, they must do so by pretending to be far right.
- Larrikin 6y ago
  I don't think teenagers being racist is a normal activity that is whimsical fun. There are actual consequences to people. Do you think the black 12 to 16 year old with an interest in technology continues that interest when the chat bot starts yelling racial slurs at him and telling him his life his worthless. It's very easy for people who have never experienced racism to say he should just brush it off. "It was just a joke bro" is also one of the favorite tools of neo nazis and other groups to gauge interest from other people while being able to claim they aren't actually racist. The story is also about a former grad student so he is well into adult hood but probably started off as one of those teenagers just "trolling".
  - bayindirh 6y ago
    > I don't think teenagers being racist is a normal activity that is whimsical fun. Bullies think bullying is fun. Spectators think the event is fun. The bullied one pays the price (don't ask me how I know). Some bullies never grow. They think it's fun, it's a way of expressing themselves. Yes, neither racism nor bullying is normal, but exhibitor of these behaviors normalize it.
  - skocznymroczny 6y ago
    I think most black teenagers have a much thicker skin than you think. Which wouldn't surprise me because most of the outrage about racism seems to be coming from preemptive actions from progressive whites rather than the ones who are supposedly affected by it.

isp 6y ago

Reminds me of the "Robin Hood and Friar Tuck" hack. Excerpts: > They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called 'Robin Hood' and 'Friar Tuck'. Robin Hood and Friar Tuck were designed to run as 'ghost jobs' (daemons, in Unix terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them. > They found the bandit ghost jobs running, and killed them... and were once again surprised. When Robin Hood was gunned, the following sequence of events took place: id1: Friar Tuck... I am under attack! Pray save me! id1: Off (aborted) id2: Fear not, friend Robin! I shall rout the Sheriff of Nottingham's men! id1: Thank you, my good fellow! > Each ghost-job would detect the fact that the other had been killed, and would start a new copy of the recently slain program within a few milliseconds. The only way to kill both ghosts was to kill them simultaneously (very difficult) or to deliberately crash the system. > Finally, the system programmers did the latter - only to find that the bandits appeared once again when the system rebooted! Quoted in the Jargon File: http://www.catb.org/jargon/html/meaning-of-hack.html http://www.catb.org/jargon/html/meaning-of-hack.html Original story: https://www.cs.utah.edu/~elb/folklore/xerox.txt https://www.cs.utah.edu/~elb/folklore/xerox.txt

dale_glass 6y ago
A couple things I wonder there: How does Robin (id1) manage to print the "I am under attack" message? Shouldn't a kill be immediate? Couldn't they just do something like sending a SIGSTOP to processes, or use killall in a tight loop? I get it's not Unix, but it seems reasonable similar concepts should exist. Also, this is more in line with Ken Thompson's "Reflections on Trusting Trust"
- hectorhector 6y ago
  You can put a signal handler in your program to catch the SIGSTOP and run some code before it gets killed.
  - 6y ago

runawaybottle 6y ago

Did this sort of read like a writing-prompt? By day 4 he could have unraveled a larger ploy involving lizard people. Putting the ‘Code’ into DaVinci Code.

midjji 6y ago

Every time I read these stories I wonder if they arent, just that, stories, then I realize it would be kinda fun to create one of these lowlevel compiler infecting things, and realize nah there are atleast a few out there written for shits and giggles. If nothing else, just because of the stories.

exikyut 6y ago
The best way to do a modern rendition would be to couple it with some sort of infrastructure vulnerability like a package-squatting attack or similar, so it has a chance to infiltrate for at least a little while before getting killed. The point would be to have it simply persist - and ideally for a little while too - but do nothing useful other than for example allocate an additional string on a local stack somewhere or something. Absolute best case scenario would be to announce its existence one year after successful infiltration. :D

deleted 6y ago

[deleted]

Justsignedup 6y ago

So tl;dr - the original creator created a self-replicating virus that would modify this one program, making it unmodifiable. - actually it was in the compiler. woop. can't fix it without external fixes - creates a backdoor into the system, could have fucked with the whole campus' data - he would have gotten away with it too, if not for the fact that he made this supremacist text So basically, dang.

kazinator 6y ago

> Several days later, an AT&T tech shows up with a disk and loads the proper compile and linker source and we recompile the compiler from the source. Because he would bill for less time if he just showed up with the compiler binaries and copied them over the rogue ones? "Let's take pristine source across the air-gap, and manipulate it on the infected computer where there is nonzero risk that the wrong compiler could somehow compile it." What if the kernel, C library or utilities in /bin and /usr/bin had been involved too?