5 ms·

Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust https://fermatslibrary.com/s/refl

Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust https://fermatslibrary.com/s/reflections-on-trusting-trust

MonkeyMalarky 4y ago
That was my first impression as well. If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. Anyways, don't count your donkeys until they've flown by midnight.
- plugin-baby 4y ago
  I could care less.
  - MonkeyMalarky 4y ago
    Good for you!
    - plugin-baby 4y ago
      My comment was poorly articulated. GP said: > If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. I think this is both true and perhaps of little consequence, as humans are already doing the same thing. One example is the phrase: > I could care less.
im3w1l 4y ago
RoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.
- IncRnd 4y ago
  Requirements for this scenario to work: The checking program and the program to be checked must both be compromised in tandem. This is covered in RoTT. This has already happened in various areas. For one area, look at gambling (two decades ago). Ron Harris worked at the Nevada Gaming Control board as a tester of new systems and configued the field verification of those systems. Eventually he turned that into video gaming devices that would hit a jackpot after a certain button sequence was pressed.
- srcreigh 4y ago
  I'm sure enough people make compilers from scratch to avoid this issue. Yeah, most people bootstrap using a pre-existing compiler, but I know at least one person who compiled their initial compiler to ASM by hand before using it.
  - srcreigh 4y ago
    Nobody asked, but that person is Donald Knuth
yazzku 4y ago
PDF: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust https://fermatslibrary.com/s/reflections-on-trusting-trust

MonkeyMalarky 4y ago

That was my first impression as well. If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. Anyways, don't count your donkeys until they've flown by midnight.

plugin-baby 4y ago
I could care less.
- MonkeyMalarky 4y ago
  Good for you!
  - plugin-baby 4y ago
    My comment was poorly articulated. GP said: > If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. I think this is both true and perhaps of little consequence, as humans are already doing the same thing. One example is the phrase: > I could care less.

im3w1l 4y ago

RoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.

IncRnd 4y ago
Requirements for this scenario to work: The checking program and the program to be checked must both be compromised in tandem. This is covered in RoTT. This has already happened in various areas. For one area, look at gambling (two decades ago). Ron Harris worked at the Nevada Gaming Control board as a tester of new systems and configued the field verification of those systems. Eventually he turned that into video gaming devices that would hit a jackpot after a certain button sequence was pressed.
srcreigh 4y ago
I'm sure enough people make compilers from scratch to avoid this issue. Yeah, most people bootstrap using a pre-existing compiler, but I know at least one person who compiled their initial compiler to ASM by hand before using it.
- srcreigh 4y ago
  Nobody asked, but that person is Donald Knuth

yazzku 4y ago

PDF: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...