4 ms·
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
by kvark 4y ago
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
- mormegil 4y agoWell, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust https://fermatslibrary.com/s/reflections-on-trusting-trust
- MonkeyMalarky 4y agoThat was my first impression as well. If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. Anyways, don't count your donkeys until they've flown by midnight.
- plugin-baby 4y agoI could care less.
- MonkeyMalarky 4y agoGood for you!
- plugin-baby 4y agoMy comment was poorly articulated. GP said: > If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. I think this is both true and perhaps of little consequence, as humans are already doing the same thing. One example is the phrase: > I could care less.
- im3w1l 4y agoRoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.
- IncRnd 4y agoRequirements for this scenario to work: The checking program and the program to be checked must both be compromised in tandem. This is covered in RoTT. This has already happened in various areas. For one area, look at gambling (two decades ago). Ron Harris worked at the Nevada Gaming Control board as a tester of new systems and configued the field verification of those systems. Eventually he turned that into video gaming devices that would hit a jackpot after a certain button sequence was pressed.
- srcreigh 4y agoI'm sure enough people make compilers from scratch to avoid this issue. Yeah, most people bootstrap using a pre-existing compiler, but I know at least one person who compiled their initial compiler to ASM by hand before using it.
- srcreigh 4y agoNobody asked, but that person is Donald Knuth
- yazzku 4y agoPDF: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- pabs3 4y agoHe would probably agree with Debian; require libre training data, libre training code, libre labelling, libre models etc. https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy https://deepdive.opensource.org/podcast/why-debian-wont-distribute-ai-models-any-time-soon/ https://deepdive.opensource.org/podcast/why-debian-wont-dist... He would also have something to say about not using AI for critical decisions and allowing folks to appeal AI decisions etc.
- austinjp 4y agoThanks for this. Love the "ToxicCandy" terminology.
- kmeisthax 4y agoMachine Learning is proprietary software's final form: there is no source code.
- romwell 4y agoAnd, as this paper shows, it is impossible to reverse-engineer it. It's the purest black-box machinery we have ever created: not only we don't really understand how and why it works, it's also computationally infeasible to decipher what a model does. Fast forward to checks calendar today, where the "computer says no" types blindly trust the AI output as ground truth, and corporations throw their hands up in the air saying "our results are not biased because they were produced by an algorithm". Yay.