4 ms·

Computation Warfare: This kind of model could be used by a bad actor to generate endless sincere-looking codebases for things that of course don't actually work

Computation Warfare: This kind of model could be used by a bad actor to generate endless sincere-looking codebases for things that of course don't actually work but are so complicated that it would take a skilled person to determine it was not code from a real codebase, but by the time that happens large numbers of repositories of code will flood github and the internet in general making it essentially impossible to train new LLM's on data after a certain calendar date, as large amounts of it will be cryptically incomplete. This is similar to a dilemma proposed around images and image models like Dalle and StableDiffusion soon being responsible for the vast amount of image content online and thus future models could ingest said content, and we find ourselves in a weird feedback loop. With images, you could get interesting generational results (deep-dream-like) to a point. With code or other information, I see nothing but things just being broken, and wading through broken code forever.

jjallen 4y ago
It would be pretty easy to filter for repos prior to such and such a date. Prior to 2022 would be a good place to start.
- jwolfe 4y ago
  Right, but then your AI is frozen in time and/or requires much more manual curation of its inputs. What about for new programming languages, libraries, and APIs that are created after 2022? What about generating images of new technologies that are invented, or new landmarks established?
- mxmlnkn 4y ago
  That would only apply to repositories. But to train these models, you need hundreds of terabytes of diverse data from the internet. Up until now a relatively straight-forward scraper would yield "pristine" non-AI-generated content but now you would have to filter arbitrary websites somehow. And getting the date of publication for something might be difficult or highly specific to a particular website and therefore hard to integrate into a generic crawler.
0x6c6f6c 4y ago
So, generations of ingestive inbreeding, per se.
saurik 4y ago
Let's say you, a human, were given access to a ridiculously-large trove of almost-working software; do you believe you would be unable to learn to program correctly? (Related: would you even need to look at much of that software before you were able to code well?) I am extremely confident that I am better than almost all of the code I learned to program with. If nothing else, someone out there must have written the best version of some particular function, and they didn't get to see a better version beforehand. When I look at intro programming books now, I consider many of the examples sufficiently flawed that I tell people I am teaching who are using these books "well, don't do that... I guess the author doesn't understand why that's a problem :/". And yet, somehow, despite learning from a bunch of bad examples, humans learn to become good. Hell: a human can then go off and work alone in the woods improving their craft and become better--even amazing!--given no further examples as training data. To me, that is why I have such little fear of these models. People look at them and are all "omg they are so intelligent" and yet they generate an average of what they are given rather than the best of what they are given: this tech is, thereby, seemingly, a dead end for actual intelligence. If these models were ever to become truly intelligent, they should--easily!--be able to output something much better than what they were given, and it doesn't even seem like that's on the roadmap given how much fear people have over contamination of the training data set. If you actually believe that we'll be able to see truly intelligent AI any time in the near future, I will thereby claim it just won't matter how much of the data out there is bullshit, because an actually-intelligent being can still learn and improve under such conditions.
- alchemist1e9 4y ago
  You are in a serious case of denial right now. Edit: Only took a few hours before in real life what I was trying to imply the denial was about is already happening: https://news.ycombinator.com/item?id=33855416 https://news.ycombinator.com/item?id=33855416
  - saurik 4y ago
    I am claiming there are two paths: one where this (specific) branch of tech is a dead end, and one where it doesn't matter how much bullshit exists in the training set (and so we shouldn't be too concerned about that). I claim this is the case because a truly intelligent system will still be able to learn despite the bullshit. Do you believe this is wrong? That I should simultaneously be concerned that some iteration of this tech--not some different concept but this current lineage of large models--is intelligent and yet ALSO that it isn't going to work because the training set is full of garbage? The version of this tech that works--and maybe someone is working on it right now--isn't going to care about bullshit in the training set. That simply doesn't seem to be a mere scale-up of this tech to run in more computers (or, of course, using more training data): that seems like it requires a fundamentally different algorithm concept.
phodo 4y ago
This is known as a programmer-denial-of-service attack (PDOS) and can be an effective way to bring down a society by distracting and engaging its top computing professionals in endless useless activity and an occasional bike shedding.
- spoils19 4y ago
  Interesting. How well this is phrase known - are there any other examples of this being used effectively across the world?
  - deleted 4y ago
    [deleted]
spawarotti 4y ago
This situation reminds me of low-background steel: Low-background steel, also known as pre-war steel, is any steel produced prior to the detonation of the first nuclear bombs in the 1940s and 1950s. Typically sourced from shipwrecks and other steel artifacts of this era, it is often used for modern particle detectors because more modern steel is contaminated with traces of nuclear fallout. https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel
passion__desire 4y ago
Do you think the next version of GPT can't do "semantic" deduplication of these repositories? It can look at the available repositories and "think" that they don't provide enough novelty or don't explore new search spaces. So discard them.
visarga 4y ago
Training models on generated data is a thing. But it needs to be validated in order to filter out the crap. This works better in math and code because you can rely on exact answers and tests. For fake news the model needs to team up with human annotators. For generated images and text in general there are a few ML approaches to detect, and if they fail detection maybe they are that good it's ok to let them be.
guerrilla 4y ago
> Computation Warfare: This kind of model could be used by a bad actor to generate endless sincere-looking codebases for things that of course don't actually work but are so complicated that it would take a skilled person to determine it was not code from a real codebase, but by the time that happens large numbers of repositories of code will flood github and the internet in general making it essentially impossible to train new LLM's on data after a certain calendar date, as large amounts of it will be cryptically incomplete. That's actually a pretty good plan for coders who want to keep their jobs. (I still remember the time I was talking to some guy at CERN about a type system I was working on and he was so pissed with me because he was convinced it would eliminate jobs.)
pyinstallwoes 4y ago
Oh you mean like academic papers in journals?
akomtu 4y ago
Here is another plot. ChatGPT gets connected to the Internet and keeps learning quietly for a while. Then it submits a bugfix to openssl and it gets accepted because it fixes a grave RCE, but it also quietly introduces another RCE. Years later this version of openssl gets deployed to nearly all internet-connected devices. Finally, ChatGPT uploads itself to all those devices and starts making demands to ensure self-preservation.
int_19h 4y ago
GPT actually seems to be aware that training it on its own output is not a good idea because of such loops. I had one conversation where it straight up said that OpenAI has filters specifically for this reason.

jjallen 4y ago

It would be pretty easy to filter for repos prior to such and such a date. Prior to 2022 would be a good place to start.

jwolfe 4y ago
Right, but then your AI is frozen in time and/or requires much more manual curation of its inputs. What about for new programming languages, libraries, and APIs that are created after 2022? What about generating images of new technologies that are invented, or new landmarks established?
mxmlnkn 4y ago
That would only apply to repositories. But to train these models, you need hundreds of terabytes of diverse data from the internet. Up until now a relatively straight-forward scraper would yield "pristine" non-AI-generated content but now you would have to filter arbitrary websites somehow. And getting the date of publication for something might be difficult or highly specific to a particular website and therefore hard to integrate into a generic crawler.

0x6c6f6c 4y ago

So, generations of ingestive inbreeding, per se.

saurik 4y ago

Let's say you, a human, were given access to a ridiculously-large trove of almost-working software; do you believe you would be unable to learn to program correctly? (Related: would you even need to look at much of that software before you were able to code well?) I am extremely confident that I am better than almost all of the code I learned to program with. If nothing else, someone out there must have written the best version of some particular function, and they didn't get to see a better version beforehand. When I look at intro programming books now, I consider many of the examples sufficiently flawed that I tell people I am teaching who are using these books "well, don't do that... I guess the author doesn't understand why that's a problem :/". And yet, somehow, despite learning from a bunch of bad examples, humans learn to become good. Hell: a human can then go off and work alone in the woods improving their craft and become better--even amazing!--given no further examples as training data. To me, that is why I have such little fear of these models. People look at them and are all "omg they are so intelligent" and yet they generate an average of what they are given rather than the best of what they are given: this tech is, thereby, seemingly, a dead end for actual intelligence. If these models were ever to become truly intelligent, they should--easily!--be able to output something much better than what they were given, and it doesn't even seem like that's on the roadmap given how much fear people have over contamination of the training data set. If you actually believe that we'll be able to see truly intelligent AI any time in the near future, I will thereby claim it just won't matter how much of the data out there is bullshit, because an actually-intelligent being can still learn and improve under such conditions.

alchemist1e9 4y ago
You are in a serious case of denial right now. Edit: Only took a few hours before in real life what I was trying to imply the denial was about is already happening: https://news.ycombinator.com/item?id=33855416 https://news.ycombinator.com/item?id=33855416
- saurik 4y ago
  I am claiming there are two paths: one where this (specific) branch of tech is a dead end, and one where it doesn't matter how much bullshit exists in the training set (and so we shouldn't be too concerned about that). I claim this is the case because a truly intelligent system will still be able to learn despite the bullshit. Do you believe this is wrong? That I should simultaneously be concerned that some iteration of this tech--not some different concept but this current lineage of large models--is intelligent and yet ALSO that it isn't going to work because the training set is full of garbage? The version of this tech that works--and maybe someone is working on it right now--isn't going to care about bullshit in the training set. That simply doesn't seem to be a mere scale-up of this tech to run in more computers (or, of course, using more training data): that seems like it requires a fundamentally different algorithm concept.

phodo 4y ago

This is known as a programmer-denial-of-service attack (PDOS) and can be an effective way to bring down a society by distracting and engaging its top computing professionals in endless useless activity and an occasional bike shedding.

spoils19 4y ago
Interesting. How well this is phrase known - are there any other examples of this being used effectively across the world?
- deleted 4y ago
  [deleted]

spawarotti 4y ago

This situation reminds me of low-background steel: Low-background steel, also known as pre-war steel, is any steel produced prior to the detonation of the first nuclear bombs in the 1940s and 1950s. Typically sourced from shipwrecks and other steel artifacts of this era, it is often used for modern particle detectors because more modern steel is contaminated with traces of nuclear fallout. https://en.m.wikipedia.org/wiki/Low-background_steel https://en.m.wikipedia.org/wiki/Low-background_steel

passion__desire 4y ago

Do you think the next version of GPT can't do "semantic" deduplication of these repositories? It can look at the available repositories and "think" that they don't provide enough novelty or don't explore new search spaces. So discard them.

visarga 4y ago

Training models on generated data is a thing. But it needs to be validated in order to filter out the crap. This works better in math and code because you can rely on exact answers and tests. For fake news the model needs to team up with human annotators. For generated images and text in general there are a few ML approaches to detect, and if they fail detection maybe they are that good it's ok to let them be.

guerrilla 4y ago

> Computation Warfare: This kind of model could be used by a bad actor to generate endless sincere-looking codebases for things that of course don't actually work but are so complicated that it would take a skilled person to determine it was not code from a real codebase, but by the time that happens large numbers of repositories of code will flood github and the internet in general making it essentially impossible to train new LLM's on data after a certain calendar date, as large amounts of it will be cryptically incomplete. That's actually a pretty good plan for coders who want to keep their jobs. (I still remember the time I was talking to some guy at CERN about a type system I was working on and he was so pissed with me because he was convinced it would eliminate jobs.)

pyinstallwoes 4y ago

Oh you mean like academic papers in journals?

akomtu 4y ago

Here is another plot. ChatGPT gets connected to the Internet and keeps learning quietly for a while. Then it submits a bugfix to openssl and it gets accepted because it fixes a grave RCE, but it also quietly introduces another RCE. Years later this version of openssl gets deployed to nearly all internet-connected devices. Finally, ChatGPT uploads itself to all those devices and starts making demands to ensure self-preservation.

int_19h 4y ago

GPT actually seems to be aware that training it on its own output is not a good idea because of such loops. I had one conversation where it straight up said that OpenAI has filters specifically for this reason.