4 ms·

A neural net is not a database where the original source is sitting somewhere in an obvious place with a reference. A neural net is a black box of functions th

A neural net is not a database where the original source is sitting somewhere in an obvious place with a reference. A neural net is a black box of functions that have been automatically fit to the training data. There is no way to know what sources have been memorized vs which have made their mark by affecting other types of functions in the neural net.

aantix 3y ago
It's possible. Perplexity.ai is trying to solve this problem. E.g. "Japan's App Store antitrust case" https://www.perplexity.ai/search/Japans-App-Store-GJNTsIOVSyS58Ep5F83Wig?s=c https://www.perplexity.ai/search/Japans-App-Store-GJNTsIOVSy...
- Philpax 3y ago
  That’s not the same thing. Perplexity is using an already-trained LLM to read those sources and synthesise a new result from them. This allows them to cite the sources used for generation. LLM training sees these documents without context; it doesn’t know where they came from, and any such attribution would become part of the thing it’s trying to mimic. It’s still largely an unsolved problem.
- simonw 3y ago
  That's a different approach: they've implemented RAG, Retrieval Augmented Generation, where the tool runs additional searches as part of answering a question. ChatGPT Browse and Bing and Google Bard implement the same pattern. RAG does allow for some citation, but it doesn't help with the larger problem of not being able to cite for answers provided by the unassisted language model.
layer8 3y ago
Presumably, if a passage of any significant length is cited verbatim (or almost verbatim), there would have been a way to track that source through the weights. The issue of replicating a style is probably more difficult.
- solveit 3y ago
  > Presumably, if a passage of any significant length is cited verbatim (or almost verbatim), there would have been a way to track that source through the weights. Figure this out and you get to choose which AI lab you want to make seven figures at. It's a really difficult problem.
  - layer8 3y ago
    It’s likely first and foremost a resource problem. “How much different would the output be if that text hadn’t been part of the training data” can _in principle_ be answered by instead of training one model, training N models where N is the number of texts in the training data, omitting text i from the training data of model i, and then when using the model(s), run all N models in parallel and apply some distance metric on their outputs. In case of a verbatim quote, at least one of the models will stand out in that comparison, allowing to infer the source. The difficulty would be in finding a way to do something along those lines efficiently enough to be practical.
    - spunker540 3y ago
      each llm costs ($10-100) millions to train x billions of trainings data ~= $100 quadrillion dollars, so that is unofortunately out of reach of most countries.
  - photonthug 3y ago
    > Figure this out and you get to choose which AI lab you want to make seven figures at. It's a really difficult problem. It doesn't have to be perfect to be helpful, and even something that is very imperfect would at least send the signal that model-owners give a shit about attribution in general. Given a specific output, it might be hard to say which sections of the very large weighted network were tickled during the output, and what inputs were used to build that section of the network. But this level of "citation resolution" is not always what people are necessarily interested in. If an LLM is giving medical advice, I might want to at least know whether it's reading medical journals or facebook posts. If it's political advice/summary/synthesis, it might be relevant to know how much it's been reading Marx vs Lenin or whatever. Pin-pointing original paragraphs as sources would be great, but for most models it's not like there's anything that's very clear about the input datasets. EDIT: Building on this a bit, a lot of people are really worried about AI "poisoning the well" such that they are retraining on content generated by other AIs so that algorithmic feeds can trash the next-gen internet even worse than the current one. This shows that attribution-sourcing even at the basic level of "only human generated content is used in this model" can be useful and confidence-inspiring.
dlandis 3y ago
> There is no way to know what sources have been memorized vs which have made their mark by affecting other types of functions in the neural net. But if it's possible for the neural net to memorize passages of text then surely it could also memorize where it got those passages of text from. Perhaps not with today's exact models and technology, but if it was a requirement then someone would figure out a way to do it.
- wrs 3y ago
  Except it doesn’t memorize text. It generates text that is statistically likely. Generating a citation that is statistically likely wouldn’t really help the problem.
  - __loam 3y ago
    So it's just bullshit then.
    - fennecbutt 3y ago
      It's literally how our meat bag brains work pretty much. Anything like word association games are basically the same exercise, but with humans and hell, I bet I could play a word association game with an LLM, too.
- Tao3300 3y ago
  Neural nets don't memorize passages of text. They train on vectorized tokens. You get a model of how language statistically works, not understanding and memory.
  - FredPret 3y ago
    You can encode understanding in a vector. To use Andrew Ng's example, you have build a multi-dimensional arrow representing "king". You compare it to the arrow for "queen" and you see that it's almost identical, except it points in the opposite direction in the gender dimension. Compare it to "man" and you see that "king" and "man" have some things in common, but "man" is a broader term. That's getting really close to understanding as far as I'm concerned; especially if you have a large number of such arrows. It's statistical in a literal sense, but it's more like the computer used statistics to work out the meaning of each word by a process of elimination and now actually understands it.

aantix 3y ago

It's possible. Perplexity.ai is trying to solve this problem. E.g. "Japan's App Store antitrust case" https://www.perplexity.ai/search/Japans-App-Store-GJNTsIOVSyS58Ep5F83Wig?s=c https://www.perplexity.ai/search/Japans-App-Store-GJNTsIOVSy...

Philpax 3y ago
That’s not the same thing. Perplexity is using an already-trained LLM to read those sources and synthesise a new result from them. This allows them to cite the sources used for generation. LLM training sees these documents without context; it doesn’t know where they came from, and any such attribution would become part of the thing it’s trying to mimic. It’s still largely an unsolved problem.
simonw 3y ago
That's a different approach: they've implemented RAG, Retrieval Augmented Generation, where the tool runs additional searches as part of answering a question. ChatGPT Browse and Bing and Google Bard implement the same pattern. RAG does allow for some citation, but it doesn't help with the larger problem of not being able to cite for answers provided by the unassisted language model.

layer8 3y ago

Presumably, if a passage of any significant length is cited verbatim (or almost verbatim), there would have been a way to track that source through the weights. The issue of replicating a style is probably more difficult.

solveit 3y ago
> Presumably, if a passage of any significant length is cited verbatim (or almost verbatim), there would have been a way to track that source through the weights. Figure this out and you get to choose which AI lab you want to make seven figures at. It's a really difficult problem.
- layer8 3y ago
  It’s likely first and foremost a resource problem. “How much different would the output be if that text hadn’t been part of the training data” can _in principle_ be answered by instead of training one model, training N models where N is the number of texts in the training data, omitting text i from the training data of model i, and then when using the model(s), run all N models in parallel and apply some distance metric on their outputs. In case of a verbatim quote, at least one of the models will stand out in that comparison, allowing to infer the source. The difficulty would be in finding a way to do something along those lines efficiently enough to be practical.
  - spunker540 3y ago
    each llm costs ($10-100) millions to train x billions of trainings data ~= $100 quadrillion dollars, so that is unofortunately out of reach of most countries.
- photonthug 3y ago
  > Figure this out and you get to choose which AI lab you want to make seven figures at. It's a really difficult problem. It doesn't have to be perfect to be helpful, and even something that is very imperfect would at least send the signal that model-owners give a shit about attribution in general. Given a specific output, it might be hard to say which sections of the very large weighted network were tickled during the output, and what inputs were used to build that section of the network. But this level of "citation resolution" is not always what people are necessarily interested in. If an LLM is giving medical advice, I might want to at least know whether it's reading medical journals or facebook posts. If it's political advice/summary/synthesis, it might be relevant to know how much it's been reading Marx vs Lenin or whatever. Pin-pointing original paragraphs as sources would be great, but for most models it's not like there's anything that's very clear about the input datasets. EDIT: Building on this a bit, a lot of people are really worried about AI "poisoning the well" such that they are retraining on content generated by other AIs so that algorithmic feeds can trash the next-gen internet even worse than the current one. This shows that attribution-sourcing even at the basic level of "only human generated content is used in this model" can be useful and confidence-inspiring.

dlandis 3y ago

> There is no way to know what sources have been memorized vs which have made their mark by affecting other types of functions in the neural net. But if it's possible for the neural net to memorize passages of text then surely it could also memorize where it got those passages of text from. Perhaps not with today's exact models and technology, but if it was a requirement then someone would figure out a way to do it.

wrs 3y ago
Except it doesn’t memorize text. It generates text that is statistically likely. Generating a citation that is statistically likely wouldn’t really help the problem.
- __loam 3y ago
  So it's just bullshit then.
  - fennecbutt 3y ago
    It's literally how our meat bag brains work pretty much. Anything like word association games are basically the same exercise, but with humans and hell, I bet I could play a word association game with an LLM, too.
Tao3300 3y ago
Neural nets don't memorize passages of text. They train on vectorized tokens. You get a model of how language statistically works, not understanding and memory.
- FredPret 3y ago
  You can encode understanding in a vector. To use Andrew Ng's example, you have build a multi-dimensional arrow representing "king". You compare it to the arrow for "queen" and you see that it's almost identical, except it points in the opposite direction in the gender dimension. Compare it to "man" and you see that "king" and "man" have some things in common, but "man" is a broader term. That's getting really close to understanding as far as I'm concerned; especially if you have a large number of such arrows. It's statistical in a literal sense, but it's more like the computer used statistics to work out the meaning of each word by a process of elimination and now actually understands it.