5 ms·

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of predicti

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in an LLM only trained to predict protein structure sequences. No folding, nothing explicit in the corpus itself. Yet those truths manifest as a necessity of accurate predictions. Prediction is not bound by what the data explicitly shows. >A language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about. This is not true and is easy enough to test.

dvt 3y ago
> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.
- Davidzheng 3y ago
  But because the tokens are not word level they are capable of making up words that humans have never used before.
  - dvt 3y ago
    Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
    - _ea1k 3y ago
      You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".
    - jstanley 3y ago
      Does this satisfy you? Me: give me a random character sequence that is not a pronounceable word ChatGPT: Sure, here's a random character sequence that is not a pronounceable word: "Jkplqxzv"
- K0balt 3y ago
  It is possible to draw new conclusions from complex data. It’s not something that a human might not also see, just that it’s an undiscovered relationship. (Or at least not directly expressed except by inference in the training corpus)
- cle 3y ago
  Test it! > Can you make up a word? > Certainly! Here's a made-up word for you: > *Glimberfleck*: The sparkling reflection of sunlight that dances on water, making it seem like the surface is sprinkled with tiny, shimmering gems. > Feel free to use it or let me know if you'd like me to create another word! > When I’m looking out of the window of an airplane at the ocean, am I looking at a glimberfleck? > Yes, if you're looking out of the airplane window at the ocean and observing the sparkling reflections of sunlight on the water's surface, you could describe that beautiful phenomenon as a "glimberfleck." It's a whimsical term for those captivating, shimmering patterns you see.
- zarzavat 3y ago
  It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition. Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.
  - dvt 3y ago
    > It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.
    - jstanley 3y ago
      It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
      - deleted 3y ago
        [deleted]
    - deleted 3y ago
      [deleted]
  - visarga 3y ago
    Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies. Some characteristics of hackernewsitis include: - Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed. - Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them. - Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream. - Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals. - Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains. - Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence. So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics. (Claude) Me: Does this look like AI not understanding?
- squeaky-clean 3y ago
  Embeddings are usually common n-grams of bytes, not whole words unless it's a very common word like "it". It's also not all English. > Hey ChatGPT, can you give me a brief explanation of AI embeddings, but respond in Chinese? > 当然可以。 > AI 嵌入（embeddings）是一种将词汇、句子或其他类型的信息转化为高维度的向量的技术。这些向量捕捉到了原始信息的某些语义或关系特性。通过嵌入，我们可以更容易地在高维空间中比较和处理这些信息，如在机器学习模型中。 > 希望这有助于你了解！ As for saying it can only mimic words it knew in it's training data, this one gives zero results in google > Hey ChatGPT can you make up a word for when I'm hungry but there's nothing in my kitchen I want to eat, so I'm not that hungry. > Sure, how about "snackstipated"? It's a blend of "snack" and "constipated", suggesting a feeling of hunger but being stuck with unsatisfying options.
MildlySerious 3y ago
Tangential: https://www.thisworddoesnotexist.com/ https://www.thisworddoesnotexist.com/
mrcode007 3y ago
You mentioned the corpus prediction being the core of the LLM. Because of this, my prediction is we will see way more data withholding to prevent LLM learning just like we’ve seen with stack overflow, Reddit, X. I myself have started doing this. For example, I don’t publish code on GitHub anymore to prevent copilot training on my own code. Normally I like to get paid for work, instead of paying for GitHub and doing work for them:) I’d much rather take a page out of an artists playbook and require royalties. I have also built a coding test harness to test for AGI and let me tell you, that the LLMs are completely unable to reason on the most intermediate and advanced algorithmic problems you present to them the moment they are not in the corpus. This involves a nearly complete lack of the ability to generalize and infer new concepts needed to solve the problem even if the concept is explained in the problem description. Now the issue is that it is impossible to present the test because once it ends up in the corpus, then someone will put the solution in the corpus as well and claim the ability to reason. You can believe me or not; I won’t be making my test available either way, but it is not too hard to make one of your own to convince yourself. I wish it was possible to make the tests like that available but alas we will likely end up with a medical-style double blind kind of tests to get rid of AI FUD.
- visarga 3y ago
  You are not fundamentally wrong, but there are degrees. LLMs can do some symbolic manipulation and a few reasoning steps. Maybe they are just the result of pattern matching, but I have a hunch that most of the time humans do the same. We fake our understanding as much as we can, we use all shortcuts.
- vidarh 3y ago
  Withholding like that won't achieve anything other than your own obscurity.
  - LikelyABurner 3y ago
    Don’t kid yourself, posting your code on GitHub won’t lead to your being “discovered”, either. Online portfolios are the worst lie we tell young developers. It’s the software industry’s version of “exposure” gigs: they get a few more bits they can shove into their models, you get jack shit. Every, and I mean, every job I got I got because I either knew a guy or I knew a guy that knew a guy. You are far better off going to conferences and networking.
lsy 3y ago
The problem is that new knowledge comes not only from combination but discovery, and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured in human writing, and any structural properties thereof. To your point, prediction may not be bound by what the data explicitly shows, but it is necessarily bound by the data and its implicit and explicit structure. But to argue that the LLM can make any discovery a person can, or interact fluidly with the world as it exists, would require one to believe that every property of the world is either explicitly captured in human-written text or in the structure of that text, which I think would be pretty difficult to argue.
- Davidzheng 3y ago
  I think discovery for llm is possible both through its knowledge of the world from training data and also from some fields which are theoretical enough that interaction is not needed for discover--e.g pure mathematics but also many applied science
- osmarks 3y ago
  "Directly access"? Computers can use cameras and robotics hardware and such. It is not as if humans are special because of using eyes and muscles instead. Also, text does say a lot about the world. It includes descriptions of physics experiments and people's best knowledge of the laws, for instance.
- vidarh 3y ago
  People are hooking up LLM's with "access to the world" all over the place. Some of that will increasingly result in output that gets fed into training set. I'd be shocked if there aren't people running fairly large-scale research on letting LLMs using various patterns like ReAct to augment synthetic training data as we discuss this, because that'd be near the top of my list of things to do if I had the resources to turn around and train a large-scale model on those outputs, because testing ways of accelerating the training by letting the models build augmented training datasets themselves is such an obvious step.
masswerk 3y ago
However, any such information must be already contained in the semantic field that is provided to the LLM. Either explicitly, or as "nascent" relations in the data that may align with the structure of concepts in that data, since what we may regard as concepts in the LLM is either a matter of embedding or a matter of gradient descent (and related combinatory productions) – and thus again limited by the structure of the semantic field.

dvt 3y ago

> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.

Davidzheng 3y ago
But because the tokens are not word level they are capable of making up words that humans have never used before.
- dvt 3y ago
  Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
  - _ea1k 3y ago
    You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".
  - jstanley 3y ago
    Does this satisfy you? Me: give me a random character sequence that is not a pronounceable word ChatGPT: Sure, here's a random character sequence that is not a pronounceable word: "Jkplqxzv"
K0balt 3y ago
It is possible to draw new conclusions from complex data. It’s not something that a human might not also see, just that it’s an undiscovered relationship. (Or at least not directly expressed except by inference in the training corpus)
cle 3y ago
Test it! > Can you make up a word? > Certainly! Here's a made-up word for you: > *Glimberfleck*: The sparkling reflection of sunlight that dances on water, making it seem like the surface is sprinkled with tiny, shimmering gems. > Feel free to use it or let me know if you'd like me to create another word! > When I’m looking out of the window of an airplane at the ocean, am I looking at a glimberfleck? > Yes, if you're looking out of the airplane window at the ocean and observing the sparkling reflections of sunlight on the water's surface, you could describe that beautiful phenomenon as a "glimberfleck." It's a whimsical term for those captivating, shimmering patterns you see.
zarzavat 3y ago
It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition. Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.
- dvt 3y ago
  > It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.
  - jstanley 3y ago
    It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
    - deleted 3y ago
      [deleted]
  - deleted 3y ago
    [deleted]
- visarga 3y ago
  Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies. Some characteristics of hackernewsitis include: - Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed. - Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them. - Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream. - Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals. - Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains. - Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence. So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics. (Claude) Me: Does this look like AI not understanding?
squeaky-clean 3y ago
Embeddings are usually common n-grams of bytes, not whole words unless it's a very common word like "it". It's also not all English. > Hey ChatGPT, can you give me a brief explanation of AI embeddings, but respond in Chinese? > 当然可以。 > AI 嵌入（embeddings）是一种将词汇、句子或其他类型的信息转化为高维度的向量的技术。这些向量捕捉到了原始信息的某些语义或关系特性。通过嵌入，我们可以更容易地在高维空间中比较和处理这些信息，如在机器学习模型中。 > 希望这有助于你了解！ As for saying it can only mimic words it knew in it's training data, this one gives zero results in google > Hey ChatGPT can you make up a word for when I'm hungry but there's nothing in my kitchen I want to eat, so I'm not that hungry. > Sure, how about "snackstipated"? It's a blend of "snack" and "constipated", suggesting a feeling of hunger but being stuck with unsatisfying options.

MildlySerious 3y ago

Tangential: https://www.thisworddoesnotexist.com/ https://www.thisworddoesnotexist.com/

mrcode007 3y ago

You mentioned the corpus prediction being the core of the LLM. Because of this, my prediction is we will see way more data withholding to prevent LLM learning just like we’ve seen with stack overflow, Reddit, X. I myself have started doing this. For example, I don’t publish code on GitHub anymore to prevent copilot training on my own code. Normally I like to get paid for work, instead of paying for GitHub and doing work for them:) I’d much rather take a page out of an artists playbook and require royalties. I have also built a coding test harness to test for AGI and let me tell you, that the LLMs are completely unable to reason on the most intermediate and advanced algorithmic problems you present to them the moment they are not in the corpus. This involves a nearly complete lack of the ability to generalize and infer new concepts needed to solve the problem even if the concept is explained in the problem description. Now the issue is that it is impossible to present the test because once it ends up in the corpus, then someone will put the solution in the corpus as well and claim the ability to reason. You can believe me or not; I won’t be making my test available either way, but it is not too hard to make one of your own to convince yourself. I wish it was possible to make the tests like that available but alas we will likely end up with a medical-style double blind kind of tests to get rid of AI FUD.

visarga 3y ago
You are not fundamentally wrong, but there are degrees. LLMs can do some symbolic manipulation and a few reasoning steps. Maybe they are just the result of pattern matching, but I have a hunch that most of the time humans do the same. We fake our understanding as much as we can, we use all shortcuts.
vidarh 3y ago
Withholding like that won't achieve anything other than your own obscurity.
- LikelyABurner 3y ago
  Don’t kid yourself, posting your code on GitHub won’t lead to your being “discovered”, either. Online portfolios are the worst lie we tell young developers. It’s the software industry’s version of “exposure” gigs: they get a few more bits they can shove into their models, you get jack shit. Every, and I mean, every job I got I got because I either knew a guy or I knew a guy that knew a guy. You are far better off going to conferences and networking.

lsy 3y ago

The problem is that new knowledge comes not only from combination but discovery, and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured in human writing, and any structural properties thereof. To your point, prediction may not be bound by what the data explicitly shows, but it is necessarily bound by the data and its implicit and explicit structure. But to argue that the LLM can make any discovery a person can, or interact fluidly with the world as it exists, would require one to believe that every property of the world is either explicitly captured in human-written text or in the structure of that text, which I think would be pretty difficult to argue.

Davidzheng 3y ago
I think discovery for llm is possible both through its knowledge of the world from training data and also from some fields which are theoretical enough that interaction is not needed for discover--e.g pure mathematics but also many applied science
osmarks 3y ago
"Directly access"? Computers can use cameras and robotics hardware and such. It is not as if humans are special because of using eyes and muscles instead. Also, text does say a lot about the world. It includes descriptions of physics experiments and people's best knowledge of the laws, for instance.
vidarh 3y ago
People are hooking up LLM's with "access to the world" all over the place. Some of that will increasingly result in output that gets fed into training set. I'd be shocked if there aren't people running fairly large-scale research on letting LLMs using various patterns like ReAct to augment synthetic training data as we discuss this, because that'd be near the top of my list of things to do if I had the resources to turn around and train a large-scale model on those outputs, because testing ways of accelerating the training by letting the models build augmented training datasets themselves is such an obvious step.

masswerk 3y ago

However, any such information must be already contained in the semantic field that is provided to the LLM. Either explicitly, or as "nascent" relations in the data that may align with the structure of concepts in that data, since what we may regard as concepts in the LLM is either a matter of embedding or a matter of gradient descent (and related combinatory productions) – and thus again limited by the structure of the semantic field.