5 ms·

> My current (tentative) resolution of the surprise is that language encoded way more information about reality than we thought it did. I think you are close t

> My current (tentative) resolution of the surprise is that language encoded way more information about reality than we thought it did. I think you are close to the mark, but you have been subtly mislead: language is not the data we are working with. We are working with text. Once you fix that particular failure of word choice, everything else becomes much more clear: text contains much more information than language. We aren't dealing with just any text, either: that would be noise. We're training LLMs on written text. Natural language is infamous for one specific feature: ambiguity. There are many possible ways to write something, but we can only write one. We must choose: in doing so, we record the choice itself, and all of the entropy that informed it. That entropy is the secret sauce: the extra data that LLMs are sometimes able to model. We don't see it, because we read language, not text. The big surprise is that LLMs aren't able to write language: they can only write text. They don't get tripped up reading ambiguity, but they can't avoid writing it, either. Who chooses what an LLM writes? Is it a mystery character who lives in a black box, or a continuation of the entropy that was encoded into the text that LLM was trained on?

fwlr 3y ago
The idea that text contains more information than language is fascinating, wow. Thank you!
blowski 3y ago
I don't really understand your distinction between language and text, but it sounds intrguing. Would you be able to give more detail? I searched but couldn't find anything that seemed to explain it.
- j16sdiz 3y ago
  Spoken language have pitch, stress, mood, etc.. Written text contain some, but not all. Text, on the other hand, can present in list or table, with varies formatting, indentation. You can't reproduce them in spoken text.
- bruce511 3y ago
  Not the poster, but for me it comes down to a mix of clarity and permanence. I teach both verbally (interactive question/answer) and I've also written text books. Verbally by language is "loose". I'll say class when I mean object, unicode when I mean utf-8 and so on. Sentences are not all well formed, and sometimes change mid-thought. It's very "real time" Writing is a lot more deliberate. I have to be sure of each fact I state. I often re-test things I'm only 95% sure about. I edit, restructure, remove, add, until I'm happy. Of course all communication falls on a spectrum. Think phone call at one end, text book on the other. When I do a verbal lecture I'm usually careful with my speech, and when I post on hacker-news less rigorous. Language covers all of it. Text skews to the more deliberate side. Cunningly the language models are trained using (mostly) text, not speech. That will have an impact on them.
- espe 3y ago
  from a linguistic standpoint a text is a whole lot more than language: it is an externalisation of thought that is fixed onto a medium using writing utensils and most of all, cultural norms in the form of a wild variety of different genres and forms of text, ranging from something like a stream of consciousness to something like a speech act. furthermore, text can be conceptually written or spoken and with the internet we got an explosion of text that is conceptually spoken. those are the things OP might be refering to in regards to the "entropy" that encodes much more than just the tokens themselves.
- gmt2027 3y ago
  Text is an instance of language. Think of it as the difference between the python language and a large collection of python programs. The language describes syntactic and semantic rules, the collection is a sampling of possible programs that encodes a significant amount of information about the world. You could learn a lot about the laws of nature, the internet, even human society and laws by examining all the python programs ever written. An extreme version of the same idea is the difference between understanding DNA vs the genome of every individual organism that has lived on earth. The species record encodes a ton of information about the laws of nature, the composition and history of our planet. You could deduce physical laws and constants from looking at this information, wars and natural disasters, economic performance, historical natural boundaries, the industrial revolution and a lot more.
  - Dwolb 3y ago
    This is a fascinating point. Let me see if I can play this back. If a student studies DNA sequencing, they’ll learn about the compounds that make up DNA, how traits get encoded, etc. Therefore the student might expect an AI trained on people’s DNA to be able to tell you about whether certain traits are more prevalent in one geography or the other. However, since DNA responds to changes in environment, the AI would start to see time, population, and geography-based patterns emerge. The AI for example could infer that a given person in the US who’s settled in NYC had ancestors from a given region of the world who left due to an environmental disaster just by looking at a given DNA sequence. To the student this result would look like magic. But in the end, it’s a result of individual’s DNA having much more information encoded in it than just human traits.
  - espe 3y ago
    text and language intersect. in some ways, text is a superset of language, mostly due to social, or what is also called pragmatic, factors that complement semantics. also, the semantics/syntax interface is everything else than clear cut, at least in natural human languages.
    - catgoose 3y ago
      I think about this as shape and color. No one ever saw a shape that wasn’t colored and likewise there are no colored things that do not have a shape. Also, displaying text without a font is not possible. Text is the surface of the ocean where waves emerge, and while they have their own properties and may seem to naively have agency, they are an expression of the underlying ocean.
zainhoda 3y ago
There’s an exercise that some people do when learning programming, which is to write down the steps to make a sandwich. Then the teacher follows the exact instructions to make a sandwich and most people don’t put enough detail for a computer to follow (I.e. open the fridge etc) and the teacher will run around bumping into things. That used to be a teaching exercise to show people the amount of precision required when telling a machine what to do. Now with LLMs, I think one of the great leaps is the idea that it’s no longer necessary to be “pedantic” when giving computers instructions because LLMs have somehow learned to fill in the blanks with a similar shared “understanding” of the world that we have (I.e. cheese is stored in the fridge so you have to go open the fridge to fetch the cheese for the sandwich).
- hattmall 3y ago
  I don't get the "magic" people are seeing. It makes sense. >LLMs have somehow learned to fill in the blanks It's not somehow, it's because they have read a ton of books, documents, etc and can make enough links between cheese and refrigerator and follow that back to know that a refrigerator needs to be opened. I have seen a lot of very clever AI examples using the latest tools, but I haven't seen anything that seems difficult to deconstruct.
  - lee101 3y ago
    [dead]
- deleted 3y ago
  [deleted]
- prox 3y ago
  Don’t forget they hold vector spaces. So fridge and cheese score high together for cohesiveness, but fridge and Antarctica less so, but both have something to do with cold. Together with all the training on texts creates a good ability to make inferences and “conclusions”. It has a net of lines of meanings of all concepts we fed it that give it the ability that it has, without actually understanding.
tornato7 3y ago
One experiment that I would love to see is an LLM-like model for audio. Feed it hours and hours of lectures, sound effects, animal calls, music etc. You would be able to talk to it and it would ingest the raw waveform then produce audio as a response. Would it learn the fundamentals of music theory? Would it learn to produce "the sound of a bowling ball hitting a dozen windchimes?" Would it learn to talk in English and communicate with whales? We've already done text and images, now someone please do sound!
- cornel_io 3y ago
  Uhhh...this is out there, from like a dozen different groups. Not going to do a full Googling for you on my phone because it's literally everywhere but "LLM for audio" gives https://ai.googleblog.com/2022/10/audiolm-language-modeling-approach-to.html https://ai.googleblog.com/2022/10/audiolm-language-modeling-... as the first result...some of this stuff is already really impressive.
- thomastjeffery 3y ago
  > Would it learn the fundamentals of music theory? No, but you might convince yourself it did. It would map the patterns that exist in its training set. It would then follow those patterns. The result would look like a human understanding music theory, but it would not be that. It would be stumbling around exactly the domain we gave it: impressive because that domain is not noise, it's good data. It still wouldn't be able to find its way around, only stumble.
  - satvikpendem 3y ago
    > The result would look like a human understanding music theory, but it would not be that. The question then becomes, what is understand? Is what a human does any different than what this LLM is doing?
    - thomastjeffery 3y ago
      Objectivity. A human can do something with the model. An LLM can only present the model to you.
kelseyfrog 3y ago
This has Embassytown vibes and I don't like it.

fwlr 3y ago

The idea that text contains more information than language is fascinating, wow. Thank you!

blowski 3y ago

I don't really understand your distinction between language and text, but it sounds intrguing. Would you be able to give more detail? I searched but couldn't find anything that seemed to explain it.

j16sdiz 3y ago
Spoken language have pitch, stress, mood, etc.. Written text contain some, but not all. Text, on the other hand, can present in list or table, with varies formatting, indentation. You can't reproduce them in spoken text.
bruce511 3y ago
Not the poster, but for me it comes down to a mix of clarity and permanence. I teach both verbally (interactive question/answer) and I've also written text books. Verbally by language is "loose". I'll say class when I mean object, unicode when I mean utf-8 and so on. Sentences are not all well formed, and sometimes change mid-thought. It's very "real time" Writing is a lot more deliberate. I have to be sure of each fact I state. I often re-test things I'm only 95% sure about. I edit, restructure, remove, add, until I'm happy. Of course all communication falls on a spectrum. Think phone call at one end, text book on the other. When I do a verbal lecture I'm usually careful with my speech, and when I post on hacker-news less rigorous. Language covers all of it. Text skews to the more deliberate side. Cunningly the language models are trained using (mostly) text, not speech. That will have an impact on them.
espe 3y ago
from a linguistic standpoint a text is a whole lot more than language: it is an externalisation of thought that is fixed onto a medium using writing utensils and most of all, cultural norms in the form of a wild variety of different genres and forms of text, ranging from something like a stream of consciousness to something like a speech act. furthermore, text can be conceptually written or spoken and with the internet we got an explosion of text that is conceptually spoken. those are the things OP might be refering to in regards to the "entropy" that encodes much more than just the tokens themselves.
gmt2027 3y ago
Text is an instance of language. Think of it as the difference between the python language and a large collection of python programs. The language describes syntactic and semantic rules, the collection is a sampling of possible programs that encodes a significant amount of information about the world. You could learn a lot about the laws of nature, the internet, even human society and laws by examining all the python programs ever written. An extreme version of the same idea is the difference between understanding DNA vs the genome of every individual organism that has lived on earth. The species record encodes a ton of information about the laws of nature, the composition and history of our planet. You could deduce physical laws and constants from looking at this information, wars and natural disasters, economic performance, historical natural boundaries, the industrial revolution and a lot more.
- Dwolb 3y ago
  This is a fascinating point. Let me see if I can play this back. If a student studies DNA sequencing, they’ll learn about the compounds that make up DNA, how traits get encoded, etc. Therefore the student might expect an AI trained on people’s DNA to be able to tell you about whether certain traits are more prevalent in one geography or the other. However, since DNA responds to changes in environment, the AI would start to see time, population, and geography-based patterns emerge. The AI for example could infer that a given person in the US who’s settled in NYC had ancestors from a given region of the world who left due to an environmental disaster just by looking at a given DNA sequence. To the student this result would look like magic. But in the end, it’s a result of individual’s DNA having much more information encoded in it than just human traits.
- espe 3y ago
  text and language intersect. in some ways, text is a superset of language, mostly due to social, or what is also called pragmatic, factors that complement semantics. also, the semantics/syntax interface is everything else than clear cut, at least in natural human languages.
  - catgoose 3y ago
    I think about this as shape and color. No one ever saw a shape that wasn’t colored and likewise there are no colored things that do not have a shape. Also, displaying text without a font is not possible. Text is the surface of the ocean where waves emerge, and while they have their own properties and may seem to naively have agency, they are an expression of the underlying ocean.

zainhoda 3y ago

There’s an exercise that some people do when learning programming, which is to write down the steps to make a sandwich. Then the teacher follows the exact instructions to make a sandwich and most people don’t put enough detail for a computer to follow (I.e. open the fridge etc) and the teacher will run around bumping into things. That used to be a teaching exercise to show people the amount of precision required when telling a machine what to do. Now with LLMs, I think one of the great leaps is the idea that it’s no longer necessary to be “pedantic” when giving computers instructions because LLMs have somehow learned to fill in the blanks with a similar shared “understanding” of the world that we have (I.e. cheese is stored in the fridge so you have to go open the fridge to fetch the cheese for the sandwich).

hattmall 3y ago
I don't get the "magic" people are seeing. It makes sense. >LLMs have somehow learned to fill in the blanks It's not somehow, it's because they have read a ton of books, documents, etc and can make enough links between cheese and refrigerator and follow that back to know that a refrigerator needs to be opened. I have seen a lot of very clever AI examples using the latest tools, but I haven't seen anything that seems difficult to deconstruct.
- lee101 3y ago
  [dead]
deleted 3y ago
[deleted]
prox 3y ago
Don’t forget they hold vector spaces. So fridge and cheese score high together for cohesiveness, but fridge and Antarctica less so, but both have something to do with cold. Together with all the training on texts creates a good ability to make inferences and “conclusions”. It has a net of lines of meanings of all concepts we fed it that give it the ability that it has, without actually understanding.

tornato7 3y ago

One experiment that I would love to see is an LLM-like model for audio. Feed it hours and hours of lectures, sound effects, animal calls, music etc. You would be able to talk to it and it would ingest the raw waveform then produce audio as a response. Would it learn the fundamentals of music theory? Would it learn to produce "the sound of a bowling ball hitting a dozen windchimes?" Would it learn to talk in English and communicate with whales? We've already done text and images, now someone please do sound!

cornel_io 3y ago
Uhhh...this is out there, from like a dozen different groups. Not going to do a full Googling for you on my phone because it's literally everywhere but "LLM for audio" gives https://ai.googleblog.com/2022/10/audiolm-language-modeling-approach-to.html https://ai.googleblog.com/2022/10/audiolm-language-modeling-... as the first result...some of this stuff is already really impressive.
thomastjeffery 3y ago
> Would it learn the fundamentals of music theory? No, but you might convince yourself it did. It would map the patterns that exist in its training set. It would then follow those patterns. The result would look like a human understanding music theory, but it would not be that. It would be stumbling around exactly the domain we gave it: impressive because that domain is not noise, it's good data. It still wouldn't be able to find its way around, only stumble.
- satvikpendem 3y ago
  > The result would look like a human understanding music theory, but it would not be that. The question then becomes, what is understand? Is what a human does any different than what this LLM is doing?
  - thomastjeffery 3y ago
    Objectivity. A human can do something with the model. An LLM can only present the model to you.

kelseyfrog 3y ago

This has Embassytown vibes and I don't like it.