9 ms·

I don't really understand your distinction between language and text, but it sounds intrguing. Would you be able to give more detail? I searched but couldn't fi

I don't really understand your distinction between language and text, but it sounds intrguing. Would you be able to give more detail? I searched but couldn't find anything that seemed to explain it.

j16sdiz 3y ago
Spoken language have pitch, stress, mood, etc.. Written text contain some, but not all. Text, on the other hand, can present in list or table, with varies formatting, indentation. You can't reproduce them in spoken text.
bruce511 3y ago
Not the poster, but for me it comes down to a mix of clarity and permanence. I teach both verbally (interactive question/answer) and I've also written text books. Verbally by language is "loose". I'll say class when I mean object, unicode when I mean utf-8 and so on. Sentences are not all well formed, and sometimes change mid-thought. It's very "real time" Writing is a lot more deliberate. I have to be sure of each fact I state. I often re-test things I'm only 95% sure about. I edit, restructure, remove, add, until I'm happy. Of course all communication falls on a spectrum. Think phone call at one end, text book on the other. When I do a verbal lecture I'm usually careful with my speech, and when I post on hacker-news less rigorous. Language covers all of it. Text skews to the more deliberate side. Cunningly the language models are trained using (mostly) text, not speech. That will have an impact on them.
espe 3y ago
from a linguistic standpoint a text is a whole lot more than language: it is an externalisation of thought that is fixed onto a medium using writing utensils and most of all, cultural norms in the form of a wild variety of different genres and forms of text, ranging from something like a stream of consciousness to something like a speech act. furthermore, text can be conceptually written or spoken and with the internet we got an explosion of text that is conceptually spoken. those are the things OP might be refering to in regards to the "entropy" that encodes much more than just the tokens themselves.
gmt2027 3y ago
Text is an instance of language. Think of it as the difference between the python language and a large collection of python programs. The language describes syntactic and semantic rules, the collection is a sampling of possible programs that encodes a significant amount of information about the world. You could learn a lot about the laws of nature, the internet, even human society and laws by examining all the python programs ever written. An extreme version of the same idea is the difference between understanding DNA vs the genome of every individual organism that has lived on earth. The species record encodes a ton of information about the laws of nature, the composition and history of our planet. You could deduce physical laws and constants from looking at this information, wars and natural disasters, economic performance, historical natural boundaries, the industrial revolution and a lot more.
- Dwolb 3y ago
  This is a fascinating point. Let me see if I can play this back. If a student studies DNA sequencing, they’ll learn about the compounds that make up DNA, how traits get encoded, etc. Therefore the student might expect an AI trained on people’s DNA to be able to tell you about whether certain traits are more prevalent in one geography or the other. However, since DNA responds to changes in environment, the AI would start to see time, population, and geography-based patterns emerge. The AI for example could infer that a given person in the US who’s settled in NYC had ancestors from a given region of the world who left due to an environmental disaster just by looking at a given DNA sequence. To the student this result would look like magic. But in the end, it’s a result of individual’s DNA having much more information encoded in it than just human traits.
- espe 3y ago
  text and language intersect. in some ways, text is a superset of language, mostly due to social, or what is also called pragmatic, factors that complement semantics. also, the semantics/syntax interface is everything else than clear cut, at least in natural human languages.
  - catgoose 3y ago
    I think about this as shape and color. No one ever saw a shape that wasn’t colored and likewise there are no colored things that do not have a shape. Also, displaying text without a font is not possible. Text is the surface of the ocean where waves emerge, and while they have their own properties and may seem to naively have agency, they are an expression of the underlying ocean.

I don't really understand your distinction between language and text, but it sounds intrguing. Would you be able to give more detail? I searched but couldn't find anything that seemed to explain it.

j16sdiz 3y ago

Spoken language have pitch, stress, mood, etc.. Written text contain some, but not all. Text, on the other hand, can present in list or table, with varies formatting, indentation. You can't reproduce them in spoken text.

bruce511 3y ago

Not the poster, but for me it comes down to a mix of clarity and permanence. I teach both verbally (interactive question/answer) and I've also written text books. Verbally by language is "loose". I'll say class when I mean object, unicode when I mean utf-8 and so on. Sentences are not all well formed, and sometimes change mid-thought. It's very "real time" Writing is a lot more deliberate. I have to be sure of each fact I state. I often re-test things I'm only 95% sure about. I edit, restructure, remove, add, until I'm happy. Of course all communication falls on a spectrum. Think phone call at one end, text book on the other. When I do a verbal lecture I'm usually careful with my speech, and when I post on hacker-news less rigorous. Language covers all of it. Text skews to the more deliberate side. Cunningly the language models are trained using (mostly) text, not speech. That will have an impact on them.

espe 3y ago

from a linguistic standpoint a text is a whole lot more than language: it is an externalisation of thought that is fixed onto a medium using writing utensils and most of all, cultural norms in the form of a wild variety of different genres and forms of text, ranging from something like a stream of consciousness to something like a speech act. furthermore, text can be conceptually written or spoken and with the internet we got an explosion of text that is conceptually spoken. those are the things OP might be refering to in regards to the "entropy" that encodes much more than just the tokens themselves.

gmt2027 3y ago

Text is an instance of language. Think of it as the difference between the python language and a large collection of python programs. The language describes syntactic and semantic rules, the collection is a sampling of possible programs that encodes a significant amount of information about the world. You could learn a lot about the laws of nature, the internet, even human society and laws by examining all the python programs ever written. An extreme version of the same idea is the difference between understanding DNA vs the genome of every individual organism that has lived on earth. The species record encodes a ton of information about the laws of nature, the composition and history of our planet. You could deduce physical laws and constants from looking at this information, wars and natural disasters, economic performance, historical natural boundaries, the industrial revolution and a lot more.

Dwolb 3y ago
This is a fascinating point. Let me see if I can play this back. If a student studies DNA sequencing, they’ll learn about the compounds that make up DNA, how traits get encoded, etc. Therefore the student might expect an AI trained on people’s DNA to be able to tell you about whether certain traits are more prevalent in one geography or the other. However, since DNA responds to changes in environment, the AI would start to see time, population, and geography-based patterns emerge. The AI for example could infer that a given person in the US who’s settled in NYC had ancestors from a given region of the world who left due to an environmental disaster just by looking at a given DNA sequence. To the student this result would look like magic. But in the end, it’s a result of individual’s DNA having much more information encoded in it than just human traits.
espe 3y ago
text and language intersect. in some ways, text is a superset of language, mostly due to social, or what is also called pragmatic, factors that complement semantics. also, the semantics/syntax interface is everything else than clear cut, at least in natural human languages.
- catgoose 3y ago
  I think about this as shape and color. No one ever saw a shape that wasn’t colored and likewise there are no colored things that do not have a shape. Also, displaying text without a font is not possible. Text is the surface of the ocean where waves emerge, and while they have their own properties and may seem to naively have agency, they are an expression of the underlying ocean.