5 ms·

> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.

by dvt 3y ago

Davidzheng 3y ago
But because the tokens are not word level they are capable of making up words that humans have never used before.
- dvt 3y ago
  Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
  - _ea1k 3y ago
    You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".
  - jstanley 3y ago
    Does this satisfy you? Me: give me a random character sequence that is not a pronounceable word ChatGPT: Sure, here's a random character sequence that is not a pronounceable word: "Jkplqxzv"
K0balt 3y ago
It is possible to draw new conclusions from complex data. It’s not something that a human might not also see, just that it’s an undiscovered relationship. (Or at least not directly expressed except by inference in the training corpus)
cle 3y ago
Test it! > Can you make up a word? > Certainly! Here's a made-up word for you: > *Glimberfleck*: The sparkling reflection of sunlight that dances on water, making it seem like the surface is sprinkled with tiny, shimmering gems. > Feel free to use it or let me know if you'd like me to create another word! > When I’m looking out of the window of an airplane at the ocean, am I looking at a glimberfleck? > Yes, if you're looking out of the airplane window at the ocean and observing the sparkling reflections of sunlight on the water's surface, you could describe that beautiful phenomenon as a "glimberfleck." It's a whimsical term for those captivating, shimmering patterns you see.
zarzavat 3y ago
It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition. Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.
- dvt 3y ago
  > It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.
  - jstanley 3y ago
    It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
    - deleted 3y ago
      [deleted]
  - deleted 3y ago
    [deleted]
- visarga 3y ago
  Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies. Some characteristics of hackernewsitis include: - Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed. - Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them. - Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream. - Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals. - Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains. - Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence. So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics. (Claude) Me: Does this look like AI not understanding?
squeaky-clean 3y ago
Embeddings are usually common n-grams of bytes, not whole words unless it's a very common word like "it". It's also not all English. > Hey ChatGPT, can you give me a brief explanation of AI embeddings, but respond in Chinese? > 当然可以。 > AI 嵌入（embeddings）是一种将词汇、句子或其他类型的信息转化为高维度的向量的技术。这些向量捕捉到了原始信息的某些语义或关系特性。通过嵌入，我们可以更容易地在高维空间中比较和处理这些信息，如在机器学习模型中。 > 希望这有助于你了解！ As for saying it can only mimic words it knew in it's training data, this one gives zero results in google > Hey ChatGPT can you make up a word for when I'm hungry but there's nothing in my kitchen I want to eat, so I'm not that hungry. > Sure, how about "snackstipated"? It's a blend of "snack" and "constipated", suggesting a feeling of hunger but being stuck with unsatisfying options.

> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.

by dvt 3y ago

> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.

Davidzheng 3y ago

But because the tokens are not word level they are capable of making up words that humans have never used before.

dvt 3y ago
Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
- _ea1k 3y ago
  You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".
- jstanley 3y ago
  Does this satisfy you? Me: give me a random character sequence that is not a pronounceable word ChatGPT: Sure, here's a random character sequence that is not a pronounceable word: "Jkplqxzv"

K0balt 3y ago

It is possible to draw new conclusions from complex data. It’s not something that a human might not also see, just that it’s an undiscovered relationship. (Or at least not directly expressed except by inference in the training corpus)

cle 3y ago

Test it! > Can you make up a word? > Certainly! Here's a made-up word for you: > *Glimberfleck*: The sparkling reflection of sunlight that dances on water, making it seem like the surface is sprinkled with tiny, shimmering gems. > Feel free to use it or let me know if you'd like me to create another word! > When I’m looking out of the window of an airplane at the ocean, am I looking at a glimberfleck? > Yes, if you're looking out of the airplane window at the ocean and observing the sparkling reflections of sunlight on the water's surface, you could describe that beautiful phenomenon as a "glimberfleck." It's a whimsical term for those captivating, shimmering patterns you see.

zarzavat 3y ago

It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition. Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.

dvt 3y ago
> It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.
- jstanley 3y ago
  It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
  - deleted 3y ago
    [deleted]
- deleted 3y ago
  [deleted]
visarga 3y ago
Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies. Some characteristics of hackernewsitis include: - Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed. - Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them. - Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream. - Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals. - Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains. - Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence. So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics. (Claude) Me: Does this look like AI not understanding?

squeaky-clean 3y ago

Embeddings are usually common n-grams of bytes, not whole words unless it's a very common word like "it". It's also not all English. > Hey ChatGPT, can you give me a brief explanation of AI embeddings, but respond in Chinese? > 当然可以。 > AI 嵌入（embeddings）是一种将词汇、句子或其他类型的信息转化为高维度的向量的技术。这些向量捕捉到了原始信息的某些语义或关系特性。通过嵌入，我们可以更容易地在高维空间中比较和处理这些信息，如在机器学习模型中。 > 希望这有助于你了解！ As for saying it can only mimic words it knew in it's training data, this one gives zero results in google > Hey ChatGPT can you make up a word for when I'm hungry but there's nothing in my kitchen I want to eat, so I'm not that hungry. > Sure, how about "snackstipated"? It's a blend of "snack" and "constipated", suggesting a feeling of hunger but being stuck with unsatisfying options.