4 ms·

what is the benefit to such splitting of text based on seemingly meaningless lines? isn't there a better way to do it?

rmellow 4y ago
Breaking text into sub-word units... 1. Greatly reduces memory usage. Instead of memorizing every inflection of the word "walk", it memorizes the root (walk) and the modifiers (ing, ed, er, ...). These modifiers can be reused for other words. 2. Allows for word compositions that weren't in the training set. This is great for uncommon or new expressions like "googlification" or "unalive".
- shagie 4y ago
  The walk example doesn't quite hold up. If you put: test walk walker walking walked into the tokenizer you will see the following tokens: [test][ walk][ walk][er][ walking][ walked] Only walker is broken up into two different tokens. I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens. For even more fun, [walker] is a distinct token if it doesn't include the leading space. test walker floorwalker foowalker becomes: [test][ walk][er][ floor][walker][ fo][ow][alker] How we think of words doesn't cleanly map to tokens. (Late edit) walker floorwalker becomes tokenized as: [walker][ floor][walker] So in that case, they're the same token. It's curious how white space influences the word to token making.
  - nonfamous 4y ago
    There’s no syntax or structure to the token set. The actual tokens were algorithmically selected based on the training data to (putting things loosely) optimize compression of the training data given a token set size.
- PeterisP 4y ago
  Sure, but what I'm hearing in the parent post is a question about why we don't use linguistically motivated subword units (of similar length/vocabulary size and thus memory usage) e.g. cutting across morpheme boundaries instead of whatever an algorithm like BPE caclulates.
- LeonB 4y ago
  Gratitudes for the explanatories, trousering this for rethinkalysing tomorn.
roel_v 4y ago
There is no 'meaning' inside these AI's. It's terribly confusing to think about these LLM's as having 'meaning' in the same way we humans do. It's all just statistics. Given a sequence of numbers (each representing some abstract token), what is most likely to come next. That's how 'simple' it is. It's also what makes it so amazing that these things work as well as they do. I giggle like a schoolgirl every time I get it to add some functionality to a function, or write an entire new function, and that's several times a day for what is now months on end. But the key to using them is seeing that there is no 'meaning' in them. It's all just streams of (to the machine) meaningless tokens.
- goldfeld 4y ago
  If it just decides on a single token at a time, can it backtrack and choose differently under that operation, given the next tokens? What I wonder is, how can it plan ahead and output meaningful (to us) responses, like working code or useful articles? How can it "reason" logically when it needs to solve a problem, a riddle etc, by only selecting a token at a time? Wouldn't that dumbed down approach prove myopic for complex compositions? Doesn't it need some over-ruling goal-based heuristic system?
  - psychphysic 4y ago
    Bard at least produces multiple drafts. I believe that is preferred over backtracking. Generation is ultimately deterministic (seeded prng) so backtracking wouldn't make sense.
  - wingspar 4y ago
    There’s no planning, no reason. It’s all ‘what word is next…’ I found Stephen Wolframs explanation helpful. He has a YouTube video version which I enjoyed too. This blog post was on HN last month, but I never get good search results on hn https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
    - zeven7 4y ago
ruuda 4y ago
You could split on words instead of tokens, but then you need a large vocabulary, you can't deal with inputs that contain a word which is not in the vocabulary, and it's not so clear what a "word" even is. Instead of coming up with more and more heuristics to chop a sequence of bytes up in "words" in a vocabulary, we could simply set a limit on the size of the vocabulary (number of tokens), put all bytes in there (so we can at least handle any input byte by byte), and pack the remaining space with the most common multi-byte byte sequences. Then you end up with tokens like here.
sharkjacobs 4y ago
ChatGPT models syntax, not semantics There's no "better way" to do it because the tokens are all meaningless to ChatGPT, it only cares about how efficiently they can be parsed and processed. The competing desires are to model all language with the biggest tokens possible, and the fewest tokens possible. The lines aren't meaningless, text is split into the largest possible chunks using a set of the most common tokens. Common words, like "the", "fast", "unity", "flying" are all tokens, but it's not because they're words, it's because they're common letter clusters, undistinguished from "fl", "ing", "un", "ple" "gadflying" is tokenized into [g, ad, flying], even though it's only loosely semantically related to "flying", it's just the most efficient way to tokenize it.

Breaking text into sub-word units... 1. Greatly reduces memory usage. Instead of memorizing every inflection of the word "walk", it memorizes the root (walk) and the modifiers (ing, ed, er, ...). These modifiers can be reused for other words. 2. Allows for word compositions that weren't in the training set. This is great for uncommon or new expressions like "googlification" or "unalive".

shagie 4y ago
The walk example doesn't quite hold up. If you put: test walk walker walking walked into the tokenizer you will see the following tokens: [test][ walk][ walk][er][ walking][ walked] Only walker is broken up into two different tokens. I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens. For even more fun, [walker] is a distinct token if it doesn't include the leading space. test walker floorwalker foowalker becomes: [test][ walk][er][ floor][walker][ fo][ow][alker] How we think of words doesn't cleanly map to tokens. (Late edit) walker floorwalker becomes tokenized as: [walker][ floor][walker] So in that case, they're the same token. It's curious how white space influences the word to token making.
- nonfamous 4y ago
  There’s no syntax or structure to the token set. The actual tokens were algorithmically selected based on the training data to (putting things loosely) optimize compression of the training data given a token set size.
PeterisP 4y ago
Sure, but what I'm hearing in the parent post is a question about why we don't use linguistically motivated subword units (of similar length/vocabulary size and thus memory usage) e.g. cutting across morpheme boundaries instead of whatever an algorithm like BPE caclulates.
LeonB 4y ago
Gratitudes for the explanatories, trousering this for rethinkalysing tomorn.

roel_v 4y ago

There is no 'meaning' inside these AI's. It's terribly confusing to think about these LLM's as having 'meaning' in the same way we humans do. It's all just statistics. Given a sequence of numbers (each representing some abstract token), what is most likely to come next. That's how 'simple' it is. It's also what makes it so amazing that these things work as well as they do. I giggle like a schoolgirl every time I get it to add some functionality to a function, or write an entire new function, and that's several times a day for what is now months on end. But the key to using them is seeing that there is no 'meaning' in them. It's all just streams of (to the machine) meaningless tokens.

goldfeld 4y ago
If it just decides on a single token at a time, can it backtrack and choose differently under that operation, given the next tokens? What I wonder is, how can it plan ahead and output meaningful (to us) responses, like working code or useful articles? How can it "reason" logically when it needs to solve a problem, a riddle etc, by only selecting a token at a time? Wouldn't that dumbed down approach prove myopic for complex compositions? Doesn't it need some over-ruling goal-based heuristic system?
- psychphysic 4y ago
  Bard at least produces multiple drafts. I believe that is preferred over backtracking. Generation is ultimately deterministic (seeded prng) so backtracking wouldn't make sense.
- wingspar 4y ago
  There’s no planning, no reason. It’s all ‘what word is next…’ I found Stephen Wolframs explanation helpful. He has a YouTube video version which I enjoyed too. This blog post was on HN last month, but I never get good search results on hn https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
  - zeven7 4y ago

ruuda 4y ago

You could split on words instead of tokens, but then you need a large vocabulary, you can't deal with inputs that contain a word which is not in the vocabulary, and it's not so clear what a "word" even is. Instead of coming up with more and more heuristics to chop a sequence of bytes up in "words" in a vocabulary, we could simply set a limit on the size of the vocabulary (number of tokens), put all bytes in there (so we can at least handle any input byte by byte), and pack the remaining space with the most common multi-byte byte sequences. Then you end up with tokens like here.

sharkjacobs 4y ago

ChatGPT models syntax, not semantics There's no "better way" to do it because the tokens are all meaningless to ChatGPT, it only cares about how efficiently they can be parsed and processed. The competing desires are to model all language with the biggest tokens possible, and the fewest tokens possible. The lines aren't meaningless, text is split into the largest possible chunks using a set of the most common tokens. Common words, like "the", "fast", "unity", "flying" are all tokens, but it's not because they're words, it's because they're common letter clusters, undistinguished from "fl", "ing", "un", "ple" "gadflying" is tokenized into [g, ad, flying], even though it's only loosely semantically related to "flying", it's just the most efficient way to tokenize it.