9 ms·

Probably pay attention to tokenizers

321 points by ingve 2y ago

cebu_blue 2y ago
[flagged]
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
Spivak 2y ago
I think I take something different away from the article, yes tokenizers are important but they're a means to get at something much much bigger which is how to clean up and normalize unstructured data. It's a current endeavor of mine at $dayjob for how to do this in a way that can work reasonably well even for badly mangled documents. I don't have any silver bullets, at least nothing worthy of a blog-post yet, but since this is needed when dealing with OCR documents so "post-ocr correction" turns up quite a few different approaches. And this is an aside, but I see folks using LLMs to do this correction in the first place. I don't think using LLMs to do correction in a multi-pass system is inherently bad but I haven't been able to get good results out of "call/response" (i.e. a prompt to clean up this text). The best results are when you're running an LLM locally and cleaning incrementally by using token probabilities to help guide you. You get some candidate words from your wordlist based on the fuzzy match of the text you do have, and candidate words predicted from the previous text and when both align -- ding! It's (obviously) not the fastest method however.
- SEGyges 2y ago
  you might have better luck giving the LM the original document and having it generate its own OCR independently, then asking the llm to tiebreak between its own generation and the OCR output while the image is still in the context window until it is satisfied that it got things correct
- 7thpower 2y ago
  This is interesting. What types of content are you using this approach on and how does it handle semi structured data? For instance, embedded tables.
kelseyfrog 2y ago
Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the tokenization. LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. GPT-4o, for example, writes a small python program and executes it in order to count letter instances. We all know that tokenization effectively erases knowledge about letters in prompts and directly negatively impacts performance at these tasks, yet we haven't found a way to solve it. 1. https://ar5iv.labs.arxiv.org/html/2310.02989 https://ar5iv.labs.arxiv.org/html/2310.02989
- IncreasePosts 2y ago
  What's the issue with character-level tokenization(I assume this would be much better at count-the-letter tasks)? The article mentions it as an option but doesn't talk about why subword tokenization is preferred by most of the big LLMs out there.
  - SEGyges 2y ago
    tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attention calculation. you can hypothetically try to ameliorate this by other means, but if you just naively drop from tokenization to character or byte level models this is what goes wrong
    - p1esk 2y ago
      4x seq length expansion doesn’t sound that bad.
      - lechatonnoir 2y ago
        I mean, it's not completely fatal, but it means an approximately 16x increase in runtime cost, if I'm not mistaken. That's probably not worth trying to solve letter counting in most applications.
Joker_vD 2y ago
> You need to understand [the input data] before you can do anything meaningful with it. IMHO that's the main reason people turn to any sort of automated data-processing tools in the first place: they don't want to look at the input data. They'd rather have "the computer" look at it and maybe query them back with some additional info gathering requests. But thinking on their own? Ugh. So I boldly propose the new definition of AGI: it's the data-processing entity that will (at last!) reliably liberate you from having to look at your data before you start shoving this data into that processing entity.
- bunderbunder 2y ago
  Over the past year I've encountered so many situations where a person's opinion of how well an LLM accomplishes a task actually says more about that person's reading comprehension skills than it does the LLM's performance. This applies to both positive and negative opinions.
ratedgene 2y ago
Can't someone expand on this > Chunking is more or less a fixable problem with some clever techniques: these are pretty well documented around the internet; Curious about what chunking solutions are out there for different sets of data/problems
- pphysch 2y ago
  Most data has semantic boundaries: whether tokens, words, lines, paragraphs, blocks, sections, articles, chapters, versions, etc. and ideally the chunking algorithm will align with those boundaries in the actual data. But there is a lot of variety.
- hansvm 2y ago
  It's only "solved" if you're okay with a 50-90% retrieval rate or have particularly nice data. There's a lot of stuff like "referencing the techniques from Chapter 2 we do <blah>" in the wild, and any chunking solution is unlikely to correctly answer queries involving both Chapter 2 and <blah>, at least not without significant false positive rates. That said, the chunking people are doing is worse than the SOTA. The core thing you want to do is understand your data well enough to ensure that any question, as best as possible, has relevant data within a single chunk. Details vary (maybe the details are what you're asking for?).
- haolez 2y ago
  I had some success with simple aliasing at the beginning and end of chunks. In my next project, I'll try an idea that I saw somewhere: 1. do naive chunking like before 2. calculate the embeddings of each chunk 3. clusterize the chunks by their embeddings to see which chunks actually bring new information to the corpus 4. summarize similar chunks into smaller chunks Sounds like a smart way of using embeddings to reduce the amount of context misses. I'm not sure it works well, though :)
bcherry 2y ago
It's kind of interesting because I think most people implementing RAG aren't even thinking about tokenization at all. They're thinking about embeddings: 1. chunk the corpus of data (various strategies but they're all somewhat intuitive) 2. compute embedding for each chunk 3. generate search query/queries 4. compute embedding for each query 5. rank corpus chunks by distance to query (vector search) 6. construct return values (e.g chunk + surrounding context, or whole doc, etc) So this article really gets at the importance of a hidden, relatively mundane-feeling, operation that occurs which can have an outsized impact on the performance of the system. I do wish it had more concrete recommendations in the last section and code sample of a robust project with normalization, fine-tuning, and eval.
yoelhacks 2y ago
I used to work on an app that very heavily leaned on Elasticsearch to do advanced text querying for similarities between a 1-2 sentence input and a corpus of paragraph+ length documents. It was fascinating how much tokenization strategies could affect a particular subset of queries. A really great example is a "W-4" or "W4" Standard tokenization might split on the "-" or split on letter / number boundaries. That input now becomes completely unidentifiable in the index, when it otherwise would have been a very rich factor in matching HR / salary / tax related content. Different domain, but this doesn't shock me at all.
- carom 2y ago
  The trained embedding vectors for the token equivalents of W4 and W-4 would be mapped to a similar space due to their appearance in the same contexts.
  - dangerlibrary 2y ago
    The point of the GP post is that the "w-4" token had very different results from ["w", "-4"] or similar algorithms where the "w" and "4" wound up in separate tokens.
- AStrangeMorrow 2y ago
  Yes, used to work on a system that has elasticsearch and also some custom Word2Vec models. What had the most impact on the quality of the search is ES and on the quality of our W2V model were tokenization and a custom ngrams system.
Xenoamorphous 2y ago
> One of the things I noticed over the past year is how a lot of developers who are used to developing in the traditional (deterministic) space fail to change the way they should think about problems in the statistical space which is ultimately what LLM apps are. I’m a developer and don’t struggle with this, where I really struggle is trying to explain this to users.
halyax7 2y ago
an issue I've seen in several RAG implementations is assuming that the target documents, however cleverly they're chunked, will be good search keys for incoming queries. Unless your incoming search text looks semantically like the documents you're searching over (not the case in general), you'll get bad hits. On a recent project, we saw a big improvement in retrieval relevance when we separated the search keys from the returned values (chunked documents), and we used an LM to generate appropriate keys which were then embedded. Appropriate in this case means "sentences like what the user might input if theyre expecting this chunk back"
- marlott 2y ago
  Interesting! So you basically got a LM to rephrase the search phrase/keys into the style of the target documents, then used that in the RAG pipeline? Did you do an initial search first to limit the documents?
  - NitpickLawyer 2y ago
    IIUC they're doing some sort of "q/a" for each chunk from documents, where they ask an LLM to "play the user role and ask a question that would be answered by this chunk". They then embed those questions, and match live user queries with those questions first, then maybe re-rank on the document chunks retrieved.
    - deleted 2y ago
      [deleted]
quirkot 2y ago
Is this true? >> Do not panic! A lot of the large LLM vocabularies are pretty huge (30k-300k tokens large) Seems small by an order of magnitude (at least). English alone is 1+ millions words
- deleted 2y ago
  [deleted]
- mmoskal 2y ago
  Tokens are often sub-word, all the way down to bytes (which are implicitly understood as UTF8 but models will sometimes generate invalid UTF8...).
- macleginn 2y ago
  Most of these 1+ million words are almost never used, so 200k is plenty for English. Optimistically, we hope that rarer words would be longer and to some degree compositional (optim-ism, optim-istic, etc.), but unfortunately this is not what tokenisers arrive at (and you are more likely to get "opt-i-mis-m" or something like that). People have tried to optimise tokenisation and the main part of LLM training jointly, which leads to more sensible results, but this is unworkable for larger models, so we are stuck with inflated basic vocabularies. It is also probably possible now to go even for larger vocabularies, in the 1-2 million range (by factorising the embedding matrix, for example), but this does not lead to noticeable improvements in performance, AFAIK.
  - Der_Einzige 2y ago
    Performance would be massively improved on constrained text tasks. That alone makes it worth it to expand the vocabulary size.
- spott 2y ago
  BPE is complete. Every valid Unicode string can be encoded with any BPE tokenizer. BPE basically starts with a token for every valid value for a Unicode byte and then creates new tokens by looking at common pairs of bytes (‘t’ followed by ‘h’ becomes a new token ’th’)
HanClinto 2y ago
I really appreciated this blog post, and in particular I appreciated the segment talking about typos. We were discussing this earlier this week -- I'm helping with a RAG-like application for a project right now, and we're concerned with how much small typos or formatting differences in users' queries can throw off our embedding distances. One thought was: Should we be augmenting our training data (or at the very least, our pretraining data) with intentional typos / substitutions / capitalizations, just to help it learn that "wrk" and "work" are probably synonyms? I looked briefly around for typo augmentation for (pre)training, and didn't see anything at first blush, so I'm guessing that if this is a common practice, that it's called something else.
- tmikaeld 2y ago
  I work with full text search where this is common. Here is some points. Stemming: Reducing words to their base or root form (e.g., “working,” “worked” becoming “work”). Lemmatization: Similar to stemming, but more sophisticated, accounting for context (e.g., “better” lemmatizes to “good”). Token normalization: Standardizing tokens, such as converting “wrk” to “work” through predefined rules (case folding, character replacement). Fuzzy matching: Allowing approximate matches based on edit distance (e.g., “wrk” matches “work” due to minimal character difference). Phonetic matching: Matching words that sound similar, sometimes used to match abbreviations or common misspellings. Thesaurus-based search: Using a predefined list of synonyms or alternative spellings to expand search queries. Most of these are open and free lists you can use, check the sources on manticore search for example.
  - soared 2y ago
    Porter stemming is currently widely used in adtech for keywords.
  - thaumasiotes 2y ago
    > Lemmatization: Similar to stemming, but more sophisticated, accounting for context (e.g., “better” lemmatizes to “good”). I don't understand. How is that different from stemming? What's the base form of "better" if not "good"? The nature of the relationship between "better" and "good" is no different from that between "work" and "worked".
woolr 2y ago
Can't repro some of the numbers in this blog post, for example: from sentence_transformers import SentenceTransformer from sentence_transformers import util model = SentenceTransformer('all-MiniLM-L6-v2') data_to_check = [ "I have recieved wrong package", "I hve recieved wrong package" ] embeddings = model.encode(data_to_check) util.cos_sim(embeddings, embeddings) Outputs: tensor([[1.0000, 0.9749], [0.9749, 1.0000]])
- 1986 2y ago
  Your data differs from theirs - they have "I have received wrong package" vs "I hve received wrong pckage", you misspelled "received" in both and didn't omit an "a" from "package" in the "bad" data
andix 2y ago
This is an awesome article, but I’m missing the part where solutions for each of the problems were discussed. Run a spell check before tokenizing? Maybe even tokenize the misspelled word and the potential corrected word next to each other like „misspld (misspelled)“? For the issue with the brand names the tokenizer doesn’t know, I have no idea how to handle it. This problem is probably even worse in less common languages, or in languages which use a lot of compound words.
r_hanz 2y ago
Very nicely written article. Personally, I find RAG (and more abstractly, vector search) the only mildly interesting development in the latest LLM fad, and have always felt that LLMs sit way too far down the diminishing returns curve to be interesting. However, I can’t believe tokenization and embeddings in general, are not broadly considered the absolutely most paramount aspect of all deep learning. The latent space your model captures is the most important aspect of the whole pipeline, or else what is any deep learning model even doing?
cranium 2y ago
I finally understood the weirdness of tokenizers after watching the video Andrej Karpathy made: "Let's build the GPT Tokenizer" (https://www.youtube.com/watch?v=zduSFxRajkE https://www.youtube.com/watch?v=zduSFxRajkE). He goes through why we need them instead of raw byte sequences (too expensive) and how the Byte Pair Encoding algorithm works. Worth spending 2h for the deeper understanding if you deal with LLMs.
maytc 2y ago
The difference in the dates example seems right to me 20 October 2024 and 2024-20-10 are not the same. Months in different locales can be written as yyyy-MM-dd. It can also be a catalog/reference number. So, it seems right that their embedding similarity is not perfectly aligned. So, it's not a tokenizer problem. The text meant different things according to the LLM.
gavin_gee 2y ago
do pictograms represent a way to reduce tokens?

321 points by ingve 2y ago

cebu_blue 2y ago

[flagged]

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

Spivak 2y ago

I think I take something different away from the article, yes tokenizers are important but they're a means to get at something much much bigger which is how to clean up and normalize unstructured data. It's a current endeavor of mine at $dayjob for how to do this in a way that can work reasonably well even for badly mangled documents. I don't have any silver bullets, at least nothing worthy of a blog-post yet, but since this is needed when dealing with OCR documents so "post-ocr correction" turns up quite a few different approaches. And this is an aside, but I see folks using LLMs to do this correction in the first place. I don't think using LLMs to do correction in a multi-pass system is inherently bad but I haven't been able to get good results out of "call/response" (i.e. a prompt to clean up this text). The best results are when you're running an LLM locally and cleaning incrementally by using token probabilities to help guide you. You get some candidate words from your wordlist based on the fuzzy match of the text you do have, and candidate words predicted from the previous text and when both align -- ding! It's (obviously) not the fastest method however.

SEGyges 2y ago
you might have better luck giving the LM the original document and having it generate its own OCR independently, then asking the llm to tiebreak between its own generation and the OCR output while the image is still in the context window until it is satisfied that it got things correct
7thpower 2y ago
This is interesting. What types of content are you using this approach on and how does it handle semi structured data? For instance, embedded tables.

kelseyfrog 2y ago

Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the tokenization. LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. GPT-4o, for example, writes a small python program and executes it in order to count letter instances. We all know that tokenization effectively erases knowledge about letters in prompts and directly negatively impacts performance at these tasks, yet we haven't found a way to solve it. 1. https://ar5iv.labs.arxiv.org/html/2310.02989 https://ar5iv.labs.arxiv.org/html/2310.02989

IncreasePosts 2y ago
What's the issue with character-level tokenization(I assume this would be much better at count-the-letter tasks)? The article mentions it as an option but doesn't talk about why subword tokenization is preferred by most of the big LLMs out there.
- SEGyges 2y ago
  tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attention calculation. you can hypothetically try to ameliorate this by other means, but if you just naively drop from tokenization to character or byte level models this is what goes wrong
  - p1esk 2y ago
    4x seq length expansion doesn’t sound that bad.
    - lechatonnoir 2y ago
      I mean, it's not completely fatal, but it means an approximately 16x increase in runtime cost, if I'm not mistaken. That's probably not worth trying to solve letter counting in most applications.

Joker_vD 2y ago

> You need to understand [the input data] before you can do anything meaningful with it. IMHO that's the main reason people turn to any sort of automated data-processing tools in the first place: they don't want to look at the input data. They'd rather have "the computer" look at it and maybe query them back with some additional info gathering requests. But thinking on their own? Ugh. So I boldly propose the new definition of AGI: it's the data-processing entity that will (at last!) reliably liberate you from having to look at your data before you start shoving this data into that processing entity.

bunderbunder 2y ago
Over the past year I've encountered so many situations where a person's opinion of how well an LLM accomplishes a task actually says more about that person's reading comprehension skills than it does the LLM's performance. This applies to both positive and negative opinions.

ratedgene 2y ago

Can't someone expand on this > Chunking is more or less a fixable problem with some clever techniques: these are pretty well documented around the internet; Curious about what chunking solutions are out there for different sets of data/problems

pphysch 2y ago
Most data has semantic boundaries: whether tokens, words, lines, paragraphs, blocks, sections, articles, chapters, versions, etc. and ideally the chunking algorithm will align with those boundaries in the actual data. But there is a lot of variety.
hansvm 2y ago
It's only "solved" if you're okay with a 50-90% retrieval rate or have particularly nice data. There's a lot of stuff like "referencing the techniques from Chapter 2 we do <blah>" in the wild, and any chunking solution is unlikely to correctly answer queries involving both Chapter 2 and <blah>, at least not without significant false positive rates. That said, the chunking people are doing is worse than the SOTA. The core thing you want to do is understand your data well enough to ensure that any question, as best as possible, has relevant data within a single chunk. Details vary (maybe the details are what you're asking for?).
haolez 2y ago
I had some success with simple aliasing at the beginning and end of chunks. In my next project, I'll try an idea that I saw somewhere: 1. do naive chunking like before 2. calculate the embeddings of each chunk 3. clusterize the chunks by their embeddings to see which chunks actually bring new information to the corpus 4. summarize similar chunks into smaller chunks Sounds like a smart way of using embeddings to reduce the amount of context misses. I'm not sure it works well, though :)

bcherry 2y ago

It's kind of interesting because I think most people implementing RAG aren't even thinking about tokenization at all. They're thinking about embeddings: 1. chunk the corpus of data (various strategies but they're all somewhat intuitive) 2. compute embedding for each chunk 3. generate search query/queries 4. compute embedding for each query 5. rank corpus chunks by distance to query (vector search) 6. construct return values (e.g chunk + surrounding context, or whole doc, etc) So this article really gets at the importance of a hidden, relatively mundane-feeling, operation that occurs which can have an outsized impact on the performance of the system. I do wish it had more concrete recommendations in the last section and code sample of a robust project with normalization, fine-tuning, and eval.

yoelhacks 2y ago

I used to work on an app that very heavily leaned on Elasticsearch to do advanced text querying for similarities between a 1-2 sentence input and a corpus of paragraph+ length documents. It was fascinating how much tokenization strategies could affect a particular subset of queries. A really great example is a "W-4" or "W4" Standard tokenization might split on the "-" or split on letter / number boundaries. That input now becomes completely unidentifiable in the index, when it otherwise would have been a very rich factor in matching HR / salary / tax related content. Different domain, but this doesn't shock me at all.

carom 2y ago
The trained embedding vectors for the token equivalents of W4 and W-4 would be mapped to a similar space due to their appearance in the same contexts.
- dangerlibrary 2y ago
  The point of the GP post is that the "w-4" token had very different results from ["w", "-4"] or similar algorithms where the "w" and "4" wound up in separate tokens.
AStrangeMorrow 2y ago
Yes, used to work on a system that has elasticsearch and also some custom Word2Vec models. What had the most impact on the quality of the search is ES and on the quality of our W2V model were tokenization and a custom ngrams system.

Xenoamorphous 2y ago

> One of the things I noticed over the past year is how a lot of developers who are used to developing in the traditional (deterministic) space fail to change the way they should think about problems in the statistical space which is ultimately what LLM apps are. I’m a developer and don’t struggle with this, where I really struggle is trying to explain this to users.

halyax7 2y ago

an issue I've seen in several RAG implementations is assuming that the target documents, however cleverly they're chunked, will be good search keys for incoming queries. Unless your incoming search text looks semantically like the documents you're searching over (not the case in general), you'll get bad hits. On a recent project, we saw a big improvement in retrieval relevance when we separated the search keys from the returned values (chunked documents), and we used an LM to generate appropriate keys which were then embedded. Appropriate in this case means "sentences like what the user might input if theyre expecting this chunk back"

marlott 2y ago
Interesting! So you basically got a LM to rephrase the search phrase/keys into the style of the target documents, then used that in the RAG pipeline? Did you do an initial search first to limit the documents?
- NitpickLawyer 2y ago
  IIUC they're doing some sort of "q/a" for each chunk from documents, where they ask an LLM to "play the user role and ask a question that would be answered by this chunk". They then embed those questions, and match live user queries with those questions first, then maybe re-rank on the document chunks retrieved.
  - deleted 2y ago
    [deleted]

quirkot 2y ago

Is this true? >> Do not panic! A lot of the large LLM vocabularies are pretty huge (30k-300k tokens large) Seems small by an order of magnitude (at least). English alone is 1+ millions words

deleted 2y ago
[deleted]
mmoskal 2y ago
Tokens are often sub-word, all the way down to bytes (which are implicitly understood as UTF8 but models will sometimes generate invalid UTF8...).
macleginn 2y ago
Most of these 1+ million words are almost never used, so 200k is plenty for English. Optimistically, we hope that rarer words would be longer and to some degree compositional (optim-ism, optim-istic, etc.), but unfortunately this is not what tokenisers arrive at (and you are more likely to get "opt-i-mis-m" or something like that). People have tried to optimise tokenisation and the main part of LLM training jointly, which leads to more sensible results, but this is unworkable for larger models, so we are stuck with inflated basic vocabularies. It is also probably possible now to go even for larger vocabularies, in the 1-2 million range (by factorising the embedding matrix, for example), but this does not lead to noticeable improvements in performance, AFAIK.
- Der_Einzige 2y ago
  Performance would be massively improved on constrained text tasks. That alone makes it worth it to expand the vocabulary size.
spott 2y ago
BPE is complete. Every valid Unicode string can be encoded with any BPE tokenizer. BPE basically starts with a token for every valid value for a Unicode byte and then creates new tokens by looking at common pairs of bytes (‘t’ followed by ‘h’ becomes a new token ’th’)

HanClinto 2y ago

I really appreciated this blog post, and in particular I appreciated the segment talking about typos. We were discussing this earlier this week -- I'm helping with a RAG-like application for a project right now, and we're concerned with how much small typos or formatting differences in users' queries can throw off our embedding distances. One thought was: Should we be augmenting our training data (or at the very least, our pretraining data) with intentional typos / substitutions / capitalizations, just to help it learn that "wrk" and "work" are probably synonyms? I looked briefly around for typo augmentation for (pre)training, and didn't see anything at first blush, so I'm guessing that if this is a common practice, that it's called something else.

tmikaeld 2y ago
I work with full text search where this is common. Here is some points. Stemming: Reducing words to their base or root form (e.g., “working,” “worked” becoming “work”). Lemmatization: Similar to stemming, but more sophisticated, accounting for context (e.g., “better” lemmatizes to “good”). Token normalization: Standardizing tokens, such as converting “wrk” to “work” through predefined rules (case folding, character replacement). Fuzzy matching: Allowing approximate matches based on edit distance (e.g., “wrk” matches “work” due to minimal character difference). Phonetic matching: Matching words that sound similar, sometimes used to match abbreviations or common misspellings. Thesaurus-based search: Using a predefined list of synonyms or alternative spellings to expand search queries. Most of these are open and free lists you can use, check the sources on manticore search for example.
- soared 2y ago
  Porter stemming is currently widely used in adtech for keywords.
- thaumasiotes 2y ago
  > Lemmatization: Similar to stemming, but more sophisticated, accounting for context (e.g., “better” lemmatizes to “good”). I don't understand. How is that different from stemming? What's the base form of "better" if not "good"? The nature of the relationship between "better" and "good" is no different from that between "work" and "worked".

woolr 2y ago

Can't repro some of the numbers in this blog post, for example: from sentence_transformers import SentenceTransformer from sentence_transformers import util model = SentenceTransformer('all-MiniLM-L6-v2') data_to_check = [ "I have recieved wrong package", "I hve recieved wrong package" ] embeddings = model.encode(data_to_check) util.cos_sim(embeddings, embeddings) Outputs: tensor([[1.0000, 0.9749], [0.9749, 1.0000]])

1986 2y ago
Your data differs from theirs - they have "I have received wrong package" vs "I hve received wrong pckage", you misspelled "received" in both and didn't omit an "a" from "package" in the "bad" data

andix 2y ago

This is an awesome article, but I’m missing the part where solutions for each of the problems were discussed. Run a spell check before tokenizing? Maybe even tokenize the misspelled word and the potential corrected word next to each other like „misspld (misspelled)“? For the issue with the brand names the tokenizer doesn’t know, I have no idea how to handle it. This problem is probably even worse in less common languages, or in languages which use a lot of compound words.

r_hanz 2y ago

Very nicely written article. Personally, I find RAG (and more abstractly, vector search) the only mildly interesting development in the latest LLM fad, and have always felt that LLMs sit way too far down the diminishing returns curve to be interesting. However, I can’t believe tokenization and embeddings in general, are not broadly considered the absolutely most paramount aspect of all deep learning. The latent space your model captures is the most important aspect of the whole pipeline, or else what is any deep learning model even doing?

cranium 2y ago

I finally understood the weirdness of tokenizers after watching the video Andrej Karpathy made: "Let's build the GPT Tokenizer" (https://www.youtube.com/watch?v=zduSFxRajkE https://www.youtube.com/watch?v=zduSFxRajkE). He goes through why we need them instead of raw byte sequences (too expensive) and how the Byte Pair Encoding algorithm works. Worth spending 2h for the deeper understanding if you deal with LLMs.

maytc 2y ago

The difference in the dates example seems right to me 20 October 2024 and 2024-20-10 are not the same. Months in different locales can be written as yyyy-MM-dd. It can also be a catalog/reference number. So, it seems right that their embedding similarity is not perfectly aligned. So, it's not a tokenizer problem. The text meant different things according to the LLM.

gavin_gee 2y ago

do pictograms represent a way to reduce tokens?