5 ms·

> The Qualcomm AI researchers found that 97%+ of outlier activations in LLMs occur in whitespace and punctuation positions. This is striking. If true, why not

by bambax 3y ago

> The Qualcomm AI researchers found that 97%+ of outlier activations in LLMs occur in whitespace and punctuation positions. This is striking. If true, why not try to ignore whitespace and puctuation? In old Latin, scripto continua [1] was a way to write continuously, for the exact same reason: to save space. Other modern languages still do that, and are no less parseable. Granted, it's unlikely a commercial LLM would become popular if it produced output without spaces or punctuation; but an open source one that promised to be much more compressible, and therefore work on smaller machines, might be super useful. It's not hard for a human to add spaces afterwards. It used to be a job for beginning journalists at the time of telex machines: press releases were sent in all caps without spaces, and interns were tasked with adding slashes between words. In French it was called "bâtonner les dépêches" (literally: add sticks to press releases -- not sure about the idiomatic English translation). [1] https://simple.wikipedia.org/wiki/Scriptio_continua https://simple.wikipedia.org/wiki/Scriptio_continua

colordrops 3y ago
Seems you could make a pipeline where a much simpler model adds spaces and punctuation to output from the main model.
- samwillis 3y ago
  I suspect punctuation adds significant meaning to the models, that could be why so much computation is applied to it. That's not to say a pipeline couldn't be effective.
- bambax 3y ago
  Yes I was thinking about that, it should be quite easy afterwards.
- GTP 3y ago
  For the spaces and for some (maybe most?) languages you don't even need a NN to add spaces: as words made of two or more words aren't that common, and when those occur you probably want to use the composite one, it boils down to start from the beginning of the text and look in a dictionary what's the longest string that is a valid word. The only language that I know of that uses a lot of composite words (I mean words made by sticking two or more words togheter) is German, but I think that looking for the longest sequence occurring in a dictionary would be correct most of the times.
  - thfuran 3y ago
    I think you're significantly underestimating how many words could be retokenized into multiple words even before considering how concatenation affects things. For example: Concatenate is a word, but so are con, catenate, cat, and enate. Yes, no two of those are likely to be used in sequence, but I don't think that's a very reliable rule overall—"a" and "an" are both common words and negative prefixes.
    - GTP 3y ago
      Maybe you're right. I was biased by my native language, which doesn't have the a/an problem that English has.
espe 3y ago
those paratextual phenomena probably are important for the model's representations.. not to get rid of and not easily compressable either. have a look at predicitive features for authorship attribution in stylometry for example. whitespace and punctuation are always decisive.
TeMPOraL 3y ago
> This is striking. If true, why not try to ignore whitespace and puctuation? It is initially, but thinking about it some more, there's a lot of information packed in whitespace and punctuation choice. Scripto continua may have worked because the few readers who lived back then expected it to encode some form of legal or religious prose, but even then they could learn things from the overall shape of the document. LLMs are working in a much richer domain of document types, but the only thing they can "see" is a stream of tokens. There's no spatial or geometric data attached there. So whitespace and punctuation are the only thing an LLM has to make inferences about otherwise textually identical inputs. Such as: (see: other) -- vs -- {see: other} One being likely a text fragment, the other likely a piece of code. Or how spacing may imply Markdown or YAML being used. Or how it may imply a list. Or a poem. Or a song. Or specific writing style, such as "lol im a casual who not care bout comms" vs. "I am a distinguished professor, about to retire. Elites like us put two spaces after full stop."
- bambax 3y ago
  > the few readers who lived back then expected it to encode some form of legal or religious prose The Latin literature was extremely rich, from Cicero to Tacitus, and was certainly not limited to legal information. Here's part of your comment with white space and punctuation stripped: scriptocontinuamayhaveworkedbecausethefewreaderswholivedbackthenexpectedittoencodesomeformoflegalorreligiousprosebuteventhentheycouldlearnthingsfromtheoverallshapeofthedocumentllmsareworkinginamuchricherdomainofdocumenttypesbuttheonlythingtheycanseeisastreamoftokenstheresnospatialorgeometricdataattachedtheresowhitespaceandpunctuationaretheonlythinganllmhastomakeinferencesaboutotherwisetextuallyidenticalinputs It's a little hard to read, but not that hard. I think one would get used to it. Also, for creative use of LLM, it may be a feature, as trying to find the words could be inspiring. I think it would be worth a try.
  - hgsgm 3y ago
    Now do a modern structured document with sections and bullet points and logical connectives.
    - 3y ago
dxuh 3y ago
LLMs are also pretty good at programming and I would expect this to nuke programming ability completely, wouldn't it?
numeri 3y ago
In the Qualcomm paper cited, they explain/hypothesize that Transformers learn to attend to these low-meaning tokens when they want to avoid adding too much extra info to the residual stream. So it's not an issue that the models attend to spaces and punctuation during in these outliers – it's the workaround the models come up with to get around the fact that attention has to go somewhere. This post's author has a different solution, and one that theoretically could avoid causing large outliers that prevent efficient quantization. These large outliers seem to be an unfortunate side-effect of the models' learned solution. So getting rid of spaces would do nothing to solve the problem, and would instead force the models to learn a new solution, one that presumably isn't as optimal.

by bambax 3y ago

colordrops 3y ago

Seems you could make a pipeline where a much simpler model adds spaces and punctuation to output from the main model.

samwillis 3y ago
I suspect punctuation adds significant meaning to the models, that could be why so much computation is applied to it. That's not to say a pipeline couldn't be effective.
bambax 3y ago
Yes I was thinking about that, it should be quite easy afterwards.
GTP 3y ago
For the spaces and for some (maybe most?) languages you don't even need a NN to add spaces: as words made of two or more words aren't that common, and when those occur you probably want to use the composite one, it boils down to start from the beginning of the text and look in a dictionary what's the longest string that is a valid word. The only language that I know of that uses a lot of composite words (I mean words made by sticking two or more words togheter) is German, but I think that looking for the longest sequence occurring in a dictionary would be correct most of the times.
- thfuran 3y ago
  I think you're significantly underestimating how many words could be retokenized into multiple words even before considering how concatenation affects things. For example: Concatenate is a word, but so are con, catenate, cat, and enate. Yes, no two of those are likely to be used in sequence, but I don't think that's a very reliable rule overall—"a" and "an" are both common words and negative prefixes.
  - GTP 3y ago
    Maybe you're right. I was biased by my native language, which doesn't have the a/an problem that English has.

espe 3y ago

those paratextual phenomena probably are important for the model's representations.. not to get rid of and not easily compressable either. have a look at predicitive features for authorship attribution in stylometry for example. whitespace and punctuation are always decisive.

TeMPOraL 3y ago

> This is striking. If true, why not try to ignore whitespace and puctuation? It is initially, but thinking about it some more, there's a lot of information packed in whitespace and punctuation choice. Scripto continua may have worked because the few readers who lived back then expected it to encode some form of legal or religious prose, but even then they could learn things from the overall shape of the document. LLMs are working in a much richer domain of document types, but the only thing they can "see" is a stream of tokens. There's no spatial or geometric data attached there. So whitespace and punctuation are the only thing an LLM has to make inferences about otherwise textually identical inputs. Such as: (see: other) -- vs -- {see: other} One being likely a text fragment, the other likely a piece of code. Or how spacing may imply Markdown or YAML being used. Or how it may imply a list. Or a poem. Or a song. Or specific writing style, such as "lol im a casual who not care bout comms" vs. "I am a distinguished professor, about to retire. Elites like us put two spaces after full stop."

bambax 3y ago
> the few readers who lived back then expected it to encode some form of legal or religious prose The Latin literature was extremely rich, from Cicero to Tacitus, and was certainly not limited to legal information. Here's part of your comment with white space and punctuation stripped: scriptocontinuamayhaveworkedbecausethefewreaderswholivedbackthenexpectedittoencodesomeformoflegalorreligiousprosebuteventhentheycouldlearnthingsfromtheoverallshapeofthedocumentllmsareworkinginamuchricherdomainofdocumenttypesbuttheonlythingtheycanseeisastreamoftokenstheresnospatialorgeometricdataattachedtheresowhitespaceandpunctuationaretheonlythinganllmhastomakeinferencesaboutotherwisetextuallyidenticalinputs It's a little hard to read, but not that hard. I think one would get used to it. Also, for creative use of LLM, it may be a feature, as trying to find the words could be inspiring. I think it would be worth a try.
- hgsgm 3y ago
  Now do a modern structured document with sections and bullet points and logical connectives.
  - 3y ago

dxuh 3y ago

LLMs are also pretty good at programming and I would expect this to nuke programming ability completely, wouldn't it?

numeri 3y ago

In the Qualcomm paper cited, they explain/hypothesize that Transformers learn to attend to these low-meaning tokens when they want to avoid adding too much extra info to the residual stream. So it's not an issue that the models attend to spaces and punctuation during in these outliers – it's the workaround the models come up with to get around the fact that attention has to go somewhere. This post's author has a different solution, and one that theoretically could avoid causing large outliers that prevent efficient quantization. These large outliers seem to be an unfortunate side-effect of the models' learned solution. So getting rid of spaces would do nothing to solve the problem, and would instead force the models to learn a new solution, one that presumably isn't as optimal.