5 ms·
Using subwords makes your sequences shorter, which makes them cost less. Besides that, for alphabetic languages, there exists almost no relation between form a
by stephantul 2y ago
Using subwords makes your sequences shorter, which makes them cost less.
Besides that, for alphabetic languages, there exists almost no relation between form and meaning. I.e.: “ring” and “wing” differ by one letter but have no real common meaning. By picking the character or byte as your choice of representation, the model basically has to learn to distinguish ring and wing in context. This is a lot of work!
So, while working on the character or byte level saves you some embeddings and thus makes your model smaller, it puts all of the work of distinguishing similar sequences with divergent meanings on the model itself, which means you need a larger model.
By having subwords, a part of this distinguishing work already has been done by the vocabulary itself. As the article points out, this sometimes fails.
- bunderbunder 2y agoI suspect that the holy grail here is figuring out how to break the input into a sequence of morphemes and non-morpheme lexical units.
- thaumasiotes 2y agoWhat do you mean by non-morpheme lexical units? Syntactic particles, units too small to be morphemes? Lexical items that contain multiple morphemes? In either case, isn't this something we already do well?
- bunderbunder 2y agoPunctuation, for example. And no, at least for the languages with which I'm familiar SOTA tokenizers tend to only capture the easy cases. For example, the GPT-4 tokenizer breaks the first sentence of your post like so: What/ do/ you/ mean/ by/ non/-m/orp/heme/ lexical/ units/? Notice how "morpheme" gets broken into three tokens, and none of them matches "morpheme"'s two morphemes. "Lexical" and "units" are each a single token, when they have three and two morphemes respectively. Or in French, the word "cafetière" gets chopped willy-nilly into "c/afet/ière". The canonical breakdown is "cafe/t/ière".
- p1esk 2y agoHas anyone tried to combine a token embedding with some representation of the characters in the (sub)word? For example, use a 512 long vector to represent a token, and reserve the last 12 values to spell out the word.
- RicoElectrico 2y agoWell, fastText uses character n-grams to compute embeddings for out-of-vocabulary words. This is pre-transformers work BTW.
- p1esk 2y agoIIRC, overlapping ngram vectors are summed to form the token embedding - doesn’t it effectively destroy any character level representation of the token? Doesn’t really make sense to me.
- stephantul 2y agoIt works because they use really large ngram values, up to 6. So most character-level information is in these subwords.
- p1esk 2y agoLet’s say we want to use 6-grams and build an embedding vector for the word “because”: we add integer vectors for “becaus” and “ecause”, right? For example: [1,2,3,4,5,6] + [2,3,4,5,6,2] = [3,5,7,9,11,8]. Obviously we cannot use this resulting numerical vector to spell the input word. Pretty much all character level info is lost.
- mattnewton 2y agoI'm not following - spell out the word how? Like put the actual bytes as numerical input to the transformer layer?
- p1esk 2y ago
- sundarurfriend 2y ago> Besides that, for alphabetic languages, there exists almost no relation between form and meaning. Also true for Abugida-based languages, for eg. சரம் (saram = string) vs மரம் (maram = tree), and many more. I think your intention with specifying "alphabetic languages" was to say "non-logographic languages", right?
- bunderbunder 2y agoI'll do you one more and say "non-Chinese languages". Written Japanese - including the kanji portion of the script - has the same characteristic. And even in Chinese it's a fairly weak relationship. A large portion of the meanings of individual characters come from sound loan. For example the 英 in 英雄 means "hero", in 英语 means "England", an in 精英 means "flower". The relationship there is simple homophony. On the other hand, one thing you do get with written Chinese is that "1 character = 1 morpheme" very nearly works. So mechanistically breaking a text into a sequence of morphemes can be done pretty reliably without the aid of a semantic model or exhaustive hard-coded mapping. I think that for many other languages you can't even get close using only syntactic analysis.
- thaumasiotes 2y ago> I'll do you one more and say "non-Chinese languages". Written Japanese - including the kanji portion of the script - has the same characteristic. Written Japanese is much more ideographic than written Chinese. Japanese spelling is determined, such as it is, by semantics. Chinese spelling is determined by sound. Thus, 女的, 娘们, and 妮子, all meaning 'girl' or 'woman', have no spelling in common because they are different words, while Japanese uses 女 for "jo" and "onna" despite a total lack of any relationship between those words.
- deleted 2y ago[deleted]
- stephantul 2y agoI was trying to say “at least for alphabetic languages”. I don’t like to say things about languages I can’t speak or write. So, no, it wasn’t my intention to say “non-logographic languages”