3 ms·

Then English has a vocabulary of 27 words: A -> Z + space

• nerdponx 3y ago
  If you tokenize using character n-grams, this isn't totally wrong.
  • Izkata 3y ago
    Exactly. There was a post on here a few months ago about training using single characters as tokens instead of words, and it worked really well, being able to create new Shakespeare-like text despite not using human words as tokens. What a (human) word is can be learned by the model instead of encoded in the training set.
• sterlind 3y ago
  I think a fairer comparison would be Toki Pona, a micro-language with ~120 words. you can express lots of things if you have a great deal of patience, Up-Goer Five style. In logic there's also SKI combinator calculus, which is Turing-complete with three symbols, and unlike the Morse or A-Z examples, but like the Toki Pona example, each symbol has a semantic meaning. If you just want to describe ideas in an abstract realm like sequences of colors, like this paper, it's not surprising you don't need many words.