3 ms·
> If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values
by weird-eye-issue 3y ago
> If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values
Reminds me of tokenization for LLMs. 48 dashes ("------------------------------------------------") is only a single token for GPT-3.5 / GPT-4 (they use the cl100k_base encoding). I suppose since that is used in Markdown. Also "professional illustration" is only two tokens despite being a long string. Whereas if you convert that to a language like Thai it is 17 tokens which sucks in some cases but I suppose tradeoffs had to be made