5 ms·
wouldn't it be better to use something like tiktoken's tokens to represent words and then use the RGB values as indexes into that encoding? you'd get higher com
by cpill 3y ago
wouldn't it be better to use something like tiktoken's tokens to represent words and then use the RGB values as indexes into that encoding? you'd get higher compression still
- edent 3y agoThat does sound interesting. Write it up and Let us know.
- lifthrasiir 3y agoTiktoken uses the byte pair encoding, which is a form of compression. (LZ78 can be thought as an automatic form of this scheme.) So it's like compressing JPEG with ZIP, which may or may not work, but in either case is not a fair game.