2 ms·
Tiktoken uses the byte pair encoding, which is a form of compression. (LZ78 can be thought as an automatic form of this scheme.) So it's like compressing JPEG w
by lifthrasiir 3y ago
Tiktoken uses the byte pair encoding, which is a form of compression. (LZ78 can be thought as an automatic form of this scheme.) So it's like compressing JPEG with ZIP, which may or may not work, but in either case is not a fair game.