6 ms·
No, LLMs only do this for language. They don't try to do this for arbitrary data.
by eru 5mo ago
No, LLMs only do this for language. They don't try to do this for arbitrary data.
- woadwarrior01 5mo agoThere are many approaches around this, the simplest being to treat bytes as tokens (cf: Google's ByT5[1]). Also, BLT[2] from Meta and ByteFormer[3] from Apple. [1]: https://arxiv.org/abs/2105.13626 https://arxiv.org/abs/2105.13626 [2]: https://arxiv.org/abs/2412.09871 https://arxiv.org/abs/2412.09871 [3]: https://arxiv.org/abs/2306.00238 https://arxiv.org/abs/2306.00238
- energy123 5mo agoTransformers do this for any stream of tokens, those tokens can map to anything you want, and you will get lossy compression. Text produced by humans just happens to be dense, available, and a useful prior, but it is not intrinsically required. See 3D vision transformers for example.
- jeremyjh 5mo agoIt is not possible to compress arbitrary data. If the data is already compressed, or it is encrypted, or it is randomly generated, it cannot be compressed with any method. This is foundational information theory. https://en.wikipedia.org/wiki/Lossless_compression#Limitations https://en.wikipedia.org/wiki/Lossless_compression#Limitatio...
- eru 5mo agoEh, you can't compress truly arbitrary data, sure. But you can compress almost any natural occurring data: reality is very sparse. For some reason.
- thadt 5mo agoWhereas if we're talking about lossy compression (as is the person to whom you replied) we certainly can compress arbitrary data - almost as much as we want. The hard question, then, is how much the decompressed output looks like the original.