4 ms·

I'm not saying the result is completely useless, I am comparing it to the age-old technique of using a dictionary. Does this new LLM-powered technique improve u

by remram 2y ago

I'm not saying the result is completely useless, I am comparing it to the age-old technique of using a dictionary. Does this new LLM-powered technique improve upon the old dictionary technique? Dictionaries also don't require a GPU or this amount of RAM. Where I assume LLMs would shine is lossy compression.

binary132 2y ago
Ah ok, I think we made different assumptions about whether the model was specific to the particular dataset so each one would need a new model — a dictionary is specific to the particular dataset being compressed, right? I was thinking the LLM would be a general-purpose text compression model.
- remram 2y ago
  Not particularly. You could make a dictionary from "the English web", with common character sequences found on those sites you use as input.
ksec 2y ago
I have the same question, what is the different between LLM and Dictionary in the context of compression. Can I not "train" a dictionary?
- binary132 2y ago
  AIUI, a dictionary is built during compression to specify the heuristics of a particular dataset and belongs to that specific dataset only. For example, it could be a ranking of the most frequent 10 symbols in the compressed file. That will be different for every input file.
  - mbreese 2y ago
    > That will be different for every input file That could be different for every input file, but it doesn't have to be. It could also be a fixed dictionary. For example, ZLIB allows for a user-defined dictionary [1]. In this case, I'd consider the LLM to be a fixed dictionary of sorts. A very large, fixed dictionary with probabilistic return values. [1] https://www.rfc-editor.org/rfc/rfc1950#page-9 https://www.rfc-editor.org/rfc/rfc1950#page-9
    - binary132 2y ago
      Ah, I see. I’d never thought of the possibility of using a dictionary not created specifically from the given input dataset, heh
      - mbreese 2y ago
        Admittedly, I don’t think it is common, but I think there was a project a few years ago (Google?) that tried to compress HTML using at least a partially fixed dictionary. Nowadays though, it’s apparently still something that’s being tried. Chrome now supports shared dictionaries for Zstd and Brotli. One idea being, you would likely benefit from having a shared dictionary used to decompress multiple artifacts for a site. But, you many not want everything compressed all together, so this way you get the compression benefit, but can have those artifacts split into different files. https://developer.chrome.com/blog/shared-dictionary-compression https://developer.chrome.com/blog/shared-dictionary-compress...

by remram 2y ago

binary132 2y ago

Ah ok, I think we made different assumptions about whether the model was specific to the particular dataset so each one would need a new model — a dictionary is specific to the particular dataset being compressed, right? I was thinking the LLM would be a general-purpose text compression model.

remram 2y ago
Not particularly. You could make a dictionary from "the English web", with common character sequences found on those sites you use as input.

ksec 2y ago

I have the same question, what is the different between LLM and Dictionary in the context of compression. Can I not "train" a dictionary?

binary132 2y ago
AIUI, a dictionary is built during compression to specify the heuristics of a particular dataset and belongs to that specific dataset only. For example, it could be a ranking of the most frequent 10 symbols in the compressed file. That will be different for every input file.
- mbreese 2y ago
  > That will be different for every input file That could be different for every input file, but it doesn't have to be. It could also be a fixed dictionary. For example, ZLIB allows for a user-defined dictionary [1]. In this case, I'd consider the LLM to be a fixed dictionary of sorts. A very large, fixed dictionary with probabilistic return values. [1] https://www.rfc-editor.org/rfc/rfc1950#page-9 https://www.rfc-editor.org/rfc/rfc1950#page-9
  - binary132 2y ago
    Ah, I see. I’d never thought of the possibility of using a dictionary not created specifically from the given input dataset, heh
    - mbreese 2y ago
      Admittedly, I don’t think it is common, but I think there was a project a few years ago (Google?) that tried to compress HTML using at least a partially fixed dictionary. Nowadays though, it’s apparently still something that’s being tried. Chrome now supports shared dictionaries for Zstd and Brotli. One idea being, you would likely benefit from having a shared dictionary used to decompress multiple artifacts for a site. But, you many not want everything compressed all together, so this way you get the compression benefit, but can have those artifacts split into different files. https://developer.chrome.com/blog/shared-dictionary-compression https://developer.chrome.com/blog/shared-dictionary-compress...