3 ms·

And also the LLM version, and LLMZip https://bellard.org/ts_zip/ https://bellard.org/ts_zip/ https://arxiv.org/abs/2306.04050 https://arxiv.org/abs/2306.04050

speedgoose 2mo ago
I tried to reproduce those results, at least in terms of compression ratios, not speed. However I would say that testing on alice29, enwiki8, text8 data is kinda cheating. Alice in Wonderland and Wikipedia are very likely part of the training data of the LLM models used there. So I tried on HN comments from a few days ago, extracted from the text column of the public HN bigquery dataset. Using RWKV v7 0.1B instead of RWKV v4, I get 0.962 bits per byte on alice29, and 1.156 bits per bytes on the HN comments. Still a lot better than 2.826 bits per bytes of xz level 9.
- adamgordonbell 2mo ago
  Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce. Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]? I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's numbers using a human prediction for compression, from what i can see. https://github.com/hkust-nlp/llm-compression-intelligence https://github.com/hkust-nlp/llm-compression-intelligence https://gwern.net/hutter-prize https://gwern.net/hutter-prize
  - krackers 2mo ago
    >I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric This is the premise of https://huggingface.co/spaces/Jellyfish042/UncheatableEval https://huggingface.co/spaces/Jellyfish042/UncheatableEval
    - adamgordonbell 1mo ago
      very cool
    - andai 1mo ago
      Nice. This ranking basically matches other benchmarks, from what I can tell. Which implies this would probably also hold for the larger models, which are sadly not included in the leaderboard.
      - krackers 1mo ago
        I think it's limited to using base models (non post-trained), because the post-training would skew the logit distribution. There are ways to "coax" post-trained models back into behaving "like" a base model, I wonder if the benchmark could be unofficially updated with those somehow.

speedgoose 2mo ago

I tried to reproduce those results, at least in terms of compression ratios, not speed. However I would say that testing on alice29, enwiki8, text8 data is kinda cheating. Alice in Wonderland and Wikipedia are very likely part of the training data of the LLM models used there. So I tried on HN comments from a few days ago, extracted from the text column of the public HN bigquery dataset. Using RWKV v7 0.1B instead of RWKV v4, I get 0.962 bits per byte on alice29, and 1.156 bits per bytes on the HN comments. Still a lot better than 2.826 bits per bytes of xz level 9.

adamgordonbell 2mo ago
Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce. Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]? I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's numbers using a human prediction for compression, from what i can see. https://github.com/hkust-nlp/llm-compression-intelligence https://github.com/hkust-nlp/llm-compression-intelligence https://gwern.net/hutter-prize https://gwern.net/hutter-prize
- krackers 2mo ago
  >I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric This is the premise of https://huggingface.co/spaces/Jellyfish042/UncheatableEval https://huggingface.co/spaces/Jellyfish042/UncheatableEval
  - adamgordonbell 1mo ago
    very cool
  - andai 1mo ago
    Nice. This ranking basically matches other benchmarks, from what I can tell. Which implies this would probably also hold for the larger models, which are sadly not included in the leaderboard.
    - krackers 1mo ago
      I think it's limited to using base models (non post-trained), because the post-training would skew the logit distribution. There are ways to "coax" post-trained models back into behaving "like" a base model, I wonder if the benchmark could be unofficially updated with those somehow.