I’d be interested in hearing more about how to “abuse” compression algorithms to detect outliers in datasets or to filter out repetitions. For example, LLMs or

I’d be interested in hearing more about how to “abuse” compression algorithms to detect outliers in datasets or to filter out repetitions. For example, LLMs or Whisper sometimes get stuck in a loop, repeating a word, a group of words, or whole sentences multiple times. I imagine a compression algorithm would be an interesting tool to filter out such noise. And there’s maybe even more shenanigans you can do with them besides actual compression…