3 ms·

Not only that, it’s provable that this approach doesn’t scale. That is to say, specifically, that it is not possible, by any means, to take any block of text a

Not only that, it’s provable that this approach doesn’t scale. That is to say, specifically, that it is not possible, by any means, to take any block of text and reduce it to a smaller block of text without losing some information. That is infinite compression; if you could do this, you could reduce any dataset to 1 bit and recover the data seamlessly. You can’t. What that means is that this approach is fundamentally just improving performance, it’s not solving the problem. When you compress a conversation into a summary, some information will be lost. You can tweak and fold and do whatever clever things you want, but fundamentally you are losing information. …and the process is recursive. So at some point you will be summarising a set of summaries… and you will lose some information you’ve summarised, to some extent. You can’t not lose information with this process. So… while, it probably helps in trivial cases, putting recursive summaries into your prompt is kind of daft, and almost certainly doesn’t actually work when you try use it to do useful things. It probably looks like it’s working when you don’t use the recursive summaries heavily, because at that point you’re not losing much information. …but, I’m going to bet that it doesn’t scale, and people using it will find that out pretty quickly as they use it.

anon012012 3y ago
Technically you can reduce any amount of bits to 1 bit if the compressor/decompressor agree about what informations that bit means. So there would be the full data on client-side/server-side, and you can "communicate" the finale desired data with solely 1 bit. (it's like the blockchain algorithm) This sounds trite/heretic but it is as fundamental in its nature as the concept of "incompressible data". I agree that you cannot spare any data in the absolute, but I argue that you can communicate information with less bits than the data holds if the peers and the "model" incorporates knowledge about what kind of data is being transmitted.
azeirah 3y ago
I don't disagree with your proof but I would like to add that your example doesn't mean it's not practical to recursively summarise a couple times. One bit of information can mean more the more parameters an LLM has. The text "a8ckd" could easily mean the entire Declaration of Independence or whatever to an LLM (it doesn't, of course). It has memorised a lot of stuff and similar to us "bubble sort" means // An optimized version of Bubble Sort void bubbleSort(int arr[], int n) { int i, j; bool swapped; for (i = 0; i < n - 1; i++) { swapped = false; for (j = 0; j < n - i - 1; j++) { if (arr[j] > arr[j + 1]) { swap(&arr[j], &arr[j + 1]); swapped = true; } } // If no two elements were swapped by inner loop, // then break if (swapped == false) break; } } That's 26x as much text (counting comments and whitespace). Without comments and removing most indentation it's about 20x as much. The more information stored in its parameters, the more information a single bit can hold. All we need to do is find out how many times can you recur until it breaks down? 3 times? 5 times? Just a few times is still genuinely very useful!

anon012012 3y ago

Technically you can reduce any amount of bits to 1 bit if the compressor/decompressor agree about what informations that bit means. So there would be the full data on client-side/server-side, and you can "communicate" the finale desired data with solely 1 bit. (it's like the blockchain algorithm) This sounds trite/heretic but it is as fundamental in its nature as the concept of "incompressible data". I agree that you cannot spare any data in the absolute, but I argue that you can communicate information with less bits than the data holds if the peers and the "model" incorporates knowledge about what kind of data is being transmitted.

azeirah 3y ago

I don't disagree with your proof but I would like to add that your example doesn't mean it's not practical to recursively summarise a couple times. One bit of information can mean more the more parameters an LLM has. The text "a8ckd" could easily mean the entire Declaration of Independence or whatever to an LLM (it doesn't, of course). It has memorised a lot of stuff and similar to us "bubble sort" means // An optimized version of Bubble Sort void bubbleSort(int arr[], int n) { int i, j; bool swapped; for (i = 0; i < n - 1; i++) { swapped = false; for (j = 0; j < n - i - 1; j++) { if (arr[j] > arr[j + 1]) { swap(&arr[j], &arr[j + 1]); swapped = true; } } // If no two elements were swapped by inner loop, // then break if (swapped == false) break; } } That's 26x as much text (counting comments and whitespace). Without comments and removing most indentation it's about 20x as much. The more information stored in its parameters, the more information a single bit can hold. All we need to do is find out how many times can you recur until it breaks down? 3 times? 5 times? Just a few times is still genuinely very useful!