2 ms·
There does seem to be the trend of consciousness of a growing link between AI language models and data compression (AKA information compression)... But what we
by peter_d_sherman 3y ago
There does seem to be the trend of consciousness of a growing link between AI language models and data compression (AKA information compression)...
But what we're all looking for (ideally) is a hard mathematical proof (and/or algorithm) relating AI language models to data compression...
I'm guessing that like so much else in the history of Math and Computer Science -- that this will gradually be discovered by the world's Mathematicians and Computer Scientists in phases, in degrees...
I'm guessing that this will start via a single mathematician postulating upper and/or lower limits relating one specific compression algorithm with one specific AI language model of a specific size.
From there, others should be able to generate better/more exact formulas for upper/lower bounds, and from there we might start to see some inexact, yet steps-in-the-right direction formulas which will eventually lead to the exact relationship...
I'm guessing, but not knowing, that the way to get started on all of this, should someone choose, would be as follows:
A) Take the simplest known compression algorithm;
B) Take the simplest known AI language model;
and
C) Try to determine mathematical relationships between them (upper and lower bounds, in all likelihood, initially), no matter how inexact...
D) Publish a paper or write a web page explaining what you tried, and what results were obtained...
E) Wait for other researchers to try to improve, if even incrementally, upon those results...
There's a great history as to how mathematicians speculated over time about the exact formula governing the occurrence and distribution of prime numbers; that mystery has been solved -- but the history and speculation and incremental steps taken by various mathematicians over time are fascinating -- a part of math history, to be sure...