3 ms·
You touch on a very interesting and general concept, and that is the correspondence between lossless compression algorithms and probability distributions, and t
by partition 16y ago
You touch on a very interesting and general concept, and that is the correspondence between lossless compression algorithms and probability distributions, and the subsequent correspondence between probability distributions and procedural generation algorithms.
Consider the first correspondence. Suppose you knew a certain class of strings arose from a probability distribution. Then you can talk about the strings that are most likely to occur, or, more useful in general, about the individual characters that are most likely to occur given a history of previous characters. Using this, you can make a compression algorithm that will map (symbol, probability) or (symbol + history, conditional probability) starting from the highest probability, to the numbers 0, 1, 2 ...
This is arithmetic coding.
http://en.wikipedia.org/wiki/Arithmetic_coding http://en.wikipedia.org/wiki/Arithmetic_coding
The result is optimal in terms of expected compressed string length assuming the strings you use this on really do come from that distribution.
Now consider the other way around. Suppose you have a lossless compression algorithm. Find the (uncompressed symbol, compressed symbol) pair (possibly with context) that achieves the highest compression. Assign that a high probability. Then find the next pair. Assign that a slightly lower probability (This can be done in a more principled manner than I'm alluding here). Then you have yourself a probability distribution.
More on data compression theory:
http://en.wikipedia.org/wiki/Data_compression http://en.wikipedia.org/wiki/Data_compression
But what happens once you have a probability distribution over some data type, is that you may cast the problem of automatically generating instance of that data type as sampling from the distribution. Many procedural generation algorithms that give nondeterministic results (and the useful ones do, otherwise the work the modeler has to do is fundamentally the same) can be re-cast as sampling from a probability distribution; look at what the algorithm is generating and learn the distribution.
Note that this is uncomputable in general for the same reason Kolmogorov complexity is. This is known as Solomonoff induction:
http://singinst.org/blog/2007/06/25/solomonoff-induction/ http://singinst.org/blog/2007/06/25/solomonoff-induction/
So to answer your original questions:
1. The information (as in information entropy) in the algorithm in the CD is the entropy of the true probability distribution from which your levels originate.
2. And yes, in principle you can use the correspondence between data compression and procedural generation to generate instances of any arbitrary data type, not just 3D game levels. It may be hard to design a probability distribution that will create well-formatted instances though :)
- johnwatson11218 16y agoThank you for this thoughtful and detailed reply. I have read about Arithmetic Coding and Data Compression. I haven't read about Solomonoff-Induction but I will definitely read up on it. I think another way of thinking about my original comment would be to imagine that you had an image of the Mandelbrot Set up on your monitor (assume you have drilled down a few times). If you knew the parameters that had generated that view you could write a very concise description suitable for transmission over a network. If you have forgotten the params and had to make due with a screen print saved to jpeg it would appear to contain more 'information'. Not only would the algorithmic version be shorter (so long as you hadn't drilled down to the point at which the numbers representing the view port had millions of degrees of precision) but you would be able to keep zooming in to view arbitrary detail. That is something you wouldn't be able to do with the jpeg. I guess what I'm driving at is that I don't understand how Information Theory is able to account for things like fractals without saying that either the Mandelbrot Set is random or that it contains infinite information. Am I looking at this problem correctly?
- jacques_chester 16y ago> I guess what I'm driving at is that I don't understand how Information Theory is able to account for things like fractals without saying that either the Mandelbrot Set is random or that it contains infinite information. Am I looking at this problem correctly? This is (I think) the "coastline paradox": coast lines have different lengths depending on scale. Length grows with resolution, so in a sense the length of a coastline can be infinite. I think the escape hatch is that the information conveyed is not the points on the screen, but the rule on how to generate those points. Going back to my other post this means that the city builder conveys no information about a particular city map.
- partition 16y ago> I think the escape hatch is that the information conveyed is not the points on the screen, but the rule on how to generate those points. Going back to my other post this means that the city builder conveys no information about a particular city map. Yep. The amount of information conveyed can be considered 'equivalent' to the length of the shortest algorithm used to get from one string to the other. The field that is concerned about thinking about this problem in a disciplined manner is algorithmic information theory: http://en.wikipedia.org/wiki/Algorithmic_information_theory http://en.wikipedia.org/wiki/Algorithmic_information_theory In particular, they show that the quantity is incomputable (the Komolgorov complexity).