8 ms·

What is the theoretical limit of information density?

58 points by lpage 13y ago

vinceguidry 13y ago
This assumes that all information is stored physically. Let's say you get a telegram. The telegram itself can contain maybe a couple of paragraphs of text. You might think the bandwidth of the channel is at most a kilobyte or so. But there's plenty of other aspects that can raise the amount of information conveyed. Say you get one saying "Short Dow." Just going by the content, you'd have no idea what it was saying. Is there a guy named Dow somewhere that's short? But if you're a stockbroker, all of a sudden there's a whole lot more info there. When you allow for context, information density can approach infinity.
- itcmcgrath 13y ago
  No, let me paraphrase what you just said. 'If you can store infinite information outside of a system, then you can achieve infinite information density inside the system if you use outside the system as 'context'. I hope that paraphasing makes it clear enough the problem with this line of thinking. You have to include all the information when calculating absolute density. In your case, the context has to be stored somewhere too.
  - vinceguidry 13y ago
    If all information has to be accounted for and stored somewhere, and context is part of the information, then you can't store any information without storing all information, everywhere. Because every bit of information exists inside the context of the entire universe.
    - bradleyland 13y ago
      You're getting very metaphysical here, but reality remains the same even if you expand these principles to the universe. The rules of physics still apply.
      - gibwell 13y ago
        I think he has a point, although it's more about the semantics of the term 'information' density. Shannon information is always measured relative to a receiving context in which it the symbols are understood, and information content is related to the inverse of the probability of observing a particular signal as assessed by the receiver. So from that perspective vinceguidry has a reasonable point. However really the question being asked is something more like 'data density' and that is generally what people are talking about when the term 'information density' is invoked. Edit: I see that the original article does indeed refer to Data Density, and that HN title is just wrong.
    - smosher 13y ago
      Keep going, you're about to discover the holographic principle for yourself.
  - deleted 13y ago
    [deleted]
  - salmonellaeater 13y ago
    It's worse than that. There are a finite number of states of a message, and you can only reference that number of states of the outside system. If you think of the message as a pointer containing n bits, then you can only reference the first 2^n positions in the dictionary. If the outside system is a probability distribution over output messages (a more general case of the dictionary you described) then the problem is synonymous with compression.
- twinge 13y ago
  I think you're describing compression.
- drakaal 13y ago
  You are Dr. Who. You have a Tardis. It will convert your language in to any other language telepathically. Why not create a word that means the summation of everything you know, and say it to a person. They create a word that means the summation of everything they know and everything you know and say it to the next person...
  - logicallee 13y ago
    Even better - attribute a meaning, in that language, to refusing to say anything to the next person (remaining silent). Then you can put the summation of everything they know and everything you know into the case where you don't say anything (in that language). This is similar to making a version of gcc that outputs a Tetris program everytime it's asked to compile a 0-byte file, or maybe outputs all of Wikipedia. I mean, sure, you can hide Tetris, or the whole of human knowledge, in 0 bytes this way. But it's not a very useful exercise.
    - meta-coder 13y ago
      Remaining silent is as good as communicating one bit of information. Even a 0-byte file has metadata. Even if it is 0-byte long, GCC knows that it is reading an input file. How did it come to know? Because you communicated some amount of information by initiating the compilation.
- beagle3 13y ago
  Only chsnges things up to a constant, and not fundamentally. Read up on Kolmogorov Complexity if you are interested.
- nitrogen 13y ago
  The message didn't actually contain the extra information; your stockbroker supplied that him/herself.
- meta-coder 13y ago
  > When you allow for context, information density can approach infinity. I absolutely agree with you. You are my hero. The amount of information stored in an object (which is capable of storing at least one bit of information in traditional sense) depends on the size of the context. If the object is not even capable of storing one bit of information in traditional sense, then the amount of information that can be stored is zero. And for all objects that can store one bit or more of information in traditional sense, the total amount of information that can be stored in it = the number of bits it can store + the number of bits that can be stored in rest of the universe (context). So any one bit object can store the same amount of information that can be stored in the entire universe. And if it turns out that our universe is enclosed in yet another larger universe, then you have to include that as part of the context as well. Edit: typo
- sesqu 13y ago
  No. If you were a stockbroker, that message would contain less information - and you probably would know it already. It's useful information to a stockbroker, but it would be more informative to almost anyone else, who doesn't encounter such messages daily. As another example, the information in the Rosetta stone was diminished - if more accessible - to its carvers, and a 700 MB disk holds 700 MB regardless of what those bits are. What you're referring to is compression. High information density is indistinguishable from noise to any recipient who cannot translate it, which is why compression contests regularly require the extractor to be included in the message.
- svantana 13y ago
  On the contrary, for the stockbroker there's only a single bit of information in there - the answer to the yes/no question "should I short Dow?"
drakaal 13y ago
You have a molecule. Something that will "stay put" probably written on a "2d" surface like Graphene. Graphene is composed of lots of little hexagons. Each side of the hexagon can be broken and have an atom attached to it in "3d". You have 6 sides and the angle break can go "up" or "down". You can only use 3 sides however so that each hexagon has data and you can tell unique data. This gives you 3 positions in 3 states, Up, down, or Flat. 0.142 nanometers per bond... That's 27 states per hex, and 190 hexes per nanometer... 36,100 hexes per square nanometer... I'm sure I screwed up a calculation in there somewhere. But Based on current tech this is my answer to what is possible to write. Now Reading might be a bit harder at any speed... but hey this is all theory right?
- vilhelm_s 13y ago
  One cool idea (I saw it on Charles Stross' blog) is "diamond memory": use a diamond crystal, with two different isotopes of carbon for 1 and 0 bits. This also doesn't feel too unimaginable, in theory. According to Wolfram Alpha[1] this gives 1.75*10^23 bits (20 zettabytes) per CC. [1] http://www.wolframalpha.com/input/?i=number%20of%20atoms%20in%20one%20cc%20of%20diamond http://www.wolframalpha.com/input/?i=number%20of%20atoms%20i...
dizzystar 13y ago
Can anyone put this in terms like 100100 Petabytes or something? Or is the number so large that it really isn't conceivable at this point? I'll admit that I don't understand the problem completely, but isn't this assuming the absolute maximum with little consideration of actual technology limits? How much does it change when we consider the limits of technology and our ability to store it on said technology?
- sesqu 13y ago
  The number is far too large to have an SI name. Some people have put forward suggestions for new prefixes, but they wouldn't help you understand the number. Edit: I suppose you could say 1 exayottayottabit, or 100 pebiyobiyobibytes, per cc. And yes, this relies on using qubits for storage, in greater densities than is achievable.
- gibybo 13y ago
  There isn't a friendly name for 10^66 bits, but perhaps you can get an idea for the scale by considering the sizes of more commonly used names: bit: 10^0, kilobit: 10^3, megabit: 10^6, gigabit: 10^9, terabit: 10^12, ... zettabit: 10^21, yottabit: 10^24 So we're at some pretty ridiculously large numbers and still nowhere near 10^66.
Steuard 13y ago
The remarkable thing that this explanation doesn't really explore in depth is that the maximum information (~entropy) contained in a region scales according to its boundary area rather than to its volume. That means that "maximum information per unit volume" drops precipitously with size.[1] Just for example, this allows you to calculate an ultimate limit on Moore's law of only ~800 years, for any possible computer functioning within the bounds of the observable universe. As sketched below[1], the observable universe can hold only about 10^123 bits of information. Processors currently contain about 10^9 transistors (each of which has to be doing computations with an independent bit of information to be useful), a factor of 10^114 less. If Moore's law claims that this number doubles every 2 years, that means it grows by a factor of 10^3 every 20. And 20x(114/3) is about 760 years. (A more detailed calculation carried out in a paper by Krauss and Starkman at http://arxiv.org/abs/astro-ph/0404510 http://arxiv.org/abs/astro-ph/0404510 came up with a limit of about 600 years.) That's almost frighteningly soon. [1] As the link says, for a cubic centimeter (cc) of volume, the maximum entropy is about 10^66 bits. But if you consider a cubic meter instead, you find it can hold at most 10^70 bits, which comes out to 10^64 bits per cc! For a cubic kilometer you get 10^76 bits, which is only 10^61 bits per cc. If the solar system has radius ~10^13 m, it could hold at most 10^96 bits, or 10^51 bits/cc. And the whole observable universe (with radius ~5x10^26 m) could hold at most 2.5x10^123 bits, or 10^38 bits/cc. That's remarkably less than the direct one cc calculation! This behavior is exceedingly non-intuitive, at least to me.
- Xcelerate 13y ago
  > This behavior is exceedingly non-intuitive, at least to me. I've been thinking about the notion of the Bekenstein Bound lately, and I'm not really sure how accurate the whole idea is. There seems to be a few problems with it, and I don't get the sense that it is a particurly heavily studied problem in the physics community (compared to something like general relativity or QFT). For one thing, nobody is certain that "actual" black holes exist. What I mean by this is a black hole where matter has actually fallen into it. Instead, the matter approaches the event horizon at an increasingly slower pace for an observer until at some point in time it is effectively frozen an infinitesimal distance from the event horizon (but still not in the black hole). In this sense, of course the information content is proportional to the surface area of the black hole. (I should note that mathematically, "effective" black holes and "real" black holes end up having the same properties and behavior; they'd be indistinguishable to an observer). Another problem with the Bekenstein Bound is that quantum states aren't localized to a volume of space. You can't just hold up a beach ball and say "What's the maximum information content of this beach ball?". Why? Because you can't just have a wavefunction of just "the beachball". It's a pretty good approximation, but what about the electron-electron correlations at the edge of the beach ball? And what if the spin of an electron within the beach ball is entangled with an electron outside of it? The information describing that particular quantum state is then delocalized.
MildlySerious 13y ago
So the theories here assume qubits are the smallest unit of storage. I'm not into physics, so how's the evidence they are really the smallest units possible to store information as? After all, atoms were said to be the smallest thing at some point.
- maaku 13y ago
  Sure, if you postulate new physics anything is possible.
galaktor 13y ago
might be nitpicking a bit here, but "information" != "data". It is hard to tell how little or much information I can extract from any amount of data. data + context = information

58 points by lpage 13y ago

This assumes that all information is stored physically. Let's say you get a telegram. The telegram itself can contain maybe a couple of paragraphs of text. You might think the bandwidth of the channel is at most a kilobyte or so. But there's plenty of other aspects that can raise the amount of information conveyed. Say you get one saying "Short Dow." Just going by the content, you'd have no idea what it was saying. Is there a guy named Dow somewhere that's short? But if you're a stockbroker, all of a sudden there's a whole lot more info there. When you allow for context, information density can approach infinity.

itcmcgrath 13y ago
No, let me paraphrase what you just said. 'If you can store infinite information outside of a system, then you can achieve infinite information density inside the system if you use outside the system as 'context'. I hope that paraphasing makes it clear enough the problem with this line of thinking. You have to include all the information when calculating absolute density. In your case, the context has to be stored somewhere too.
- vinceguidry 13y ago
  If all information has to be accounted for and stored somewhere, and context is part of the information, then you can't store any information without storing all information, everywhere. Because every bit of information exists inside the context of the entire universe.
  - bradleyland 13y ago
    You're getting very metaphysical here, but reality remains the same even if you expand these principles to the universe. The rules of physics still apply.
    - gibwell 13y ago
      I think he has a point, although it's more about the semantics of the term 'information' density. Shannon information is always measured relative to a receiving context in which it the symbols are understood, and information content is related to the inverse of the probability of observing a particular signal as assessed by the receiver. So from that perspective vinceguidry has a reasonable point. However really the question being asked is something more like 'data density' and that is generally what people are talking about when the term 'information density' is invoked. Edit: I see that the original article does indeed refer to Data Density, and that HN title is just wrong.
  - smosher 13y ago
    Keep going, you're about to discover the holographic principle for yourself.
- deleted 13y ago
  [deleted]
- salmonellaeater 13y ago
  It's worse than that. There are a finite number of states of a message, and you can only reference that number of states of the outside system. If you think of the message as a pointer containing n bits, then you can only reference the first 2^n positions in the dictionary. If the outside system is a probability distribution over output messages (a more general case of the dictionary you described) then the problem is synonymous with compression.
twinge 13y ago
I think you're describing compression.
drakaal 13y ago
You are Dr. Who. You have a Tardis. It will convert your language in to any other language telepathically. Why not create a word that means the summation of everything you know, and say it to a person. They create a word that means the summation of everything they know and everything you know and say it to the next person...
- logicallee 13y ago
  Even better - attribute a meaning, in that language, to refusing to say anything to the next person (remaining silent). Then you can put the summation of everything they know and everything you know into the case where you don't say anything (in that language). This is similar to making a version of gcc that outputs a Tetris program everytime it's asked to compile a 0-byte file, or maybe outputs all of Wikipedia. I mean, sure, you can hide Tetris, or the whole of human knowledge, in 0 bytes this way. But it's not a very useful exercise.
  - meta-coder 13y ago
    Remaining silent is as good as communicating one bit of information. Even a 0-byte file has metadata. Even if it is 0-byte long, GCC knows that it is reading an input file. How did it come to know? Because you communicated some amount of information by initiating the compilation.
beagle3 13y ago
Only chsnges things up to a constant, and not fundamentally. Read up on Kolmogorov Complexity if you are interested.
nitrogen 13y ago
The message didn't actually contain the extra information; your stockbroker supplied that him/herself.
meta-coder 13y ago
> When you allow for context, information density can approach infinity. I absolutely agree with you. You are my hero. The amount of information stored in an object (which is capable of storing at least one bit of information in traditional sense) depends on the size of the context. If the object is not even capable of storing one bit of information in traditional sense, then the amount of information that can be stored is zero. And for all objects that can store one bit or more of information in traditional sense, the total amount of information that can be stored in it = the number of bits it can store + the number of bits that can be stored in rest of the universe (context). So any one bit object can store the same amount of information that can be stored in the entire universe. And if it turns out that our universe is enclosed in yet another larger universe, then you have to include that as part of the context as well. Edit: typo
sesqu 13y ago
No. If you were a stockbroker, that message would contain less information - and you probably would know it already. It's useful information to a stockbroker, but it would be more informative to almost anyone else, who doesn't encounter such messages daily. As another example, the information in the Rosetta stone was diminished - if more accessible - to its carvers, and a 700 MB disk holds 700 MB regardless of what those bits are. What you're referring to is compression. High information density is indistinguishable from noise to any recipient who cannot translate it, which is why compression contests regularly require the extractor to be included in the message.
svantana 13y ago
On the contrary, for the stockbroker there's only a single bit of information in there - the answer to the yes/no question "should I short Dow?"

drakaal 13y ago

You have a molecule. Something that will "stay put" probably written on a "2d" surface like Graphene. Graphene is composed of lots of little hexagons. Each side of the hexagon can be broken and have an atom attached to it in "3d". You have 6 sides and the angle break can go "up" or "down". You can only use 3 sides however so that each hexagon has data and you can tell unique data. This gives you 3 positions in 3 states, Up, down, or Flat. 0.142 nanometers per bond... That's 27 states per hex, and 190 hexes per nanometer... 36,100 hexes per square nanometer... I'm sure I screwed up a calculation in there somewhere. But Based on current tech this is my answer to what is possible to write. Now Reading might be a bit harder at any speed... but hey this is all theory right?

vilhelm_s 13y ago
One cool idea (I saw it on Charles Stross' blog) is "diamond memory": use a diamond crystal, with two different isotopes of carbon for 1 and 0 bits. This also doesn't feel too unimaginable, in theory. According to Wolfram Alpha[1] this gives 1.75*10^23 bits (20 zettabytes) per CC. [1] http://www.wolframalpha.com/input/?i=number%20of%20atoms%20in%20one%20cc%20of%20diamond http://www.wolframalpha.com/input/?i=number%20of%20atoms%20i...

dizzystar 13y ago

Can anyone put this in terms like 100100 Petabytes or something? Or is the number so large that it really isn't conceivable at this point? I'll admit that I don't understand the problem completely, but isn't this assuming the absolute maximum with little consideration of actual technology limits? How much does it change when we consider the limits of technology and our ability to store it on said technology?

sesqu 13y ago
The number is far too large to have an SI name. Some people have put forward suggestions for new prefixes, but they wouldn't help you understand the number. Edit: I suppose you could say 1 exayottayottabit, or 100 pebiyobiyobibytes, per cc. And yes, this relies on using qubits for storage, in greater densities than is achievable.
gibybo 13y ago
There isn't a friendly name for 10^66 bits, but perhaps you can get an idea for the scale by considering the sizes of more commonly used names: bit: 10^0, kilobit: 10^3, megabit: 10^6, gigabit: 10^9, terabit: 10^12, ... zettabit: 10^21, yottabit: 10^24 So we're at some pretty ridiculously large numbers and still nowhere near 10^66.

Steuard 13y ago

The remarkable thing that this explanation doesn't really explore in depth is that the maximum information (~entropy) contained in a region scales according to its boundary area rather than to its volume. That means that "maximum information per unit volume" drops precipitously with size.[1] Just for example, this allows you to calculate an ultimate limit on Moore's law of only ~800 years, for any possible computer functioning within the bounds of the observable universe. As sketched below[1], the observable universe can hold only about 10^123 bits of information. Processors currently contain about 10^9 transistors (each of which has to be doing computations with an independent bit of information to be useful), a factor of 10^114 less. If Moore's law claims that this number doubles every 2 years, that means it grows by a factor of 10^3 every 20. And 20x(114/3) is about 760 years. (A more detailed calculation carried out in a paper by Krauss and Starkman at http://arxiv.org/abs/astro-ph/0404510 http://arxiv.org/abs/astro-ph/0404510 came up with a limit of about 600 years.) That's almost frighteningly soon. [1] As the link says, for a cubic centimeter (cc) of volume, the maximum entropy is about 10^66 bits. But if you consider a cubic meter instead, you find it can hold at most 10^70 bits, which comes out to 10^64 bits per cc! For a cubic kilometer you get 10^76 bits, which is only 10^61 bits per cc. If the solar system has radius ~10^13 m, it could hold at most 10^96 bits, or 10^51 bits/cc. And the whole observable universe (with radius ~5x10^26 m) could hold at most 2.5x10^123 bits, or 10^38 bits/cc. That's remarkably less than the direct one cc calculation! This behavior is exceedingly non-intuitive, at least to me.

Xcelerate 13y ago
> This behavior is exceedingly non-intuitive, at least to me. I've been thinking about the notion of the Bekenstein Bound lately, and I'm not really sure how accurate the whole idea is. There seems to be a few problems with it, and I don't get the sense that it is a particurly heavily studied problem in the physics community (compared to something like general relativity or QFT). For one thing, nobody is certain that "actual" black holes exist. What I mean by this is a black hole where matter has actually fallen into it. Instead, the matter approaches the event horizon at an increasingly slower pace for an observer until at some point in time it is effectively frozen an infinitesimal distance from the event horizon (but still not in the black hole). In this sense, of course the information content is proportional to the surface area of the black hole. (I should note that mathematically, "effective" black holes and "real" black holes end up having the same properties and behavior; they'd be indistinguishable to an observer). Another problem with the Bekenstein Bound is that quantum states aren't localized to a volume of space. You can't just hold up a beach ball and say "What's the maximum information content of this beach ball?". Why? Because you can't just have a wavefunction of just "the beachball". It's a pretty good approximation, but what about the electron-electron correlations at the edge of the beach ball? And what if the spin of an electron within the beach ball is entangled with an electron outside of it? The information describing that particular quantum state is then delocalized.

MildlySerious 13y ago

So the theories here assume qubits are the smallest unit of storage. I'm not into physics, so how's the evidence they are really the smallest units possible to store information as? After all, atoms were said to be the smallest thing at some point.

maaku 13y ago
Sure, if you postulate new physics anything is possible.

galaktor 13y ago

might be nitpicking a bit here, but "information" != "data". It is hard to tell how little or much information I can extract from any amount of data. data + context = information