3 ms·

Does the Wayback Machine have a long term plan that anyone is familiar with? Is their goal to preserve the web indefinitely? Is the hope the storage and compres

Does the Wayback Machine have a long term plan that anyone is familiar with? Is their goal to preserve the web indefinitely? Is the hope the storage and compression improvements over time keep up with content creation? And just to be clear, I think the Wayback Machine is great and the fact that I can lookup my personal, basically zero traffic website from 15 years ago and see it is truly astonishing to me–I'm just curious what this look likes in 10, 20, 50 years.

toomuchtodo 7y ago
The best way to get this question answered would be to email or tweet at Brewster Kahle, who started and heads the Internet Archive. If you're in San Francisco, the Internet Archive is also hosting a block part the evening of Oct 23rd from 5pm-10pm, and staff will be there to answer questions (tickets are $15). Disclaimer: No affiliation
- jonah-archive 7y ago
  If anyone's interested in attending our party next Wednesday but the cost is presenting a difficulty, shoot me an email (in my profile) and I'll send you a ticket at no cost.
willis936 7y ago
Information is not infinitely compressible. In fact, for lossless compression, it’s proven that we can’t do much better than we already do. At some point you just need more storage. However we are several orders of magnitude away from the physical information density limit (surface of a black hole).
- meesles 7y ago
  > In fact, for lossless compression, it’s proven that we can’t do much better than we already do. Source? Sounds like one wouldn't know a better alternative exists until one discovers it...
  - Aperocky 7y ago
    It’s basically conservation of energy/entropy being the reason. As in you cannot violate physics.
  - wwwhizz 7y ago
    There is a whole scientific field on this, called Information Theory. Compressing entropy is limited by its randomness. For instance see: https://en.wikipedia.org/wiki/Entropy_%28information_theory%29#Data_compression https://en.wikipedia.org/wiki/Entropy_%28information_theory%... and https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity
    - throwaway_bad 7y ago
      What if most of the data humans care about is not random? For example to recover the state of an entire simulated universe you just need the value of the initial seed and the generator.
      - strbean 7y ago
        I think statements about theoretical limits on compression are ignoring emergent properties. We know you can "compress" certain things infinitely; for example, the Mandelbrot set. Compressing arbitrary inputs using emergent properties may never be practical, but in it seems reasonable that you could trade computation for compression to an arbitrary extent (searching an emergent series for chunks of data that match your input).
        throwaway_bad 7y ago
        I was actually thinking more about natural languages than fractals. Maybe human thought is so utterly derivative you can just generate a random stream of words and it will contain most of the text that humanity will ever produce. Then it can be compressed down to an index into the libraryofbabel
        willis936 7y ago

toomuchtodo 7y ago

The best way to get this question answered would be to email or tweet at Brewster Kahle, who started and heads the Internet Archive. If you're in San Francisco, the Internet Archive is also hosting a block part the evening of Oct 23rd from 5pm-10pm, and staff will be there to answer questions (tickets are $15). Disclaimer: No affiliation

jonah-archive 7y ago
If anyone's interested in attending our party next Wednesday but the cost is presenting a difficulty, shoot me an email (in my profile) and I'll send you a ticket at no cost.

willis936 7y ago

Information is not infinitely compressible. In fact, for lossless compression, it’s proven that we can’t do much better than we already do. At some point you just need more storage. However we are several orders of magnitude away from the physical information density limit (surface of a black hole).

meesles 7y ago
> In fact, for lossless compression, it’s proven that we can’t do much better than we already do. Source? Sounds like one wouldn't know a better alternative exists until one discovers it...
- Aperocky 7y ago
  It’s basically conservation of energy/entropy being the reason. As in you cannot violate physics.
- wwwhizz 7y ago
  There is a whole scientific field on this, called Information Theory. Compressing entropy is limited by its randomness. For instance see: https://en.wikipedia.org/wiki/Entropy_%28information_theory%29#Data_compression https://en.wikipedia.org/wiki/Entropy_%28information_theory%... and https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity
  - throwaway_bad 7y ago
    What if most of the data humans care about is not random? For example to recover the state of an entire simulated universe you just need the value of the initial seed and the generator.
    - strbean 7y ago
      I think statements about theoretical limits on compression are ignoring emergent properties. We know you can "compress" certain things infinitely; for example, the Mandelbrot set. Compressing arbitrary inputs using emergent properties may never be practical, but in it seems reasonable that you could trade computation for compression to an arbitrary extent (searching an emergent series for chunks of data that match your input).
      - throwaway_bad 7y ago
        I was actually thinking more about natural languages than fractals. Maybe human thought is so utterly derivative you can just generate a random stream of words and it will contain most of the text that humanity will ever produce. Then it can be compressed down to an index into the libraryofbabel
      - willis936 7y ago