9 ms·
128-bit storage: are you high?
- iwr 16y agoOne day, aliens will use our oceans as heat sinks for their great galactic computer. And even then, ZFS will serve them well.
- rbanffy 16y agoWe can use a ZFS storage system to make the moons of Saturn inhabitable. That's Sealand taken to a whole new level. The network latencies are terrible, however.
- count 16y agoCharles Stross wrote a book loosely incorporating this concept (Accelerando, I think).
- Tuna-Fish 16y ago> One day, we will use our oceans as heat sinks for our great galactic computer. And even then, ZFS will serve us well. FTFY
- thushan 16y agoDoesn't Douglas Adams touch on that? Wasn't Earth a big computer in the Hitchhiker's Guide to the Galacy? From a wikipedia entry... "[The computer] Deep Thought does not know the ultimate question to Life, the Universe and Everything, but offers to design an even more powerful computer, Earth, to calculate it. After ten million years of calculation, the Earth is destroyed by Vogons five minutes before the computation is complete."
- iuguy 16y agoDeep Thought used ZFS. The ZFS pool for the calculations was 98% full when the Vogons came.
- acabal 16y agoThe most interesting thing in this article is the reference to "Ultimate physical limits to computation." It's something I never thought about before and honestly it kind of blew my mind a little bit...
- dschobel 16y agoAgreed. Unfortunately the Nature article is behind a paywall but the original paper is available on arxiv: http://arxiv.org/abs/quant-ph/9908043 http://arxiv.org/abs/quant-ph/9908043
- foco1234 16y agoI've read the intro. of the paper, but I'm not a physicist, so this question might seem dumb, but here goes: The paper describes the "ultimate laptop" as being a computer that can use one atom to store 1 bit of information. Why is an atom the smallest unit of matter a computer can store a bit in?
- gwern 16y agoDoes Lloyd actually say that? IIRC, he simply reuses the Bekenstein bound: https://secure.wikimedia.org/wikipedia/en/wiki/Bekenstein_bound https://secure.wikimedia.org/wikipedia/en/wiki/Bekenstein_bo... Which is more than 1 bit per atom. (It does seem implausible for 1 bit per atom to be right. I mean, atoms themselves vary in hundreds of ways - # of protons, # of electrons, etc. Just using each element to represent a byte would seem to get you more than 1 bit per atom.)
- ten7 16y agoNice article, great calculation. There's a much simpler argument for 128 bit storage though: the observable universe has at minimum 10^79 atoms in it. Even if you could use each and every one of those atoms as a bit to store data, you'd still have more than 50 ORDERS OF MAGNITUDE left in ZFS for the rest of the data. We're probably safe.
- cperciva 16y agoYou realize that 2^128 is much less than 10^79, right?
- barrkel 16y agoA handy rule: log of 2 to base 10 is a bit more than 0.3, so divide the power of 2 by 3 (i.e. approximately multiply by 0.3), and you'll get an approximation of the power of 10. By this, 2^128 is in the region of 10^40, a lot less than 10^79. (It's actually about 10^38.5.)
- aw3c2 16y ago10^79 = 10000000000000000000000000000000000000000000000000000000000000000000000000000000 2^128= 340282366920938463463374607431768211456
- ten7 16y agoAhh shit, totally lame brained that.
- ten7 16y agoYea, I screwed up. What the hell was I thinking? Thanks for down voting: it's deserved.
- MaxMorlock 16y agoThe 128-bit storage of ZFS reminds me a bit of TeX. As far as I can remember reading the TeX book, Don Knuth also wanted to play it safe. Internally, everything relies on integers (no floating point operations), and the smallest internal unit of measurement is less than the wavelength of light (don't remember the color ;-) ).
- andrewf 16y agoAt least one game programmer is recommending sticking with integers for distances. " 64 bits of precision gets you to the furthest distance of Pluto from the Sun (7.4 billion km) with sub-micrometer precision. " http://home.comcast.net/~tom_forsyth/blog.wiki.html#%5B%5BA%20matter%20of%20precision%5D%5D http://home.comcast.net/~tom_forsyth/blog.wiki.html#%5B%5BA%...
- fexl 16y agoI too played it safe and avoided floating point operations in the loom.cc code: https://loom.cc/?function=grid_tutorial&help=1 https://loom.cc/?function=grid_tutorial&help=1 All asset quantities are signed 128 bit integers under the hood. However, when the wallet interface displays a quantity, it uses a scale factor to place the decimal point. An asset type denominated in a fiat currency like USD or EUR typically uses a scale of 2. An asset type denominated in a commodity currency like GAU (grams of gold) typically uses a scale of 7, allowing a number like 31.1034768, which is the precise number of grams in a troy ounce. If you want to see it in kilograms, you can use scale 10.
- pyre 16y agoA (2004) at the end would be nice. > I've had people tell me every year, for years, that Moore's Law was > about to end. I've said the opposite, and have yet to lose the bet. > Limits on spot density are fundamentally arguments about 2D storage. > Once we move into 3D -- and this work is already underway -- we will > get many more orders of magnitude to play with. This comment was made in 2004. Where are our '3D' storage devices?
- space-monkey 16y agohttp://www.electroiq.com/index/Semiconductors/sst-blogs/chipworks-blog-display/blogs/sst-blogs/chipworks/post987_7451506051585892995.html http://www.electroiq.com/index/Semiconductors/sst-blogs/chip... I doubt he was thinking flash memory at the time, but 3D storage is already in use. One of the bigger limitations with this is heat, so we have people writing papers like: http://scholar.google.com/scholar?q=Thermal+via+placement+in+3D+ICs&hl=en&btnG=Search&as_sdt=2001&as_sdtp=on http://scholar.google.com/scholar?q=Thermal+via+placement+in...
- pyre 16y agoI'm not sure if that really counts as 3D storage though. I'm pretty sure that the commentator was referring to holographic storage, which would theoretically have a higher information density, and actually use the third dimension for data storage. Unless I'm reading those links wrong, they are just stacking flash memory on top of each other. How is this different than finding a way to pack hard drive platters closer together? You're just taking several 2D storage mechanisms and stacking them on top of each other. You're not using the third dimension for anything.
- vilhelm_s 16y agoI'm not sure that the computation is completely to the point -- he starts out asking "what is the _smallest_ storage unit that could fit this many bytes", and then goes on to consider how much energy it would take to do so. But if we are primarily worried about energy usage, we can write it cheaper (at the cost, I guess, of using more mass for the storage medium). The usual limit that is mentioned for storage is Laundauer's principle (http://en.wikipedia.org/wiki/Landauer%27s_principle http://en.wikipedia.org/wiki/Landauer%27s_principle), which states that we need to use at least ln(2)kT Joules of energy to write one bit, where k is Boltzmann's constant and T is the temperature of the storage system. Assuming the best case T=Cosmic Background Temperature=2.7K (although this is optimistic since we will also be heated up by the sun and by the Milky way), this give 8.8e15 J to write 2^128 bits, or 3.6e19 J to write 2^128 512-byte blocks. That's much less than what Bonswick calculates, and less than the energy to boil the ocean!
- vilhelm_s 16y agoIn fact, here's a more down-to-earth idea of a high density storage: use a diamond crystal, with C-12 atoms for 'zero' bits and C-13 atoms for 'one' bits. (I learned about this idea from Charles Stross' blog). While it's not obvious how to read and write this, the concept is clearly not ruled out by the laws of physics. Then storing 2^64 512-byte disk blocks requires 1.5 grammes of diamond (7.5 carats), while storing 2^128 blocks requires 3e16 kg (a ball of diamond with a radius of 12 km). 2^128 is a big number, but perhaps not big enough to last us through the singularity.
- jing 16y agoYes, this is theoretically cool and all, but when I hear something like: "Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 250 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away." It raises a huge red flag from a business point of view. In 2004 "some customers" had datasets that would cause issues sometime around 2015 - a problem that ZFS purported to solve (among admittedly much more relavant features). In planning for a future that was possibly ten years away for "some customers" instead of focusing on what was relevant to the larger customer base, Sun managed to continue their slow profitability death march until Oracle finally snatched them up this year. Something to keep in mind when you decide to add "cool theoretical feature X" or "unparalleled scalability" to your four month old startup...
- PostOnce 16y agoLack of foresight can kill you too...
- dprice1 16y agoI lack the ability to downvote you, but this comment is clearly a series of fallacies. You've assumed at least two things I don't see in evidence: a) That the customers Sun served in 2004 didn't care about anything ten years away. That their time scale for data on disk was less than a decade. And that marketing to them about how ZFS planned for the future was not effective. b) That building a 128-bit filesystem (as opposed to a 64-bit filesystem) substantially impacted the amount of time it took to engineer the filesystem or impacted the adoption rate by customers. Clearly those are not facts in evidence. Since we know that an integer behaves pretty much the same regardless of whether it's 32, 64, or 128 bits, it's probably safer to assume the opposite. I take the opposite lesson: I think when planning your startup, a little thought into "are we representing data in a way which we can extend into the future?" is not a terrible idea, especially when the choice is between a 32, 64, or 128 bit integer.
- nl 16y agoZFS has been one of the biggest selling points for Solaris over the last 5 years. There are a huge number of places that use Solaris only for ZFS support. For example, the SmugMug people have written about it a number of times.
- TheNewAndy 16y agoIt can be useful to have a large address space, even if you don't have enough hardware to actually populate the whole thing. For example, you might split the address space into smaller sections (executable, read only, read/write, shared, etc) which immediately cuts down the space you have. Or you might decide that your method for allocating space is to pick a random part of the address space, and check if it is taken (or perhaps something slightly cleverer). In this case, you will get better performance if there aren't as many collisions.
- celoyd 16y agoOr you might have some highly drive-abstracted representation where every addressable thing in every version of a filesystem-like dataset has its own address. No one’s asking to do that now (as far as I know), but if we get 128 bits I bet we’ll come up with ways to use them.
- anamax 16y agoYou can also do things like versioning, which lets you detect use-after-delete, and typed pointers.
- mayank 16y agoWouldn't a better calculation involve (2^64 + 1) bytes of storage, or even 2^65, instead of going for the upper bound at (2^128)? In other words, what's the minimum energy requirement for 64 bits to be insufficient? A neat calculation for the wow-factor, but not really very informative.
- sedachv 16y agoFun fact: the AS/400 virtual instruction set uses 128-bit pointers (that get truncated when executing on a real machine). It's not excessive considering that the AS/400 is not only a flat-memory architecture, but a single-level store. I think storage will eventually move to the Venti model of hash addressing. It just makes much more sense on a network and with current archive-everything practices (there are some really big wins to be gained from compression and dedup there). BitTorrent is doing this already.
- konad 16y agoventi rules hard :)