6 ms·

"For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of ne

"For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage every day or about 100x the Library of Congress. As shown in the graph, this continues to grow exponentially, with a 10x increase every five years." This blew my mind. What does this even mean in terms of logistics. How many people do you need to have just to add all those hard drives? How many new datacenters do you need to build every 5 years?

nostrademons 11y ago
It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course, that's just YouTube, and Google has many other needs for data. But people forget just how big the denominators are on this quantity, and how effective Kryder's Law has been. They also forget how much reserve capacity there is in human labor; Google's datacenters have tiny employee counts because they are so automated, and could easily scale up into the exabyte/day range. A more interesting question is what the differential rates of Kryder's Law vs. Moore's Law will do to how we architect software. Already, people in the know say that "disk is the new tape" - disk drive capacity has been increasing much faster than seek times, bus bandwidth, and available processing power, which means that you have to start treating the drive as a sequential storage device and not as a random-access platter. That's behind a lot of the shift from B-trees (as in conventional RDBMSes) to LSM-trees (as in BigTable/LevelDB), and also the resurgence of batch-processing frameworks like MapReduce. How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive?
- msellout 11y ago
  > How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive? I thought we were already in that situation. Cache is king.
  - thfuran 11y ago
    The real change is that with increasingly performant and large SSDs, it's becoming reasonable to have main storage for which random access is orders of magnitude faster than HDDs will ever be. Still much slower than cache, but a few orders of magnitude here and there are likely to shift what optimal strategies are.
- dc2447 11y ago
  > It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course at this scale you provision by prebuilt rack or even by container
- imaginenore 11y ago
  I think you forget the redundancy. You probably need to double it to 256 drives per day.
  - gist 11y ago
    Do they do need to even do that? Couldn't that be engineered into the free space on other physical drives and change as needed? (I do a version of that on a very small scale and noting I am not expert in this area it was just a common sense approach.) Meaning it wouldn't have to be "doubled" or anything close to that.
    - nostrademons 11y ago
      They do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that sits on top of it and provides the abstraction of an infinite, highly-available, perfectly robust filesystem. So services like YouTube just specify a path, data, an owner, and a replication policy and then hand it off to Colossus to find free space on one of the zillions of disks. There's a side mechanism for quota & budgeting. It's unclear from the article whether YouTube's 1P/day is pre-replication or post-replication. I'd just assumed post-replication; it doesn't really change the conclusion in a material way. (Rather than the answer being "1 person", it becomes "2-3 people".) And yes, you do need 2x (or more commonly, 3x) the disk space. You can use error-correcting codes to correct single-bit errors (Colossus uses Reed-Solomon), but that won't help you if there's a fiber cut and a whole datacenter goes offline.
      - gist 11y ago
        Hah! Thanks for making me feel somewhat smart. Reminds me of when I was a kid and I said to my mom about cancer "why don't they just cut it out?".
Sanddancer 11y ago
Given 8tb drives and chassis that can fit 90 3.5" drives into a 4u enclosure, you're talking about a petabyte fitting into 8u. Sledding takes about 8 hours for that many drives, so a small team would be able to handle hardware buildouts pretty easily. Moreso if you can lean on your integrator to do the work for you.

nostrademons 11y ago

It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course, that's just YouTube, and Google has many other needs for data. But people forget just how big the denominators are on this quantity, and how effective Kryder's Law has been. They also forget how much reserve capacity there is in human labor; Google's datacenters have tiny employee counts because they are so automated, and could easily scale up into the exabyte/day range. A more interesting question is what the differential rates of Kryder's Law vs. Moore's Law will do to how we architect software. Already, people in the know say that "disk is the new tape" - disk drive capacity has been increasing much faster than seek times, bus bandwidth, and available processing power, which means that you have to start treating the drive as a sequential storage device and not as a random-access platter. That's behind a lot of the shift from B-trees (as in conventional RDBMSes) to LSM-trees (as in BigTable/LevelDB), and also the resurgence of batch-processing frameworks like MapReduce. How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive?

msellout 11y ago
> How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive? I thought we were already in that situation. Cache is king.
- thfuran 11y ago
  The real change is that with increasingly performant and large SSDs, it's becoming reasonable to have main storage for which random access is orders of magnitude faster than HDDs will ever be. Still much slower than cache, but a few orders of magnitude here and there are likely to shift what optimal strategies are.
dc2447 11y ago
> It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course at this scale you provision by prebuilt rack or even by container
imaginenore 11y ago
I think you forget the redundancy. You probably need to double it to 256 drives per day.
- gist 11y ago
  Do they do need to even do that? Couldn't that be engineered into the free space on other physical drives and change as needed? (I do a version of that on a very small scale and noting I am not expert in this area it was just a common sense approach.) Meaning it wouldn't have to be "doubled" or anything close to that.
  - nostrademons 11y ago
    They do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that sits on top of it and provides the abstraction of an infinite, highly-available, perfectly robust filesystem. So services like YouTube just specify a path, data, an owner, and a replication policy and then hand it off to Colossus to find free space on one of the zillions of disks. There's a side mechanism for quota & budgeting. It's unclear from the article whether YouTube's 1P/day is pre-replication or post-replication. I'd just assumed post-replication; it doesn't really change the conclusion in a material way. (Rather than the answer being "1 person", it becomes "2-3 people".) And yes, you do need 2x (or more commonly, 3x) the disk space. You can use error-correcting codes to correct single-bit errors (Colossus uses Reed-Solomon), but that won't help you if there's a fiber cut and a whole datacenter goes offline.
    - gist 11y ago
      Hah! Thanks for making me feel somewhat smart. Reminds me of when I was a kid and I said to my mom about cancer "why don't they just cut it out?".

Sanddancer 11y ago

Given 8tb drives and chassis that can fit 90 3.5" drives into a 4u enclosure, you're talking about a petabyte fitting into 8u. Sledding takes about 8 hours for that many drives, so a small team would be able to handle hardware buildouts pretty easily. Moreso if you can lean on your integrator to do the work for you.