5 ms·

The snowmobile was for data transfer, not data storage. The idea was that you upload your data to the truck on-site, then they drive it to the backup and downlo

The snowmobile was for data transfer, not data storage. The idea was that you upload your data to the truck on-site, then they drive it to the backup and download it again, faster than you could transfer it over the wire Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981

mlnj 2y ago
>Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981 I've always loved that quote.
- bombcar 2y ago
  It's a great way to show the difference between bandwidth and latency.
icedchai 2y ago
Yes, but I suspect the parent meant that storage density has significantly increased, so you now don't need an entire truck's worth of it. The other "AWS Snow" options may now suffice? You gotta wonder how many companies actually used the truck...
- jerf 2y ago
  Eyeballing this chart: https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_... It looks like over the past 8 or so years you're looking at a maybe a tripling in capacity. Cut the size of the truck down by a third, accounting for the fact the truck couldn't have been stuffed with every cubic inch being hard drive due to the need to walk in it, weight limits, cooling, wiring, etc., and I think it's not hard to see that even if it did make sense at the time, it's not hard to run the numbers and build some 4-6 U storage units and ship those instead. Even if you ship 4 of those it's cheaper than a truck by quite a bit. Also, if you're willing to spend the dough, flash density can be pushed even higher, I think. You might need some custom hardware to manage power to the drives, so only the drives the customer is filling right now are powered (heat management is going to be a big deal with flash), but that's not impossibly difficult. You can even drain an entire data center by cycling the same 3 or 4ish units around. Fill one up, send it in, Amazon loads it up, send the same unit back. Even if it takes 10 loads you don't necessarily need 10 hunks of hardware. Personally I have the sneaking suspicion that it was always a marketing stunt and never actually made monetary sense. It was just a grand gesture to reassure people that they could indeed move to the cloud, which was a large concern for a lot of large customers. In that sense it probably paid for itself. And once you do have it in hand, maybe the marginal costs are still attractive and you go ahead and use it sometimes. But I'd personally bet the "round robin of 4-6U storage racks" would always have been more economical regardless of hard drive densities.
lazide 2y ago
Anyone have an updated estimate for how much storage could be fit in a shipping container and its bandwidth equivalent? [https://news.ycombinator.com/item?id=1437634#:~:text=Conservatively%20speaking%2C%20you%20co...%20%7C%20Hacker%20News&text=file%20sharing%20network-,Or%20a%20shipping%20container%20filled%20with%20hard%20drives.,3.5%22%202%20TB%20hard%20drives https://news.ycombinator.com/item?id=1437634#:~:text=Conserv....] says 60PB, which for 20 TB drives would be approx 3000. A petabyte is still a lot of data! In my experience the bigger issue is that much large data sets are rarely truly static, and inventorying/tracking/validating is a massive undertaking all on it’s own.
- voidUpdate 2y ago
  Obligatory XKCD: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
  - lazide 2y ago
    As a side note since XKXD leaves this out and having gotten involved in implementing systems using physical storage for transport in the past, physical copying to/from media is a major issue in performance and capital costs. Most large physical hard drives max out around 300-400MB/s transfer rate, so it typically takes several days to fill a single drive. And you can’t ship anything until all the media is written. So either you have a ton of online, parallel drives (which dramatically increases capex, and decreases available storage due to physical space, power, and cooling issues) or you go really slow. Even having 300 parallel full speed drive interfaces for instance, it would take approximately a year to fill those 3000 drives with data, and approx. another year to read all the data off. Which dramatically changes all the calculations.

mlnj 2y ago

>Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981 I've always loved that quote.

bombcar 2y ago
It's a great way to show the difference between bandwidth and latency.

icedchai 2y ago

Yes, but I suspect the parent meant that storage density has significantly increased, so you now don't need an entire truck's worth of it. The other "AWS Snow" options may now suffice? You gotta wonder how many companies actually used the truck...

jerf 2y ago
Eyeballing this chart: https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_... It looks like over the past 8 or so years you're looking at a maybe a tripling in capacity. Cut the size of the truck down by a third, accounting for the fact the truck couldn't have been stuffed with every cubic inch being hard drive due to the need to walk in it, weight limits, cooling, wiring, etc., and I think it's not hard to see that even if it did make sense at the time, it's not hard to run the numbers and build some 4-6 U storage units and ship those instead. Even if you ship 4 of those it's cheaper than a truck by quite a bit. Also, if you're willing to spend the dough, flash density can be pushed even higher, I think. You might need some custom hardware to manage power to the drives, so only the drives the customer is filling right now are powered (heat management is going to be a big deal with flash), but that's not impossibly difficult. You can even drain an entire data center by cycling the same 3 or 4ish units around. Fill one up, send it in, Amazon loads it up, send the same unit back. Even if it takes 10 loads you don't necessarily need 10 hunks of hardware. Personally I have the sneaking suspicion that it was always a marketing stunt and never actually made monetary sense. It was just a grand gesture to reassure people that they could indeed move to the cloud, which was a large concern for a lot of large customers. In that sense it probably paid for itself. And once you do have it in hand, maybe the marginal costs are still attractive and you go ahead and use it sometimes. But I'd personally bet the "round robin of 4-6U storage racks" would always have been more economical regardless of hard drive densities.

lazide 2y ago

Anyone have an updated estimate for how much storage could be fit in a shipping container and its bandwidth equivalent? [https://news.ycombinator.com/item?id=1437634#:~:text=Conservatively%20speaking%2C%20you%20co...%20%7C%20Hacker%20News&text=file%20sharing%20network-,Or%20a%20shipping%20container%20filled%20with%20hard%20drives.,3.5%22%202%20TB%20hard%20drives https://news.ycombinator.com/item?id=1437634#:~:text=Conserv....] says 60PB, which for 20 TB drives would be approx 3000. A petabyte is still a lot of data! In my experience the bigger issue is that much large data sets are rarely truly static, and inventorying/tracking/validating is a massive undertaking all on it’s own.

voidUpdate 2y ago
Obligatory XKCD: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- lazide 2y ago
  As a side note since XKXD leaves this out and having gotten involved in implementing systems using physical storage for transport in the past, physical copying to/from media is a major issue in performance and capital costs. Most large physical hard drives max out around 300-400MB/s transfer rate, so it typically takes several days to fill a single drive. And you can’t ship anything until all the media is written. So either you have a ton of online, parallel drives (which dramatically increases capex, and decreases available storage due to physical space, power, and cooling issues) or you go really slow. Even having 300 parallel full speed drive interfaces for instance, it would take approximately a year to fill those 3000 drives with data, and approx. another year to read all the data off. Which dramatically changes all the calculations.