5 ms·
AWS kills Snowmobile data transfer truck
- Havoc 3y agoMakes sense. Storage tech has moved significantly since
- voidUpdate 3y agoThe snowmobile was for data transfer, not data storage. The idea was that you upload your data to the truck on-site, then they drive it to the backup and download it again, faster than you could transfer it over the wire Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981
- mlnj 3y ago>Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981 I've always loved that quote.
- bombcar 3y agoIt's a great way to show the difference between bandwidth and latency.
- icedchai 3y agoYes, but I suspect the parent meant that storage density has significantly increased, so you now don't need an entire truck's worth of it. The other "AWS Snow" options may now suffice? You gotta wonder how many companies actually used the truck...
- jerf 3y agoEyeballing this chart: https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_... It looks like over the past 8 or so years you're looking at a maybe a tripling in capacity. Cut the size of the truck down by a third, accounting for the fact the truck couldn't have been stuffed with every cubic inch being hard drive due to the need to walk in it, weight limits, cooling, wiring, etc., and I think it's not hard to see that even if it did make sense at the time, it's not hard to run the numbers and build some 4-6 U storage units and ship those instead. Even if you ship 4 of those it's cheaper than a truck by quite a bit. Also, if you're willing to spend the dough, flash density can be pushed even higher, I think. You might need some custom hardware to manage power to the drives, so only the drives the customer is filling right now are powered (heat management is going to be a big deal with flash), but that's not impossibly difficult. You can even drain an entire data center by cycling the same 3 or 4ish units around. Fill one up, send it in, Amazon loads it up, send the same unit back. Even if it takes 10 loads you don't necessarily need 10 hunks of hardware. Personally I have the sneaking suspicion that it was always a marketing stunt and never actually made monetary sense. It was just a grand gesture to reassure people that they could indeed move to the cloud, which was a large concern for a lot of large customers. In that sense it probably paid for itself. And once you do have it in hand, maybe the marginal costs are still attractive and you go ahead and use it sometimes. But I'd personally bet the "round robin of 4-6U storage racks" would always have been more economical regardless of hard drive densities.
- lazide 3y agoAnyone have an updated estimate for how much storage could be fit in a shipping container and its bandwidth equivalent? [https://news.ycombinator.com/item?id=1437634#:~:text=Conservatively%20speaking%2C%20you%20co...%20%7C%20Hacker%20News&text=file%20sharing%20network-,Or%20a%20shipping%20container%20filled%20with%20hard%20drives.,3.5%22%202%20TB%20hard%20drives https://news.ycombinator.com/item?id=1437634#:~:text=Conserv....] says 60PB, which for 20 TB drives would be approx 3000. A petabyte is still a lot of data! In my experience the bigger issue is that much large data sets are rarely truly static, and inventorying/tracking/validating is a massive undertaking all on it’s own.
- voidUpdate 3y agoObligatory XKCD: https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- lazide 3y agoAs a side note since XKXD leaves this out and having gotten involved in implementing systems using physical storage for transport in the past, physical copying to/from media is a major issue in performance and capital costs. Most large physical hard drives max out around 300-400MB/s transfer rate, so it typically takes several days to fill a single drive. And you can’t ship anything until all the media is written. So either you have a ton of online, parallel drives (which dramatically increases capex, and decreases available storage due to physical space, power, and cooling issues) or you go really slow. Even having 300 parallel full speed drive interfaces for instance, it would take approximately a year to fill those 3000 drives with data, and approx. another year to read all the data off. Which dramatically changes all the calculations.
- tpetr 3y agoBummer, I always enjoyed joking about this “Tell your data to get Trucked”
- binarymax 3y agoSneakernets! https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet But I also never fully understood why an enterprise would pay so much money to give their data of this size to AWS to manage. It’s so much more expensive. If you’re at the capacity of needing a semi sneakernet, it would be far more cost effective and competitively advantageous to manage it yourself.
- coldtea 3y agoGood luck achieving the realibility and security of AWS by managing it yourself at anything "competitive". For some companies this would mean creating a whole department to manage that.
- spamizbad 3y agoWith that much data you're paying a fortune for AWS Operations people either way. Every company I've seen that tries to large-scale AWS thing lean ends up driving themselves into a ditch where they spend millions on contractors, consultants and 3rd party vendors to dig themselves out of. And when they're in that ditch they are neither secure nor reliable.
- nostrebored 3y agoI can set up an S3 bucket, efs, access, and backups, all reliable and geographically distributed in a single working day.
- spicyusername 3y agoYes, indeed. This is exactly the sneaky thing about the cloud. It seems so "simple". Now try doing that for multiple multi-thousand person orgs with competing processes, architectures, product cycles, etc. You'll find very quickly that costs spiral out of control in a very unpredictable and irreversible way. Self-hosting is something a company has much tighter control of the costs for. Most of those costs are known up front as capex purchases and predictable opex expenditures. They tend to seem expensive compared to the PoC S3 demo you just did, but often aren't compared to the actual realized expenses after migrating.
- gsky 3y agoIt's a great marketing gimmick. Everyone fell for it
- Sakos 3y agoFall for what? If you need to move that much data, you'll be thankful to have solutions like this. I'm not sure how it's a gimmick.
- gsky 3y agoIn the best case, only s&p 500 companies have that kinda data and could afford AWS costs.
- letmeinhere 3y agoYeah, it's shame they didn't spruce it up a bit. e.g. some winter theming on the exterior, allowing for some commemorative swag at conferences like they had for the warehouse robots or the delivery drones (also largely gimmicks).
- simonw 3y agoMy mental model of this is that Snowmobile was for companies with their own data centers looking to transfer petabytes of data to the cloud so they could stop self-hosting it. The number of companies with that problem is going to naturally fall over time. Companies that are already using the cloud don't need Snowmobile to move their data. Illustrative point from the article: "A spokesperson for satellite operator Maxar said the company used Snowmobile once in 2017 to move more than 100 petabytes to AWS from its own servers. Snowmobile was an “ideal solution for that moment in time,” the spokesperson said." You're not going to get many repeat customers for that service!
- ofrzeta 3y agoI wonder how much they pay for 100PB in AWS, depending on the storage service/tier. Also it will be quite expensive if they ever want to move out of AWS. Or is a project like this considered to be "eternal"?
- beeeeerp 3y agoIt’s actually hard to say. The prices these companies pay can be very different than list prices for us mortals. It’s one of my biggest complaints about AWS; the prices seem high, but in reality they are very opaque.
- woleium 3y agothey probably pay around 40% of list, depending on how much their ARR is.
- andersa 3y agoThis really needs to be made illegal already. All prices should be public and you should not be allowed to charge different customers different prices for the same automated service. I should be able to store my data at the exact same $/TB price as Epic Games, for example. They once mentioned it costs them less than a penny to serve several GB from AWS - why the fuck is it more expensive for me?
- HWR_14 3y agoMost likely reason is now the data transfer truck is more likely to have people remove their data than add it to the AWS cloud.
- randomdata 3y agoOr global warming. Yamaha is also killing their snowmobile. https://snowgoer.com/latest-news/yamaha-will-exit-the-snowmobile-market-in-model-year-2025/31676 https://snowgoer.com/latest-news/yamaha-will-exit-the-snowmo...
- globular-toast 3y agoWhy "Snowmobile"? How does that relate to a truck full of data?
- helmsb 3y agoThey first introduced a product called Snowball which is a big briefcase of hard drives that would ship you, you transfer up to a petabyte of data and send back. This was the extension of that so they went with another snow related name. Also their data archival service called Glacier which all imply "cold storage."