9 ms·
Using Amazon Snowmobile to transmit 100PB of satellite images to AWS
- Piskvorrr 9y agoEverything old is new again: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." —Tanenbaum, Andrew S. (1989). Computer Networks. New Jersey: Prentice-Hall. p. 57. ISBN 0-13-166836-6.
- Posibyte 9y agoIt really is a marvel when you think about it. The latency is horrible, several hours to several days, but each packet of data can contain several Pb of information inside of it. It comes with a built-in ACK method (phone call or site update), and the packet almost never needs to be verified externally because the data inside has all of its own integrity check methods.
- gruez 9y ago>It comes with a built-in ACK method (phone call or site update) Not really built in if it's out of band.
- JackFr 9y agoI assume they're sending the checksums with Bandit (lest a truck hijacking lead to a MITM attack.)
- ajford 9y agoThis sounds like the lead in to a fantastic spy movie. The rag-tag team of ex-spies have to band together with the younger techy team to hijack a Snowmobile and modify data on the fly, barreling down the highway.
- chiph 9y agoToo bad Pontiac is out of business. I suppose we could use a Camaro, as long as it's black, with T-Tops, and has a "screaming chicken" decal on the hood. Sadly, Burt Reynolds isn't available to drive (he needs assistance to get around these days)
- monocasa 9y agoMaybe we can make a ML model of Burt Reynolds?
- Bedon292 9y agoIf I recall the talk at re:Invent right, they offer guards as an optional feature you can add during travel. "Snowmobile is protected by 24/7 video surveillance and alarm monitoring, GPS tracking, and may be escorted by a security vehicle during transit." According to the FAQ.
- JackFr 9y agoI don't want to give away the plot completely, but for the truck to leave Manahttan, they go through the Lincoln Tunnel. In decent traffic, that will give me 5 minutes to separate/overwhelm guard cars, as well as hijack the truck en route. With my large team of crack drivers and action heroes, a bunch of panel trucks and vans can obstruct the view of all the civilians driving in the tunnel. By the time we're outside, the GPS is now in my truck, the driver and data of the real truck are under my control. (The most unrealistic part of this scenario is that someone would have their main data center in Manhattan.)
- tomtoise 9y agoAt University we had a question around this type of 'transfer'. I can't remember how it went exactly but we were to calculate the bandwidth of a jumbo plane carrying x amount of hard drives with y capacity to z destination. Given that the flight takes so many hours, what is the total transfer speed of data in seconds? I saw a few students nearly have a breakdown over that one.
- kingbirdy 9y agoWhy? It's just x * y / time
- tomtoise 9y agoYes, to those who had studied it was fairly obvious what the solution was, but the phrasing was such that to those who were just trying to wing it failed miserably.
- cmdrfred 9y agoI can solve that problem and I dropped out of high school. Simple math is something that has to be studied for at the college level?
- tomtoise 9y agoYes - the first year of University was designed as a catch-up to those who hadn't previously had exposure to this sort of thing and weed out the coasters. I must stress this was a single question on a single first-year exam. Not representative of the whole course in the slightest.
- acchow 9y agoYou just need to get the units to line up. You want bytes/seconds (i.e. quantity of data per unit time). You have: - Distance per unit time (km/h) - Payloads per unit distance (# of flights per km) - Quantity of data per payload (bytes per flight)
- gist 9y agoI am still trying to figure out if the story that I read in the 80's pointing out the need to always test backup tapes that was reinforced by saying some guy in Sweden erased them by putting on a heated car seat in the Winter was true or just a way to drive home a point. (I have never forgotten that story but can't find anything on the web about it today at least.)
- Piskvorrr 9y agoThe point wasn't in the heat - but that it was heated electrically, with the requisite magnetic field causing the erasure.
- thatwebdude 9y ago(Regarding a featured photo) Man, I really gotta find a way to invest in shipping containers. It's foolproof as of late!
- comboy 9y agoI wonder when we'll achieve communication faster than moving physical medium for huge amounts of data. It's not so obvious since information density also keeps increasing.
- mjb 9y agoThe historical trend is that storage density grows faster than throughput, and throughput improves faster than latency.
- gm-conspiracy 9y agoSomething video-streaming services and media companies should consider.
- spacemanmatt 9y agoMaybe they did/do? I was surprised and skeptical at some point, about all the heat around video streaming. Codecs and speed improvements have made me a believer.
- ryandamm 9y agoPerhaps we can encode high quality video onto physical media... using microscale structures to _optically_ encode the video data? We could then use snail mail to move these optical storage devices to end viewers. Oh, wait. Darnit.
- gm-conspiracy 9y agoThing bigger. Instead of a movie or few episodes, consider physical media that could store a year or decade of TV shows and movies. What is more likely to occur in the next decade: increasing density and decreasing costs of storage media, OR, increased speeds at the local ISP level with decreasing costs?
- Symbiote 9y ago
- acidburnNSA 9y agoAs we approach Enemy of the State technology, I feel slightly better about how cloudy it is in Seattle.
- JorgeGT 9y agoSAR (Synthetic aperture radar) spy sats have been a thing for decades, and they can see through clouds: https://en.wikipedia.org/wiki/Lacrosse_(satellite) https://en.wikipedia.org/wiki/Lacrosse_(satellite)
- zardo 9y agoThe clouds don't really help with the cameras on street posts. http://www.seattletimes.com/seattle-news/crime/judge-blocks-seattle-city-light-from-disclosing-locations-of-fbi-surveillance-cameras/ http://www.seattletimes.com/seattle-news/crime/judge-blocks-...
- j_s 9y agoI believe the place you need to be is out on (or in/under?) the ocean somewhere, per Malaysia Airlines MH370. I couldn't find the discussion but if I remember correctly most satellites shut down over a big chunk of the Indian Ocean because there's nothing there.
- wging 9y agoMake sure to read the linked blog post from Digital Globe. It is more interesting, in my opinion (it includes some interesting details and seems to be written towards a slightly more technical audience): http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/ http://blog.digitalglobe.com/industry/digitalglobe-moves-to-...
- ortusdux 9y agoWow. It looks like a 16hr drive to the Oregon center. Given an 8hr break in between shifts it works out to almost 10 Tb/sec transfer rate.
- deno 9y agoUnless they’re shipping those disks as a ready-to-go container DC[1], you need to account for the time to unload and install them. Actually, you need to account for it on both ends. [1] https://en.wikipedia.org/wiki/Modular_data_center https://en.wikipedia.org/wiki/Modular_data_center
- kristjansson 9y agoThey are though! > The Snowmobile comes with a removable connector rack that needs to be mounted on one of your data center racks where it can be connected directly to your high-speed network backbone. The connector racks provides multiple 40Gb/s interfaces that can transfer up to 1 Tb/s in aggregate.[1] so there is a bit of setup and teardown, but they're not plugging disks in one-by-oen [1]: https://aws.amazon.com/snowmobile/faqs/ https://aws.amazon.com/snowmobile/faqs/
- Twirrim 9y agoThat's talking about customer premises though, not AWS side.
- jsmthrowaway 9y agoWhat's the difference? They'd probably do the same thing at both ends. Snowmobile is essentially a giant external drive hurtling down the highway, and the removable rack (which I'd assume is just switches) is its "USB" interface. Actually, it'd probably be more efficient on the AWS side, because they likely already have said interconnects in their facility and don't need to unload it. If I were designing it I'd expose a private S3 interface, or maybe NFS, to the customer from the truck, store the incoming bits on-disk in the form that S3 expects, then just copy raw blocks from disk and merge metadata databases on the AWS side so you're not doing two passes through the S3 API. I can't imagine they're doing something much different than that. Not every customer would be in a position to handle iSCSI or FC, for example. You could unload the disks at the destination, too, but that seems less efficient and more disruptive to their lifetime.
- YCode 9y agoThe Sneakernet has its advantages.
- dghughes 9y agoI have some security video on four 1TB USB 2 external drives I added the files to the drives a few files at a time over a year or two. I wanted to move all the files all onto a single large drive but I just gave up due to the drives spinning for so long (days!) and its power is over the USB cable not external brick. I guess that's the point of the article the data we humans can now create is so enormous it's impossible to move it other than in bulk physically.
- dsp1234 9y agoTake the external drives out of the USB 2 enclosure (max speed 35MB/s) and move them into a USB 3 enclosure (max speed 400MB/s). You can also directly attach the drives to a PC via an IDE or SATA interface. The USB 3, IDE or SATA interfaces are going to allow you to run the drive at whatever the drive's speed is, rather than be limited by the transfer speed of the interface.
- dghughes 9y agoA good idea but these are 12 year old Iomega portable external drives I'm sure I could try that but they may not survive the operation.
- brianwawok 9y agoOlder spinning rust may not be much faster than 35 MB a sec sustained
- Piskvorrr 9y agoNah. It is possible still, but it's far cheaper this way (in person/time and S3 bandwidth costs).