8 ms·
For a sense of scale, my company used an AWS Snowmobile to transfer 100PB of archived satellite imagery to AWS. More details here: https://blog.maxar.com/earth
by ThisIsTheWay 6y ago
For a sense of scale, my company used an AWS Snowmobile to transfer 100PB of archived satellite imagery to AWS.
More details here: https://blog.maxar.com/earth-intelligence/2017/digitalglobe-moves-to-the-cloud-with-aws-snowmobile https://blog.maxar.com/earth-intelligence/2017/digitalglobe-...
- obviyus 6y agoThat was an excellent read. Thanks for sharing.
- dcolkitt 6y ago"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
- downerending 6y agoOr these days, micro flash cards.
- ThisIsTheWay 6y agoFor real! Reminds me of this article... "A carrier pigeon with a cargo of microSD cards can transfer large amounts of data faster, and more cheaply, than just about any other method" Link: https://spectrum.ieee.org/tech-talk/computing/networks/pigeonbased-feathernet-still-wingsdown-fastest-way-of-transferring-lots-of-data https://spectrum.ieee.org/tech-talk/computing/networks/pigeo...
- firloop 6y agoThis almost got me and then I looked at the date of publication.
- CobrastanJorji 6y agoIf you're new to joke RFCs, I suggest reading up on RFC 3514, a suggestion that malicious IP packets should be required to set the high order bit of the IP fragment offset field, which is to be designated the "evil" bit. This significantly simplifies implementing effective network monitoring.
- bufferoverflow 6y agoIt's still true. A pigeon can easily carry 10 microSD cards, 1TB each. Even if you're on a gigabit connection, it will take you over 22 hours to copy 10TB.
- reitzensteinm 6y agoThey meant that it was posted on April fools, not that it's an old article.
- ramraj07 6y agoWhat was the premium of storing the data in S3 as opposed to keeping it in your own tape warehouse? Definitely curious about the cost benefits!
- ThisIsTheWay 6y agoUnfortunately I don't have access to that info, sorry!
- scarface74 6y agoIs your tape library stored redundantly across multiple data centers?
- voxic11 6y agoS3 Glacier Deep Archive is probably the closest to a tape warehouse, at the standard rate it would cost $99000/month to store 100PB of data. $0.00099 per GB per month * 100,000,000 GB = $99000
- a012 6y agoYou need to account data transfer OUT if you need to get your data from S3. _If_ you could download 100PB data, it'd cost you $1M.
- TuringNYC 6y agoHopefully it is not stored as one giant object and hopefully your retrieval would be only specific parts (small parts.)
- milesvp 6y agoYou’d want to do some research on getting data out of glacier. I’ve read some nightmare stories about the bill from trying to get a small amount of data out of glacier. I can’t remember the details, and some of the large cost was people sort of doing it wrong. But some of the tooling makes it easy to do wrong. When I was using glacier, we treated it sort of like a write only storage, with the understanding that if we ever needed data out, we’d have to think long and hard about the data’s value, and probably would have gotten an AWS rep on the line to make sure we didn’t do anything stupid.
- shawn-butler 6y agoJust FYI your company has a cert mismatch on the following link from that blog post: https://platform.digitalglobe.com/gbdx/ https://platform.digitalglobe.com/gbdx/ The cert is limited to *.wpengine.com And the link 404s so.. that's a shame. Was interested in learning more about the platform.
- ThisIsTheWay 6y agoThanks for the heads up! I'll notify the team. You can learn more here: https://www.digitalglobe.com/products/gbdx https://www.digitalglobe.com/products/gbdx
- sanguy 6y agoLike everything from DigitalGlobe -- it doesn't work.
- ThisIsTheWay 6y agoLOL. What products have you had bad experiences with?
- ies7 6y agoFor a sense of the world, I went to Manila in 2018 to find out why there are several distributor can only sent their data weekly. One of the distributor show me a 64 MB usb flashdisk that’s travelled every week with the delivery truck to get 4 MB of data from their branches (8 hours truck driving from the capital) Went to Myanmar in 2017. The nation is just upgrading to 4G in 2016. But not every places have 4G coverage 2019 in eastern Indonesia (Seram island) you’re lucky if you can get a reliable 5 KB/s internet. It’s a remote area, but we still have a branch there :(
- tener 6y agoWhy not satellite internet then?
- PedroBatista 6y ago$$$