6 ms·
Bypass the internet for large transfers: AWS Import/Export
- wheels 17y agoObligatory Tanenbaum quote: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
- jeffbarr 17y agoIndeed, so obligatory that I included it in my blog post: http://aws.typepad.com/aws/2009/05/send-us-that-data.html http://aws.typepad.com/aws/2009/05/send-us-that-data.html
- jimbokun 17y agoKudos (to all of you at Amazon) for being the first to turn that quote into a product feature. Certainly seems obvious in retrospect.
- Skeuomorph 17y agoOn the contrary, video delivery networks (VDNs) have been recommending loading data this way for years, such as one that's routinely internationally FedEx-ing those cheap 2TB "network" drives back and forth in small white mil-spec shipping cases.
- jjs 17y agoSometimes a big truck has more bandwidth than a series of tubes...
- eugenejen 17y agoBandwidth and latency are two different things. But it seems we are so often confused by these two concepts. I feel funny because Werner Vogels is Andy Tannenbaum's phd student. That's how a concept propagates from one generation to next.
- tophat02 17y agoTerrible latency, though.
- alexkay 17y agoDupe: http://news.ycombinator.com/item?id=620089 http://news.ycombinator.com/item?id=620089
- RiderOfGiraffes 17y agoAlthough this one is getting the comments.
- boundlessdreamz 17y agoProbably because of a better title giving people an idea about the service even before they read the article.
- alexkay 17y agoThe other earlier submission has been deleted and I'm being downmodded. Fair enough, I guess I won't be reporting duplicates any more.
- drawkbox 17y agoNew cloud feature: Pony Express Only kidding this is great for immense datasets.
- stevejalim 17y ago<lamejoke> Presumably a feature for Django devs</lamejoke>
- dfield 17y agoGoogle had a neat feature related to this in implementation but not goal since 2007ish. It was called Google Datasets and let you mail in a hard drive. Google would then host the data on the web. It's audience was purely academic, though... no money exchange involved. Anyway, it got cut back last Fall. Cool to see it's still living on in some form.
- danw 17y agoOh that's a shame, I didn't know it had been cut. Amazon has a similar scheme "Public Datasets", that store data as an EBS rather than an S3 bucket http://aws.amazon.com/publicdatasets/ http://aws.amazon.com/publicdatasets/
- katamole 17y agoNot really "AWS" then!
- pierrefar 17y agoHow do they do virus and security checking? I don't know how they import the data, but it could be that they use a system that can be exploited using a security vulnerability? It's a (very) long shot for this kind of thing to work, but still it's a risk worth asking about. They are at the end of they day hooking up hardware to their computer - i.e. physical access to the machine.
- cracki 17y agowould you risk your account by intentionally adding malware to your data sets? HDDs are passive, there is no danger in merely hooking one up. and as long as the file system drivers are solid, shoveling these bits into "buckets" (whatever these are) is safe too. you're just shoveling the bits, not looking at them. this is about data, not executable code.
- loop 17y agoThey only support SATA and USB-2, which unlike Firewire, do not work by DMA, IIRC. I'd assume they do not support Firewire for exactly this reason: to prevent a device from reading the systems memory directly (known as pwn-by-iPod)
- tsuraan 17y agoSATA certainly does support DMA. DMA isn't a feature unique to firewire; rather, USB is (nearly) unique in that it's the one high-speed standard that doesn't support DMA.
- mtkd 17y agoWhen I read the announcement this morning, it occurred to me that the reason I like AWS so much, and what separates them from other cloud offerings, is that they're doing stuff dev teams actually want and can use (not impenetrably complex tech we could use with some vendor consultancy). Import features like this are trivial for AWS to implement - but will be a step change for some customers. Last generation vendors saying 'anyone can do what AWS are doing' is just crap. You can't buy the enthusiasm these guys have - and its not just one individual - if you talk to anyone from AWS (and they're often around nights/weekends to talk about stuff) they are in a different lane to the competition or vendors that should be the competition.
- bravura 17y agoAgreed, except I have found it pretty tough to figure out EC2. EC2 is fairly impenetrable at first.
- kiwidrew 17y agoLiving in a country that's stuck on the end of an expensive and slow(ish) undersea cable for the past few years, sometimes my thoughts wander towards something like this. Even just a service (US-based) that would download a set of files (HTTP, FTP, bittorrent), burn them to DVD, and post them via airmail would be pretty handy. Bonus points if there was a way to use airline passengers as your couriers, such that the total latency might only be ~24 hours. :)
- TJensen 17y agoThe airlines have services for that (see Delta Dash) without needing to use airline passengers. :)
- nickb 17y agoAbout 3 months ago, I needed >50Gb of data uploaded to AWS and I asked a friend of mine who works at Amazon for help. He asked around and told me to mail the external storage to Amazon with keys/bucket info and a special code. I UPSed the stuff and within 2 days of arrival, I received an email with a notification that the transfer was done. Later on, I got the storage back. Everything worked as advertised! I guess there's so much demand for this sneakernet that they made an official service. I wonder how many other people asked for transfer help before today's announcement...
- siculars 17y agohmm... maybe i'll sneakernet all my music/photos now. been waiting for a sneakernet uploading service. it was only a matter of time before they added one. wonder how long before smugmug adds a wraper around this and lets you send your dvd/smartcard/etc media and get it directly uploaded into your account.
- callmeed 17y agoThis is going to be great for professional photographers (the industry I work in). I'm been advocating they use S3 for archiving for some time ... now it's easier to get your files there.
- wmf 17y agoAlso check out Terascale Sneakernet from Jim Gray et al. from 2002: http://research.microsoft.com/apps/pubs/default.aspx?id=64570 http://research.microsoft.com/apps/pubs/default.aspx?id=6457...