4 ms·

From the comments: > Time for a data transfer startup based on pre-programmed drones with relay terminals to recharge or change carriers. I know it's a joke,

by kypro 3y ago

From the comments: > Time for a data transfer startup based on pre-programmed drones with relay terminals to recharge or change carriers. I know it's a joke, but I'd be interested in knowing if anyone knows of any theoretical or known commercial use cases of exactly this? If you're a corporate with a petabyte of data you want to send from the US to Europe to do some data analytics on, would you not back it up to tape and load it on a plane? Seems like this must happen from time to time, no?

mhink 3y ago
For terabyte-scale data it might make sense to send it along with a person, but for larger amounts there are definitely services to help handle the process. He mentioned AWS Snowmobile in the video, but they also have a lower-volume equivalent: AWS Snowball [1]. It's basically just a suitcase-sized device that you plug in, upload data to, and ship to the destination. They have some really nice features to help make the process smooth. But yeah, at its core the idea really is "pack up a lot of hard drives and have it shipped to the destination". 1: https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
- dr_orpheus 3y ago
  Also reminds me of the XKCD What-if: "When - if ever - will the bandwidth of the Internet surpass that of FedEx" https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
  - GauntletWizard 3y ago
    What-ifs really need to be dated. Some of those assumptions are enough to date it; 160tb/kg is kind of laughable now, when talking about micro SD cards. Reasonably guessable using https://colin-scott.github.io/personal_website/research/interactive_latency.html https://colin-scott.github.io/personal_website/research/inte...
gwill 3y ago
i can't find a source for it, but i know nasa has done this in the past.
dr_orpheus 3y ago
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." –Andrew Tanenbaum, 1981
- MichaelZuo 3y ago
  Nowadays you can buy 30TB 2.5 inch SSDs which includes a durable metal housing. So a station wagon could feasibly transport close to 300 petabytes. Though that would be close to a quarter billion dollars worth of SSDs.
  - aziaziazi 3y ago
    Are ssd the best alternatives regarding price/bite and volume/bite ? It seems and SDUC card will largely beat on the volume comparison.
    - Beached 3y ago
      it's not commercially available yet, but I imagine 5d storage would be best data density.
    - MichaelZuo 3y ago
      Not if you expect to actually get data out of them after hurtling down the road. Even for top end enterprise grade SSDs made at the highest standard, I would only assign a <99.9% chance of doing so. So that's 10 bad SSDs per trip at least.
      - biomcgary 3y ago
        Erasure codes are your friend, if you have the compute.
macintux 3y ago
Not quite the same, but Peter Jackson famously used iPods to transport dailies while filming LoTR.
fdwlbs 3y ago
I suppose it mostly depends on how much the saved time is worth and how that compares to the cost of transport. I suppose there are some cases where shipping a box of drives makes sense. I think a much more common case is having a critical machine crash and needing a backup sent to you. Depending on the bandwidth you actually have, restore by mail can be faster. And that's why companies like backblaze offer it. https://www.backblaze.com/cloud-backup/features/restore https://www.backblaze.com/cloud-backup/features/restore Restoring 8TB, which is not an unreasonable amount of data in 2023, at 40Mb/s, which is not that uncommon in much of America, still takes more than 2 days when everything goes perfectly. FedEx'ing a 8TB drive should be able to beat that in many cases.
LakshyAAAgrawal 3y ago
I introduce to you https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
- dotancohen 3y ago
  Use case, really interesting: https://blog.maxar.com/earth-intelligence/2017/digitalglobe-moves-to-the-cloud-with-aws-snowmobile https://blog.maxar.com/earth-intelligence/2017/digitalglobe-...
OmarShehata 3y ago
This isn't very uncommon in geospatial companies, where data is often hundreds of terabytes. I remember there was open government data you could request from USGS (high res satellite imagery of all of the US), it was technically free, but you couldn't download it. You requested it, filled out a form, and they shipped you 4-5 drives with bubblewrap.
natemcintosh 3y ago
At my previous job we had to get data off a ship after a test. The upload rate off the ship was slow. The solution was to put the data on drives, put them in a helicopter, and fly them to land.
Simon_O_Rourke 3y ago
Ditto in my previous job, there were weekly package deliveries of hard drives from Europe, which were a magnitude faster (then) compared to what was possible for a reasonable price... Of course this was all back in the day, and might only be economical these days from somewhere with major airports but no connecting fibre.
teekert 3y ago
In the genomics field this is why we are working on making “data visiting” easier: https://www.ga4gh.org/product/workflow-execution-service-wes/ https://www.ga4gh.org/product/workflow-execution-service-wes...

by kypro 3y ago

mhink 3y ago

For terabyte-scale data it might make sense to send it along with a person, but for larger amounts there are definitely services to help handle the process. He mentioned AWS Snowmobile in the video, but they also have a lower-volume equivalent: AWS Snowball [1]. It's basically just a suitcase-sized device that you plug in, upload data to, and ship to the destination. They have some really nice features to help make the process smooth. But yeah, at its core the idea really is "pack up a lot of hard drives and have it shipped to the destination". 1: https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/

dr_orpheus 3y ago
Also reminds me of the XKCD What-if: "When - if ever - will the bandwidth of the Internet surpass that of FedEx" https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- GauntletWizard 3y ago
  What-ifs really need to be dated. Some of those assumptions are enough to date it; 160tb/kg is kind of laughable now, when talking about micro SD cards. Reasonably guessable using https://colin-scott.github.io/personal_website/research/interactive_latency.html https://colin-scott.github.io/personal_website/research/inte...

gwill 3y ago

i can't find a source for it, but i know nasa has done this in the past.

dr_orpheus 3y ago

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." –Andrew Tanenbaum, 1981

MichaelZuo 3y ago
Nowadays you can buy 30TB 2.5 inch SSDs which includes a durable metal housing. So a station wagon could feasibly transport close to 300 petabytes. Though that would be close to a quarter billion dollars worth of SSDs.
- aziaziazi 3y ago
  Are ssd the best alternatives regarding price/bite and volume/bite ? It seems and SDUC card will largely beat on the volume comparison.
  - Beached 3y ago
    it's not commercially available yet, but I imagine 5d storage would be best data density.
  - MichaelZuo 3y ago
    Not if you expect to actually get data out of them after hurtling down the road. Even for top end enterprise grade SSDs made at the highest standard, I would only assign a <99.9% chance of doing so. So that's 10 bad SSDs per trip at least.
    - biomcgary 3y ago
      Erasure codes are your friend, if you have the compute.

macintux 3y ago

Not quite the same, but Peter Jackson famously used iPods to transport dailies while filming LoTR.

fdwlbs 3y ago

I suppose it mostly depends on how much the saved time is worth and how that compares to the cost of transport. I suppose there are some cases where shipping a box of drives makes sense. I think a much more common case is having a critical machine crash and needing a backup sent to you. Depending on the bandwidth you actually have, restore by mail can be faster. And that's why companies like backblaze offer it. https://www.backblaze.com/cloud-backup/features/restore https://www.backblaze.com/cloud-backup/features/restore Restoring 8TB, which is not an unreasonable amount of data in 2023, at 40Mb/s, which is not that uncommon in much of America, still takes more than 2 days when everything goes perfectly. FedEx'ing a 8TB drive should be able to beat that in many cases.

LakshyAAAgrawal 3y ago

I introduce to you https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/

dotancohen 3y ago
Use case, really interesting: https://blog.maxar.com/earth-intelligence/2017/digitalglobe-moves-to-the-cloud-with-aws-snowmobile https://blog.maxar.com/earth-intelligence/2017/digitalglobe-...

OmarShehata 3y ago

This isn't very uncommon in geospatial companies, where data is often hundreds of terabytes. I remember there was open government data you could request from USGS (high res satellite imagery of all of the US), it was technically free, but you couldn't download it. You requested it, filled out a form, and they shipped you 4-5 drives with bubblewrap.

natemcintosh 3y ago

At my previous job we had to get data off a ship after a test. The upload rate off the ship was slow. The solution was to put the data on drives, put them in a helicopter, and fly them to land.

Simon_O_Rourke 3y ago

Ditto in my previous job, there were weekly package deliveries of hard drives from Europe, which were a magnitude faster (then) compared to what was possible for a reasonable price... Of course this was all back in the day, and might only be economical these days from somewhere with major airports but no connecting fibre.

teekert 3y ago

In the genomics field this is why we are working on making “data visiting” easier: https://www.ga4gh.org/product/workflow-execution-service-wes/ https://www.ga4gh.org/product/workflow-execution-service-wes...