6 ms·

An old friend of mine is one of the authors on this. Their instruments generate a lot of data. I don't know if this is the case for their array in Australia, bu

An old friend of mine is one of the authors on this. Their instruments generate a lot of data. I don't know if this is the case for their array in Australia, but when he was working in South Africa they had to carry hard drives out of the country because there wasn't enough fiber capacity out of the continent.

lattalayta 6y ago
I recently found out that that is referred to as the sneakernet, and is actually still a preferred way to transfer a lot of data. https://en.wikipedia.org/wiki/Sneakernet https://en.wikipedia.org/wiki/Sneakernet
- EarthLaunch 6y ago
  AWS has a little-known service for trucks transporting 100PB of data in shipping containers: https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
  - amelius 6y ago
    "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." -- Andrew S. Tanenbaum (circa 1989)
  - wolco 6y ago
    What do their prices look like for 100 pb?
    - dexterdog 6y ago
      It's probably not bad compared to the monthly recurring you're about to get hit with for sitting in all of those bits.
    - jandrese 6y ago
      Certainly less than they would charge you for the bandwidth if you sent them over the wire.
  - carlosf 6y ago
    Damn. I once spoke with a friend who helped a museum in Rio de Janeiro to migrate its backups using AWS Snowball, but thats a whole new level. https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
    - hendiatris 6y ago
      Which one? IMS?
  - asib 6y ago
    I always wondered who would actually use this. Apparently a company called DigitalGlobe[0], which has a large collection of satellite images. [0]: https://www.networkworld.com/article/3154898/this-company-is-using-amazon-snowmobile-to-transfer-petabytes-of-data-to-the-cloud.html https://www.networkworld.com/article/3154898/this-company-is...
  - mr_toad 6y ago
    For when your forklift migration needs a literal forklift.
  - hyperpallium 6y ago
    In A Fire Upon the Deep, the Skroderiders' cargo was a one-time pad.
    - yetihehe 6y ago
      If you send someone several TB of random data on hard disk, you can now very securely send them several TB worth of encrypted data. It's many years of audio of reasonable quality, 1tb is about 72 days of raw 44.khz 16 bit stereo audio. With some speech codecs, that could be years of voice data. One time pad can be bigger than a notepad.
- mattkrause 6y ago
  My lab still does this (or, will do it when we’re finally allowed back in). We do a little bit of compression right on the instrument’s computer. After that, a briskly-walking postdoc can move a few terabytes a minute. The exact rate depends on whether they stop for coffee, but it always blows our internal 100 Mbps network out of the water.
codezero 6y ago
I was just leaving radio astronomy when the MWA was in later stages of design/deployment, I remember the discussions around designing the data storage for it, and it was centered around the idea of having to fly the data set back and forth (At the time, around 2008-2011ish?) - so it was so large that it wasn't really feasible to use the Internet for each data collection run. On the order of many terabytes if I recall correctly.
torrance 6y ago
The MWA has a direct fibre connection from site to a super computing facility in Perth. Although some processing (correlation and averaging) is usually done on site to reduce the data a little, there are some observation modes that send the raw voltages.
mhh__ 6y ago
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway" - A. Tanenbaum of MINIX (amongst other things) fame
- koheripbal 6y ago
  It's actually incredibly easy to beat the bandwidth of the internet with your car, with just a couple HDs. I routinely drive HDs full of backups a couple hours when going between DC sites.
  - deleted 6y ago
    [deleted]
  - alsobrsp 6y ago
    The latency will kill you.
    - nine_k 6y ago
      Not necessarily: the car may be there much sooner than the last byte sent via the fiber arrives. Not everything is streaming video; some datasets make most sense only as a whole. Backups are an example.
      - doublerebel 6y ago
        I think the subtext is, it is easily 1000x more dangerous personally to drive the data along public roads, than to send it over a network. The value in avoiding such danger is perhaps incalculable.
        throwaway_pdp09 6y ago
        Data can be encrypted to make it safe. Data will be duplicated onto disks before being driven away. It's not like there can only ever be one copy of the data.
codingdave 6y ago
That isn't as uncommon as you may think. Especially when talking about DR (disaster recovery) plans, sneakernet is certainly a valid way to get backup data from off-site because you never know what the network status will be when you need to enact your plan.
- deleted 6y ago
  [deleted]
deleted 6y ago
[deleted]
NikolaeVarius 6y ago
Event horizon telescope used HDD's being flown around to aggregate their data
duxup 6y ago
I worked for a company CERN had bought equipment from a while back. I was on the technical support team. I had a direct path to engineering when CERN called. The amount of data in an ultra short time was seemingly endless. So many "need more interfaces, more bandwidth...buffer tuning... buffer tuning". When a card burnt up mysteriously I made a sci fi joke about black holes and CERN in my ticket to engineering. It was the one and only time I got them to make a joke back ;)
- 8ytecoder 6y ago
  Now, it’s unfair to leave your audience hanging like that.
  - duxup 6y ago
    I wish I could remember the joke, but IIRC it was pretty terrible as I was just desperate to get something back from that engineering team that seemed like ... fun. Fortunately they either didn't care for, missed or just never acknowledged most of my puns and really vague references. Outside that one interaction the only thing I knew from them was that despite me being not the best technically compared to my peers, they asked I work specific tickets at times... but I think that was mostly because they knew I didn't make things up and would be honest and say "I don't know". Also I'd give them a heads up when I saw something hitting the fan to give them time to prepare ;)
- BrandoElFollito 6y ago
  I was doing my PhD when the LHC was being designed. It was interesting to listen to the discussions about signal and data management for the detections closest to the vertex. The rate was so incredible that the triggers were incredibly optimized to get rid of the noise and keep the fist wave of useful data (it was then processed several times down the road). The other interesting thing was to see how people were fighting for real estate in the detector, to be able to place their elements somewhere interesting. I crawled once under one of the detectors (ATLAS) and imagining the several thousand tons above me was frightening.
danans 6y ago
This is how Google Street view cars work too, because the sheer size of the high resolution photograpic data is so huge that it would make no sense to transmit it over a mobile network, vs just downloading it once the car finishes it's route.
- joering2 6y ago
  They could be using peoples wifis on their way to transfer the data as they dash thru the neighbour street. ;)

lattalayta 6y ago

I recently found out that that is referred to as the sneakernet, and is actually still a preferred way to transfer a lot of data. https://en.wikipedia.org/wiki/Sneakernet https://en.wikipedia.org/wiki/Sneakernet

EarthLaunch 6y ago
AWS has a little-known service for trucks transporting 100PB of data in shipping containers: https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
- amelius 6y ago
  "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." -- Andrew S. Tanenbaum (circa 1989)
- wolco 6y ago
  What do their prices look like for 100 pb?
  - dexterdog 6y ago
    It's probably not bad compared to the monthly recurring you're about to get hit with for sitting in all of those bits.
  - jandrese 6y ago
    Certainly less than they would charge you for the bandwidth if you sent them over the wire.
- carlosf 6y ago
  Damn. I once spoke with a friend who helped a museum in Rio de Janeiro to migrate its backups using AWS Snowball, but thats a whole new level. https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
  - hendiatris 6y ago
    Which one? IMS?
- asib 6y ago
  I always wondered who would actually use this. Apparently a company called DigitalGlobe[0], which has a large collection of satellite images. [0]: https://www.networkworld.com/article/3154898/this-company-is-using-amazon-snowmobile-to-transfer-petabytes-of-data-to-the-cloud.html https://www.networkworld.com/article/3154898/this-company-is...
- mr_toad 6y ago
  For when your forklift migration needs a literal forklift.
- hyperpallium 6y ago
  In A Fire Upon the Deep, the Skroderiders' cargo was a one-time pad.
  - yetihehe 6y ago
    If you send someone several TB of random data on hard disk, you can now very securely send them several TB worth of encrypted data. It's many years of audio of reasonable quality, 1tb is about 72 days of raw 44.khz 16 bit stereo audio. With some speech codecs, that could be years of voice data. One time pad can be bigger than a notepad.
mattkrause 6y ago
My lab still does this (or, will do it when we’re finally allowed back in). We do a little bit of compression right on the instrument’s computer. After that, a briskly-walking postdoc can move a few terabytes a minute. The exact rate depends on whether they stop for coffee, but it always blows our internal 100 Mbps network out of the water.

codezero 6y ago

I was just leaving radio astronomy when the MWA was in later stages of design/deployment, I remember the discussions around designing the data storage for it, and it was centered around the idea of having to fly the data set back and forth (At the time, around 2008-2011ish?) - so it was so large that it wasn't really feasible to use the Internet for each data collection run. On the order of many terabytes if I recall correctly.

torrance 6y ago

The MWA has a direct fibre connection from site to a super computing facility in Perth. Although some processing (correlation and averaging) is usually done on site to reduce the data a little, there are some observation modes that send the raw voltages.

mhh__ 6y ago

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway" - A. Tanenbaum of MINIX (amongst other things) fame

koheripbal 6y ago
It's actually incredibly easy to beat the bandwidth of the internet with your car, with just a couple HDs. I routinely drive HDs full of backups a couple hours when going between DC sites.
- deleted 6y ago
  [deleted]
- alsobrsp 6y ago
  The latency will kill you.
  - nine_k 6y ago
    Not necessarily: the car may be there much sooner than the last byte sent via the fiber arrives. Not everything is streaming video; some datasets make most sense only as a whole. Backups are an example.
    - doublerebel 6y ago
      I think the subtext is, it is easily 1000x more dangerous personally to drive the data along public roads, than to send it over a network. The value in avoiding such danger is perhaps incalculable.
      - throwaway_pdp09 6y ago
        Data can be encrypted to make it safe. Data will be duplicated onto disks before being driven away. It's not like there can only ever be one copy of the data.

codingdave 6y ago

That isn't as uncommon as you may think. Especially when talking about DR (disaster recovery) plans, sneakernet is certainly a valid way to get backup data from off-site because you never know what the network status will be when you need to enact your plan.

deleted 6y ago
[deleted]

deleted 6y ago

[deleted]

NikolaeVarius 6y ago

Event horizon telescope used HDD's being flown around to aggregate their data

duxup 6y ago

I worked for a company CERN had bought equipment from a while back. I was on the technical support team. I had a direct path to engineering when CERN called. The amount of data in an ultra short time was seemingly endless. So many "need more interfaces, more bandwidth...buffer tuning... buffer tuning". When a card burnt up mysteriously I made a sci fi joke about black holes and CERN in my ticket to engineering. It was the one and only time I got them to make a joke back ;)

8ytecoder 6y ago
Now, it’s unfair to leave your audience hanging like that.
- duxup 6y ago
  I wish I could remember the joke, but IIRC it was pretty terrible as I was just desperate to get something back from that engineering team that seemed like ... fun. Fortunately they either didn't care for, missed or just never acknowledged most of my puns and really vague references. Outside that one interaction the only thing I knew from them was that despite me being not the best technically compared to my peers, they asked I work specific tickets at times... but I think that was mostly because they knew I didn't make things up and would be honest and say "I don't know". Also I'd give them a heads up when I saw something hitting the fan to give them time to prepare ;)
BrandoElFollito 6y ago
I was doing my PhD when the LHC was being designed. It was interesting to listen to the discussions about signal and data management for the detections closest to the vertex. The rate was so incredible that the triggers were incredibly optimized to get rid of the noise and keep the fist wave of useful data (it was then processed several times down the road). The other interesting thing was to see how people were fighting for real estate in the detector, to be able to place their elements somewhere interesting. I crawled once under one of the detectors (ATLAS) and imagining the several thousand tons above me was frightening.

danans 6y ago

This is how Google Street view cars work too, because the sheer size of the high resolution photograpic data is so huge that it would make no sense to transmit it over a mobile network, vs just downloading it once the car finishes it's route.

joering2 6y ago
They could be using peoples wifis on their way to transfer the data as they dash thru the neighbour street. ;)