7 ms·

Introducing Transfer Appliance: Sneakernet for the cloud era

95 points by nealmueller 9y ago

milesward 9y ago
1 petabyte compressed! Woot!
tpetry 9y ago
The ability to export data from GCP to maybe your own datacenter is really missing.
- jdubs 9y ago
  Hotel California model.
  - nealmueller 9y ago
    One of my favorite songs. Doesn't apply here though. Iron Mountain has a service today to move data out of GCP.
    - lmartel 9y ago
      I don't know if I'd be comfortable relying on a third party for data export.
- nealmueller 9y ago
  Good idea, thanks for the suggestion. It does seem more open for the appliance to ship data to the cloud and also the reverse. The interest is interesting, and I'll definitely feed that back.
  - tpetry 9y ago
    Shipping Petabytes to a cloud you can practically never again retrieve all the data is ... risky. The aws way of doing this is nice: You rent the appliance like when importing data to the cloud and pay a (reduced) "bandwidth" fee for exporting the data to the appliance which will be shipped to you.
  - gist 9y ago
    Same issue. Another idea would be for GCP to handle export to a physical device and that device would be shipped to a known location (to both GCP and the client) and accessible by either but offline and cold stored. And similar to how paper records are stored you could put in a request to access the records and thereby verify their existence (which you can't do with cloud data other than the copy that you are able to access). Fwiw, Iron Mountain lost about 60 physical records file boxes that we had ..
- johansch 9y ago
  In the same theme: moving very large amounts of data between GC and AWS, in either direction. I wonder what it would take for them to make this cost zero. Legislation? It likely wouldn't be very expensive to get a very high-capacity fiber connection going between GC/AWS datacenters in relative physical proximity. Not expressing a particular/immediate personal need here, just noting that this could help keep the lock-in factor down and level of competition up.
- deleted 9y ago
  [deleted]
jrowley 9y ago
This a bit tangental but I like the graphs/visualizations they used in this ad. Easy to read and interrupt
- boulos 9y ago
  Agreed! The diagram [1] is super handy, and lets me reiterate why we waited to so long to do this: 10 Gbps of peering is just not that rare (and you probably want it for updates, etc. anyway). Even as you get to the petabyte range, being able to just hit "Go" and then do differential updates is so valuable that you really have to be talking about lots of Petabytes in a location where (or reason why) you can't get 10Gbps plus of peering. Disclosure: I work on Google Cloud (but didn't work on this) [1] https://3.bp.blogspot.com/-SnFabcStXhM/WW4SEhj6adI/AAAAAAAAEIE/wd6Yqlbt9FELUxsvTc2F92GZDZq2I-_6QCLcBGAs/s1600/transfer-appliance-2.png https://3.bp.blogspot.com/-SnFabcStXhM/WW4SEhj6adI/AAAAAAAAE...
  - vgt 9y ago
    To add to Boulos point - Google Cloud has the "Cloud Transfer Service" [0] that's hugely popular. It's easy to get connectivity to Google's vast network of POPS and this service helps take advantage of that connectivity. Also S3. (work at G but not in any of these areas) https://cloud.google.com/storage/transfer/ https://cloud.google.com/storage/transfer/
bpicolo 9y ago
To be fair, AWS launched a much, much larger analogue to AWS Snowball. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
- nealmueller 9y ago
  Hahaha, you got me there. Althought, to be fair, Snowmobile is an entirely different product category. Same problem, different scale of solution. It's like Starbucks coming out with a truck solution, when all you wanted was a Venti.
  - bpicolo 9y ago
    Espresso trucks sound awesome.
    - dwyerm 9y ago
      Interestingly, Starbucks does have both a trailer solution and a shipping-container solution. The former is parked out front when the actual store is undergoing refurbishment. The latter is seemingly air-dropped into neighborhoods like Northglenn, CO and Ballard, WA. Coincidentally, Starbuck's first shipping container store is just a mile or two from Amazon's first use of the Snowmobile.
- Jyaif 9y ago
  "Snowmobile uses multiple layers of security designed to protect your data including dedicated security personnel, GPS tracking, alarm monitoring, 24/7 video surveillance, and an optional escort security vehicle" The data is encrypted, so that part is really just to impress the commoner.
  - losteric 9y ago
    I assume these services were originally created to meet government contracting requirements (transporting highly classified information).
CaliforniaKarl 9y ago
Awwwww, it's too bad this can't be used to get stuff directly into Drive.
- nealmueller 9y ago
  Good idea, thanks for the suggestion. The interest is interesting, and I'll definitely feed that back. I felt the pain recently myself when I moved all my photos and videos to Google Photos and I wished I had one of these.
binaryblitz 9y ago
$300 for a 100TB rackable drive? Can I just keep it instead?
- Veratyr 9y ago
  It seems to pretty clearly be a rental, so no...
- canes123456 9y ago
  Obviously not. Rent a car for a week seems cheap. Can I keep it forever?
- nealmueller 9y ago
  The other comments here are correct. The standard model for these appliances is a rental model. Please talk to a GCP seller.
- trevyn 9y ago
  $30 per day after 10 day grace period: https://cloud.google.com/data-transfer/pricing https://cloud.google.com/data-transfer/pricing It would be funnier if you could keep it indefinitely, but reads were $0.12/GB. ;) I wonder if that's a viable business model...
  - shiftpgdn 9y ago
    Nimble offers great local storage on a pay per month per use model. I'm sure HP is working hard to ruin this.
- binaryblitz 9y ago
  Good god... It was a joke...
westernmostcoy 9y ago
Article title is actually "Introducing Transfer Appliance: Sneakernet for the cloud era", can someone change how it's presented here? This was not a good improvised title.
- sctb 9y ago
  Yes, thank you. We've updated the title from “Google launches a larger analogue to AWS Snowball”.
twakefield 9y ago
"Like many organizations we talk to, you probably have large amounts of data that you want to use to train machine learning models." I understand's Google's bias here but doesn't it usually make more sense to bring the programs/models to where the data already is?
- bpicolo 9y ago
  The problem isn't the programs, it's the hardware. GCP has specialized / scalable hardware to build models on.
- zeroxfe 9y ago
  It is extremely expensive to setup and maintain infrastructure to process big data, especially when you get into the petabytes.
- btian 9y ago
  GCP has proprietary machine learning accelerator hardware that you can't buy.
wtvanhest 9y ago
Anyone know how to backup 1tbyte worth of images in google or AWS? My ISP throttles my upload time. I'm interested in doing it at a lower price point than $500. Is that possible?
- theDoug 9y ago
  I've happily used Arq (https://www.arqbackup.com/ https://www.arqbackup.com/) to get a lot of things into Drive and GCP. You can control your network rate to still get the files uploaded but over a longer time-span below your ISP's throttling. (Disclosure: I work on GCP but this is a personal, not professional, endorsement)
  - grogenaut 9y ago
    it's multi cloud so I think you're safe from showing a bias.
- rb808 9y ago
  I was thinking of buying a NAS just for this. It would take a while, but would be running all night which is handy. cheapest synology is $110 new + disk. https://www.synology.com/en-uk/knowledgebase/DSM/help/CloudSync/cloudsync https://www.synology.com/en-uk/knowledgebase/DSM/help/CloudS...
- wmf 9y ago
  https://awsimportexport.s3.amazonaws.com/aws-import-export-calculator.html https://awsimportexport.s3.amazonaws.com/aws-import-export-c... says ~$90.
  - rb808 9y ago
    thanks I hadn't seen that price, looks handy
- rb808 9y ago
  Try this too: S3 Transfer Acceleration, maybe gets around ISP throttling (I haven't tried this yet) https://aws.amazon.com/s3/faqs/#s3ta https://aws.amazon.com/s3/faqs/#s3ta
peterjlee 9y ago
So a Hooli Box?
- compuguy 9y ago
  Hooli Box compresses and stores data. This is just used to transfer data to GCP.
  - brianwawok 9y ago
    This compresses.
Poiesis 9y ago
Anyone know what's inside the box?

95 points by nealmueller 9y ago

milesward 9y ago

1 petabyte compressed! Woot!

tpetry 9y ago

The ability to export data from GCP to maybe your own datacenter is really missing.

jdubs 9y ago
Hotel California model.
- nealmueller 9y ago
  One of my favorite songs. Doesn't apply here though. Iron Mountain has a service today to move data out of GCP.
  - lmartel 9y ago
    I don't know if I'd be comfortable relying on a third party for data export.
nealmueller 9y ago
Good idea, thanks for the suggestion. It does seem more open for the appliance to ship data to the cloud and also the reverse. The interest is interesting, and I'll definitely feed that back.
- tpetry 9y ago
  Shipping Petabytes to a cloud you can practically never again retrieve all the data is ... risky. The aws way of doing this is nice: You rent the appliance like when importing data to the cloud and pay a (reduced) "bandwidth" fee for exporting the data to the appliance which will be shipped to you.
- gist 9y ago
  Same issue. Another idea would be for GCP to handle export to a physical device and that device would be shipped to a known location (to both GCP and the client) and accessible by either but offline and cold stored. And similar to how paper records are stored you could put in a request to access the records and thereby verify their existence (which you can't do with cloud data other than the copy that you are able to access). Fwiw, Iron Mountain lost about 60 physical records file boxes that we had ..
johansch 9y ago
In the same theme: moving very large amounts of data between GC and AWS, in either direction. I wonder what it would take for them to make this cost zero. Legislation? It likely wouldn't be very expensive to get a very high-capacity fiber connection going between GC/AWS datacenters in relative physical proximity. Not expressing a particular/immediate personal need here, just noting that this could help keep the lock-in factor down and level of competition up.
deleted 9y ago
[deleted]

jrowley 9y ago

This a bit tangental but I like the graphs/visualizations they used in this ad. Easy to read and interrupt

boulos 9y ago
Agreed! The diagram [1] is super handy, and lets me reiterate why we waited to so long to do this: 10 Gbps of peering is just not that rare (and you probably want it for updates, etc. anyway). Even as you get to the petabyte range, being able to just hit "Go" and then do differential updates is so valuable that you really have to be talking about lots of Petabytes in a location where (or reason why) you can't get 10Gbps plus of peering. Disclosure: I work on Google Cloud (but didn't work on this) [1] https://3.bp.blogspot.com/-SnFabcStXhM/WW4SEhj6adI/AAAAAAAAEIE/wd6Yqlbt9FELUxsvTc2F92GZDZq2I-_6QCLcBGAs/s1600/transfer-appliance-2.png https://3.bp.blogspot.com/-SnFabcStXhM/WW4SEhj6adI/AAAAAAAAE...
- vgt 9y ago
  To add to Boulos point - Google Cloud has the "Cloud Transfer Service" [0] that's hugely popular. It's easy to get connectivity to Google's vast network of POPS and this service helps take advantage of that connectivity. Also S3. (work at G but not in any of these areas) https://cloud.google.com/storage/transfer/ https://cloud.google.com/storage/transfer/

bpicolo 9y ago

To be fair, AWS launched a much, much larger analogue to AWS Snowball. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/

nealmueller 9y ago
Hahaha, you got me there. Althought, to be fair, Snowmobile is an entirely different product category. Same problem, different scale of solution. It's like Starbucks coming out with a truck solution, when all you wanted was a Venti.
- bpicolo 9y ago
  Espresso trucks sound awesome.
  - dwyerm 9y ago
    Interestingly, Starbucks does have both a trailer solution and a shipping-container solution. The former is parked out front when the actual store is undergoing refurbishment. The latter is seemingly air-dropped into neighborhoods like Northglenn, CO and Ballard, WA. Coincidentally, Starbuck's first shipping container store is just a mile or two from Amazon's first use of the Snowmobile.
Jyaif 9y ago
"Snowmobile uses multiple layers of security designed to protect your data including dedicated security personnel, GPS tracking, alarm monitoring, 24/7 video surveillance, and an optional escort security vehicle" The data is encrypted, so that part is really just to impress the commoner.
- losteric 9y ago
  I assume these services were originally created to meet government contracting requirements (transporting highly classified information).

CaliforniaKarl 9y ago

Awwwww, it's too bad this can't be used to get stuff directly into Drive.

nealmueller 9y ago
Good idea, thanks for the suggestion. The interest is interesting, and I'll definitely feed that back. I felt the pain recently myself when I moved all my photos and videos to Google Photos and I wished I had one of these.

binaryblitz 9y ago

$300 for a 100TB rackable drive? Can I just keep it instead?

Veratyr 9y ago
It seems to pretty clearly be a rental, so no...
canes123456 9y ago
Obviously not. Rent a car for a week seems cheap. Can I keep it forever?
nealmueller 9y ago
The other comments here are correct. The standard model for these appliances is a rental model. Please talk to a GCP seller.
trevyn 9y ago
$30 per day after 10 day grace period: https://cloud.google.com/data-transfer/pricing https://cloud.google.com/data-transfer/pricing It would be funnier if you could keep it indefinitely, but reads were $0.12/GB. ;) I wonder if that's a viable business model...
- shiftpgdn 9y ago
  Nimble offers great local storage on a pay per month per use model. I'm sure HP is working hard to ruin this.
binaryblitz 9y ago
Good god... It was a joke...

westernmostcoy 9y ago

Article title is actually "Introducing Transfer Appliance: Sneakernet for the cloud era", can someone change how it's presented here? This was not a good improvised title.

sctb 9y ago
Yes, thank you. We've updated the title from “Google launches a larger analogue to AWS Snowball”.

twakefield 9y ago

"Like many organizations we talk to, you probably have large amounts of data that you want to use to train machine learning models." I understand's Google's bias here but doesn't it usually make more sense to bring the programs/models to where the data already is?

bpicolo 9y ago
The problem isn't the programs, it's the hardware. GCP has specialized / scalable hardware to build models on.
zeroxfe 9y ago
It is extremely expensive to setup and maintain infrastructure to process big data, especially when you get into the petabytes.
btian 9y ago
GCP has proprietary machine learning accelerator hardware that you can't buy.

wtvanhest 9y ago

Anyone know how to backup 1tbyte worth of images in google or AWS? My ISP throttles my upload time. I'm interested in doing it at a lower price point than $500. Is that possible?

theDoug 9y ago
I've happily used Arq (https://www.arqbackup.com/ https://www.arqbackup.com/) to get a lot of things into Drive and GCP. You can control your network rate to still get the files uploaded but over a longer time-span below your ISP's throttling. (Disclosure: I work on GCP but this is a personal, not professional, endorsement)
- grogenaut 9y ago
  it's multi cloud so I think you're safe from showing a bias.
rb808 9y ago
I was thinking of buying a NAS just for this. It would take a while, but would be running all night which is handy. cheapest synology is $110 new + disk. https://www.synology.com/en-uk/knowledgebase/DSM/help/CloudSync/cloudsync https://www.synology.com/en-uk/knowledgebase/DSM/help/CloudS...
wmf 9y ago
https://awsimportexport.s3.amazonaws.com/aws-import-export-calculator.html https://awsimportexport.s3.amazonaws.com/aws-import-export-c... says ~$90.
- rb808 9y ago
  thanks I hadn't seen that price, looks handy
rb808 9y ago
Try this too: S3 Transfer Acceleration, maybe gets around ISP throttling (I haven't tried this yet) https://aws.amazon.com/s3/faqs/#s3ta https://aws.amazon.com/s3/faqs/#s3ta

peterjlee 9y ago

So a Hooli Box?

compuguy 9y ago
Hooli Box compresses and stores data. This is just used to transfer data to GCP.
- brianwawok 9y ago
  This compresses.

Poiesis 9y ago

Anyone know what's inside the box?