4 ms·

New physical AWS Data Transfer Terminals let you upload to the cloud faster

49 points by vinni2 2y ago

greatgib 2y ago
Just for a reference of the price, it will be 300$ per hour in US and 500$ in EU. That looks quite expensive in my opinion, even if this target big professionals.
DamonHD 2y ago
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Andrew S. Tanenbaum
- aitchnyu 2y ago
  I just learned the AWS truck has been retired for months https://www.datacenterdynamics.com/en/news/aws-retires-snowmobile-truck-based-data-transfer-service/ https://www.datacenterdynamics.com/en/news/aws-retires-snowm...
  - thanksgiving 2y ago
    I wonder how this process worked in practice. Do you simply send your only one set of hard disks and hope for the best? Do you keep your original data on one set of disks, copy it to another set of disks and send it off? Do you make multiple copies of the same data and send them all together on the same truck? Multiple copies on multiple trucks? How would you do reconciliation on the other end once the disks arrive at the destination? Like everything feels so simple and straightforward from afar but once I try to actually reason about something even the simplest of tasks feels complicated.
    - jamessb 2y ago
      > I wonder how this process worked in practice. Do you simply send your only one set of hard disks and hope for the best? No, you don't put your own disks in the Snowmobile/Snowball/Snowclone. It contains disks, so when it arrives you connect it to your network and copy data onto it, and then it is driven to an Amazon datacentre. See, e.g. https://docs.aws.amazon.com/snowball/latest/developer-guide/how-it-works.html https://docs.aws.amazon.com/snowball/latest/developer-guide/... https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-week-using-amazon-owned-storage-appliances/ https://aws.amazon.com/blogs/aws/aws-importexport-snowball-t...
      - thanksgiving 2y ago
        Thank you. That makes much more sense. So this way once the data is on the AWS device it is Amazon.com's responsibility so the customer doesn't need to worry about the truck or whatever.
      - hulitu 2y ago
        > so when it arrives you connect it to your network and copy data onto it, and then it is driven to an Amazon datacentre. And if there is an accident on the road, bad luck. /s
oatmeal_croc 2y ago
Judging by the locations (New York and LA) I wonder if this is to cater to folks from production houses who want to upload large video files for processing or backup.
- bigs 2y ago
  Or law firms with large discovery data sets
  - aitchnyu 2y ago
    How big are they? I thought its the 100gb order of magnitude as (dead tree) libraries.
    - berbec 2y ago
      It adds up quick. I know of a law firm (under a 100 employees) with over 20TB.
      - bbatha 2y ago
        That's still peanuts, you can get consumer grade HDDs with that capacity in a single drive. A business grade line would have no trouble uploading all of that data in less than a week, even with a bunch of extenuating circumstances.
        eightys3v3n 2y ago
        Some smaller businesses may have a huge data store, but not the money to pay for a business grade internet connection to upload it in a reasonable amount of time. I've worked for clients who have a 10 megabit full duplex fiber connection for over $1,000 a month (probably because of low ISP competition and because they were in a newly built, low density area). If they were working on migrating to the cloud, they would certainly consider taking a few hard drives one time to AWS rather than maxing their 10 MB full duplex connection for weeks or months.
        2y ago
arjvik 2y ago
> Don’t be surprised if there are no AWS signs in the building or room. This is for security reasons to keep your work location as secret as possible. Huh?
- trallnag 2y ago
  From time to time around midnight Bezos can be seen peeking through the "A" of random AWS signs. This prevents that
vladde 2y ago
I don't think I've ever handled so much data I'd need to increase data throughput. How much are we talking? Like petabytes? (Do you just stroll in with on a huge disk array?)
- bombcar 2y ago
  I don't think it's really so much about throughput as it is about avoiding any ingress chargers at all.
  - IanCal 2y ago
    Does aws charge ingress? I've got a good connection for the UK but still about a 100 meg upload. I have external drives that are 4TB, so if everything goes perfectly that could take about two full days to upload.
  - kondro 2y ago
    AWS doesn't charge for ingress.
    - canucker2016 2y ago
      FTA: "There will be no per GB charge for the data transfer if you upload data into AWS Regions in the same continent of your location." N.B. I'm not contradicting the parent, but reframing the concept - it seems that the Data Transfer Terminal is ALREADY in AWS so ingress isn't a thing since your data is ALREADY in AWS as soon as you connect the optic fiber cable to whatever storage you've brought onsite. But since you're only renting the connection, your data can't stay forever in AWS unless you copy to S3 or somewhere else in AWS.
- unsnap_biceps 2y ago
  They mention it as a way to take your snowball and upload it and walk out and continue using it without shipping it back and forth. Those look to go into the 210 TB range of raw storage. In the past, at some companies I was at, I could see using something like this once a quarter to upload full quarterly backups, depending on the price per hour.
unsnap_biceps 2y ago
I presume that given they mention that it only supports public endpoints, that it's just a directly peered connection on the public internet and there's no special security stuff at play here?
- donavanm 2y ago
  Kind of. Im pretty sure “public endpoint” is to disambiguate from virtual network functionality like VPC Endpoints and VPC Peering etc which are specific to your account/VPC. To your other comment, not many companies own the “layer 0” physical infra long haul. But yes, this is almost absolutely going over “dark fiber” that AWS is leasing from someone like zayo or L3. Since 2016ish effectively all AWS regions and DX POPs use this model to interconnect. The LAX & NYC sites are almost definitely the same DCs & TCs used for “local zones.” So yes, a few hundred gb to tb of leased lines back to PDX and IAD. Search for “aws backbone” content. And for security Im not sure whats public these days, but fiber taps are a real and known threat. Source: ex AWS PE who spent some time in and adjacent to edge networking.
- vineyardmike 2y ago
  They probably mentioned public endpoints to highlight that they allow access to AWS services but they’re not renting out access to a massive bandwidth pipe for non-customers or to upload to non-AWS. The “public endpoints” part probably is used to warn customers that existing technical limitations apply wrt abilities- eg you can’t dump data directly to the S3 data plane, you still have to go through the public API. Considering the fiber connection is “part of the AWS network”, it may not have access to the outside internet, but also contextually probably doesn’t have any privileged access to AWS servers.
  - unsnap_biceps 2y ago
    My statement that it's on the public internet is more that it's directly connected to someone like level3 and not a direct fiber drop to an AWS datacenter. I would be surprised if the building in NYC has direct fiber to IAD or if the building in LA has direct fiber to SFO.
Snoozus 2y ago
It looks like there is no local buffer, so one needs to stick around till the upload is done? Did anyone see a mention of upload bandwidth?
- canucker2016 2y ago
  from the FAQ, https://aws.amazon.com/data-transfer-terminal/faqs/ https://aws.amazon.com/data-transfer-terminal/faqs/ - What is the connection type? - Each Data Transfer Terminal facility will have at least two (2) 100G optical fiber cables that are connected to the AWS network. - What are the key requirements for preparing my device to use the Data Transfer Terminal facility? - To prepare for using the Data Transfer Terminal facility and connecting to the network, you need to ensure your uploading device is prepared to connect to the network. You should have the following for an optimal data upload experience: • A transceiver type 100G LR4 QSFP • An active IP auto configuration (DHCP) • Up-to-date software/transceiver drivers Also, from https://aws.amazon.com/data-transfer-terminal/pricing/ https://aws.amazon.com/data-transfer-terminal/pricing/: Per port charge for data transfer US to US @ $300 US to EU @ $500 US to Other @ Contact us
solardev 2y ago
I wonder if this will cause devices like ATM skimmers to pop up at these secret locations, skimming the traffic via MITM attacks on the network or tampered laptops.
- averageRoyalty 2y ago
  They appear to be rooms inside data centers, so it's pretty doubtful.
DonHopkins 2y ago
Are the facilities available to rent late at night for LAN parties? And do they serve good refreshments?
Havoc 2y ago
I would have thought download would be more interesting. Dodge the egress charges on cloud migrations
- canucker2016 2y ago
  Egress charges for migrations hasn't been a problem since 2024 March. from https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i... We believe this choice must include the one to migrate your data to another cloud provider or on-premises. That’s why, starting today, we’re waiving data transfer out to the internet (DTO) charges when you want to move outside of AWS. If you need more than 100 gigabytes of data transfer out per month while transitioning, you can contact AWS Support to ask for free DTO rates for the additional data. It’s necessary to go through support because you make hundreds of millions of data transfers each day, and we generally do not know if the data transferred out to the internet is a normal part of your business or a one-time transfer as part of a switch to another cloud provider or on premises. We will review requests at the AWS account level. Once approved, we will provide credits for the data being migrated. We don’t require you to close your account or change your relationship with AWS in any way. You’re welcome to come back at any time. We will, of course, apply additional scrutiny if the same AWS account applies multiple times for free DTO.
  - EwanToo 2y ago
    If you request this, AWS requires: "After your move away from AWS services, within the 60-day period, you must delete all remaining data and workloads from your AWS account, or you can close your AWS account." So egress charges are still a significant problem. https://aws.amazon.com/ec2/faqs/#Data_transfer_fees_when_moving_all_data_off_AWS https://aws.amazon.com/ec2/faqs/#Data_transfer_fees_when_mov...

49 points by vinni2 2y ago

Just for a reference of the price, it will be 300$ per hour in US and 500$ in EU. That looks quite expensive in my opinion, even if this target big professionals.

DamonHD 2y ago

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Andrew S. Tanenbaum

aitchnyu 2y ago
I just learned the AWS truck has been retired for months https://www.datacenterdynamics.com/en/news/aws-retires-snowmobile-truck-based-data-transfer-service/ https://www.datacenterdynamics.com/en/news/aws-retires-snowm...
- thanksgiving 2y ago
  I wonder how this process worked in practice. Do you simply send your only one set of hard disks and hope for the best? Do you keep your original data on one set of disks, copy it to another set of disks and send it off? Do you make multiple copies of the same data and send them all together on the same truck? Multiple copies on multiple trucks? How would you do reconciliation on the other end once the disks arrive at the destination? Like everything feels so simple and straightforward from afar but once I try to actually reason about something even the simplest of tasks feels complicated.
  - jamessb 2y ago
    > I wonder how this process worked in practice. Do you simply send your only one set of hard disks and hope for the best? No, you don't put your own disks in the Snowmobile/Snowball/Snowclone. It contains disks, so when it arrives you connect it to your network and copy data onto it, and then it is driven to an Amazon datacentre. See, e.g. https://docs.aws.amazon.com/snowball/latest/developer-guide/how-it-works.html https://docs.aws.amazon.com/snowball/latest/developer-guide/... https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-week-using-amazon-owned-storage-appliances/ https://aws.amazon.com/blogs/aws/aws-importexport-snowball-t...
    - thanksgiving 2y ago
      Thank you. That makes much more sense. So this way once the data is on the AWS device it is Amazon.com's responsibility so the customer doesn't need to worry about the truck or whatever.
    - hulitu 2y ago
      > so when it arrives you connect it to your network and copy data onto it, and then it is driven to an Amazon datacentre. And if there is an accident on the road, bad luck. /s

oatmeal_croc 2y ago

Judging by the locations (New York and LA) I wonder if this is to cater to folks from production houses who want to upload large video files for processing or backup.

bigs 2y ago
Or law firms with large discovery data sets
- aitchnyu 2y ago
  How big are they? I thought its the 100gb order of magnitude as (dead tree) libraries.
  - berbec 2y ago
    It adds up quick. I know of a law firm (under a 100 employees) with over 20TB.
    - bbatha 2y ago
      That's still peanuts, you can get consumer grade HDDs with that capacity in a single drive. A business grade line would have no trouble uploading all of that data in less than a week, even with a bunch of extenuating circumstances.
      - eightys3v3n 2y ago
        Some smaller businesses may have a huge data store, but not the money to pay for a business grade internet connection to upload it in a reasonable amount of time. I've worked for clients who have a 10 megabit full duplex fiber connection for over $1,000 a month (probably because of low ISP competition and because they were in a newly built, low density area). If they were working on migrating to the cloud, they would certainly consider taking a few hard drives one time to AWS rather than maxing their 10 MB full duplex connection for weeks or months.
      - 2y ago

arjvik 2y ago

> Don’t be surprised if there are no AWS signs in the building or room. This is for security reasons to keep your work location as secret as possible. Huh?

trallnag 2y ago
From time to time around midnight Bezos can be seen peeking through the "A" of random AWS signs. This prevents that

vladde 2y ago

I don't think I've ever handled so much data I'd need to increase data throughput. How much are we talking? Like petabytes? (Do you just stroll in with on a huge disk array?)

bombcar 2y ago
I don't think it's really so much about throughput as it is about avoiding any ingress chargers at all.
- IanCal 2y ago
  Does aws charge ingress? I've got a good connection for the UK but still about a 100 meg upload. I have external drives that are 4TB, so if everything goes perfectly that could take about two full days to upload.
- kondro 2y ago
  AWS doesn't charge for ingress.
  - canucker2016 2y ago
    FTA: "There will be no per GB charge for the data transfer if you upload data into AWS Regions in the same continent of your location." N.B. I'm not contradicting the parent, but reframing the concept - it seems that the Data Transfer Terminal is ALREADY in AWS so ingress isn't a thing since your data is ALREADY in AWS as soon as you connect the optic fiber cable to whatever storage you've brought onsite. But since you're only renting the connection, your data can't stay forever in AWS unless you copy to S3 or somewhere else in AWS.
unsnap_biceps 2y ago
They mention it as a way to take your snowball and upload it and walk out and continue using it without shipping it back and forth. Those look to go into the 210 TB range of raw storage. In the past, at some companies I was at, I could see using something like this once a quarter to upload full quarterly backups, depending on the price per hour.

unsnap_biceps 2y ago

I presume that given they mention that it only supports public endpoints, that it's just a directly peered connection on the public internet and there's no special security stuff at play here?

donavanm 2y ago
Kind of. Im pretty sure “public endpoint” is to disambiguate from virtual network functionality like VPC Endpoints and VPC Peering etc which are specific to your account/VPC. To your other comment, not many companies own the “layer 0” physical infra long haul. But yes, this is almost absolutely going over “dark fiber” that AWS is leasing from someone like zayo or L3. Since 2016ish effectively all AWS regions and DX POPs use this model to interconnect. The LAX & NYC sites are almost definitely the same DCs & TCs used for “local zones.” So yes, a few hundred gb to tb of leased lines back to PDX and IAD. Search for “aws backbone” content. And for security Im not sure whats public these days, but fiber taps are a real and known threat. Source: ex AWS PE who spent some time in and adjacent to edge networking.
vineyardmike 2y ago
They probably mentioned public endpoints to highlight that they allow access to AWS services but they’re not renting out access to a massive bandwidth pipe for non-customers or to upload to non-AWS. The “public endpoints” part probably is used to warn customers that existing technical limitations apply wrt abilities- eg you can’t dump data directly to the S3 data plane, you still have to go through the public API. Considering the fiber connection is “part of the AWS network”, it may not have access to the outside internet, but also contextually probably doesn’t have any privileged access to AWS servers.
- unsnap_biceps 2y ago
  My statement that it's on the public internet is more that it's directly connected to someone like level3 and not a direct fiber drop to an AWS datacenter. I would be surprised if the building in NYC has direct fiber to IAD or if the building in LA has direct fiber to SFO.

Snoozus 2y ago

It looks like there is no local buffer, so one needs to stick around till the upload is done? Did anyone see a mention of upload bandwidth?

canucker2016 2y ago
from the FAQ, https://aws.amazon.com/data-transfer-terminal/faqs/ https://aws.amazon.com/data-transfer-terminal/faqs/ - What is the connection type? - Each Data Transfer Terminal facility will have at least two (2) 100G optical fiber cables that are connected to the AWS network. - What are the key requirements for preparing my device to use the Data Transfer Terminal facility? - To prepare for using the Data Transfer Terminal facility and connecting to the network, you need to ensure your uploading device is prepared to connect to the network. You should have the following for an optimal data upload experience: • A transceiver type 100G LR4 QSFP • An active IP auto configuration (DHCP) • Up-to-date software/transceiver drivers Also, from https://aws.amazon.com/data-transfer-terminal/pricing/ https://aws.amazon.com/data-transfer-terminal/pricing/: Per port charge for data transfer US to US @ $300 US to EU @ $500 US to Other @ Contact us

solardev 2y ago

I wonder if this will cause devices like ATM skimmers to pop up at these secret locations, skimming the traffic via MITM attacks on the network or tampered laptops.

averageRoyalty 2y ago
They appear to be rooms inside data centers, so it's pretty doubtful.

DonHopkins 2y ago

Are the facilities available to rent late at night for LAN parties? And do they serve good refreshments?

Havoc 2y ago

I would have thought download would be more interesting. Dodge the egress charges on cloud migrations

canucker2016 2y ago
Egress charges for migrations hasn't been a problem since 2024 March. from https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i... We believe this choice must include the one to migrate your data to another cloud provider or on-premises. That’s why, starting today, we’re waiving data transfer out to the internet (DTO) charges when you want to move outside of AWS. If you need more than 100 gigabytes of data transfer out per month while transitioning, you can contact AWS Support to ask for free DTO rates for the additional data. It’s necessary to go through support because you make hundreds of millions of data transfers each day, and we generally do not know if the data transferred out to the internet is a normal part of your business or a one-time transfer as part of a switch to another cloud provider or on premises. We will review requests at the AWS account level. Once approved, we will provide credits for the data being migrated. We don’t require you to close your account or change your relationship with AWS in any way. You’re welcome to come back at any time. We will, of course, apply additional scrutiny if the same AWS account applies multiple times for free DTO.
- EwanToo 2y ago
  If you request this, AWS requires: "After your move away from AWS services, within the 60-day period, you must delete all remaining data and workloads from your AWS account, or you can close your AWS account." So egress charges are still a significant problem. https://aws.amazon.com/ec2/faqs/#Data_transfer_fees_when_moving_all_data_off_AWS https://aws.amazon.com/ec2/faqs/#Data_transfer_fees_when_mov...