55 ms·
AWS Customers Rack Up Hefty Bills for Moving Data
- toomuchtodo 7y agoAnyone (no longer under NDA) know if Apple intends to migrate user iCloud data from AWS and GCP object stores to their own data centers at some point?
- whoisjuan 7y ago> no longer under NDA Do NDAs expire? AFAIK you sign an NDA and you’re bound to respect that forever. Unless the knowledge becomes publicly available.
- taormina 7y agoI mean, it depends on the language of the NDA. As a quick example, this random free template I found online[0] uses "5 years". [0] https://www.docracy.com/1/generic-nda https://www.docracy.com/1/generic-nda
- penagwin 7y agoQuestion for any lawyers here, do NDA's have any form of "natural expiration" if no explicit expiration date is set? What happens to them if the company goes out of business?
- brightball 7y agoIANAL I have spoken to a lot of lawyers about contract terms and what I've been told for both NDA and Non-compete style agreements is that they have to have very clearly defined scope to be enforceable. For a non-compete for example, it requires distance and specific field with a clearly defined time period (that's reasonable). For an NDA you have to ensure that the covered information is explicitly labeled (which is why people have confidentiality labels in email footers) and that there is a defined time to expiration. An NDA without those two criteria place an undue burden on a person who is not being compensated for their compliance. Reasonable time period is generally 2 years.
- brodouevencode 7y agoThis is typically true of all contract law. "In perpetuity" is not a thing and if it is, it usually nullifies the contract.
- skunkworker 7y agoFor a service I built on AWS lambda, the far majority of the cost was just data egress. Unfortunately I haven’t found a competitor that offers the same kind of features that I’m using lambda for (running golang with additional custom binaries)
- deleted 7y ago[deleted]
- manigandham 7y agoBoth Azure and GCP let you run arbitrary containers in a serverless fashion. GCP Cloud Run is per-request too.
- penagwin 7y agoYes but both still charge crazy money for the bandwidth. (Although Azure is apart of the bandwidth alliance if you use cloudflare/b2)
- cosmie 7y agoAzure is part of Cloudflare's Bandwidth Alliance[1]. If you use Azure's serverless functions and put your service behind Cloudflare, you'll get free egress. [1] https://www.cloudflare.com/bandwidth-alliance/ https://www.cloudflare.com/bandwidth-alliance/
- skunkworker 7y agoI was aware of the bandwidth alliance, though looking on their KB for now it's only discounted egress [1]. Same with Google, though not part of the bandwidth alliance, data delivered through their CDN Interconnect program starts at $0.04/gb (NA) [2] [1] https://support.cloudflare.com/hc/en-us/articles/360016143912-How-do-I-get-discounted-data-transfer-with-my-cloud-provider- https://support.cloudflare.com/hc/en-us/articles/36001614391... [2] https://cloud.google.com/interconnect/docs/how-to/cdn-interconnect https://cloud.google.com/interconnect/docs/how-to/cdn-interc...
- 7y ago
- tyingq 7y ago"The charges don’t appear to be a case of cloud providers gouging their customers" I disagree on this one. The margins on egress are, well...egregious.
- snagglegaggle 7y agoAt some point everyone realized they could charge a massive premium per bit and as long as everyone did it, customers would have to pay. So here we are.
- notaround1111 7y agoCustomers are also to blame, when comparing the costs of two services they tend to look at the cost of an instance hour, or lambda execution and often don't look at transfer costs. Even if a cloud provider had competitive transfer costs they likely wouldn't attract any new customers and would have less margin left over to subsidize the main cost customers look at, $ per instance hour. The less attention is paid to transfer costs the better for AWS/GCP/Azure. Why hasn't a spot-market for transfer been introduced? Same reason why I can't sell my unused home internet bandwidth to my neighbors, the money is in controlling the means of transportation/communication and the providers want to keep as tight a control on that as possible.
- coleca 7y agoI wouldn't blame customers when the pricing for data transfer looks like this: https://raw.githubusercontent.com/open-guides/og-aws/master/figures/aws-data-transfer-costs.png https://raw.githubusercontent.com/open-guides/og-aws/master/... (Source Open Guide to AWS - https://github.com/open-guides/og-aws https://github.com/open-guides/og-aws)
- Pfhreak 7y agoThis seems a little deliberately obtuse -- for example, showing two arrows from an EC2 instance to an EC2 instance that exits the VPC. But I generally don't find this too hard to follow? Traffic within an AZ is generally free, but there are some cases where it's not and they generally make sense to me (leaving the VPC, pushing data from your CDN back upstream, etc.) Then again, I worked for AWS for years, so maybe I'm just used to thinking this way so I'm not really surprised by it.
- gruez 7y agoIs there a plausible explanation why egress fees from cloud providers costs around $0.1/GB? "Traditional" server providers such as Hetzner are able to offer bandwidth at orders of magnitude lower price (eg. $1.1/TB). I understand that cloud providers may have better interconnects or better uptimes, but that doesn't justify the magnitudes higher pricing.
- deleted 7y ago[deleted]
- qaq 7y agoThats one of the big money makers plus it locks you in so there is 0 chance they will change it.
- jrockway 7y agoIt has no basis in reality. You can buy a 10Gbps transit circuit for $1500/month. That's $0.00046296296 per gigabyte.
- danpalmer 7y agoThat depends on how much saturation you can get. Most end users/companies can’t saturate this reliably, or have much spoiler traffic. Only the biggest players can minimise their seasonality enough to get close to this, and doing that is a service that it’s probably worth paying for.
- bsder 7y agoWell, reverse it: Amazon is about $1K per 10TB. That's an expensive disk drive.
- nostrebored 7y agoThinking of it as a disk drive is fallacious
- mlyle 7y ago
- ceejayoz 7y ago> A person close to AWS said its data transfer charges reflect a range of technology costs customers would normally pay if they weren’t using cloud services, including fiber optic connections, networking hardware devices and software, cybersecurity services and network monitoring software. I'd believe this more if the pricing of bandwidth on AWS hadn't stayed pretty flat since its launch. Plus, it's frustrating that AWS Lightsail (https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/) offers a $3.50/month plan with 1 TB of transfer. That terabyte alone will cost you $92.07 on a normal instance, and the $3.50 includes storage and an instance!
- deleted 7y ago[deleted]
- Polyisoprene 7y agoCheap shovels and expensive ladders.
- jrockway 7y agoAmazon has gone one step past that and made the shovels expensive too. The fear of buying a computer has made them billions.
- frandroid 7y agoIt's not just the fear of buying a computer, it's also the fear of hiring staff...
- jrockway 7y agoI don't think Amazon really provides any products that are going to prevent you from hiring staff. That's the domain of companies like Zendesk or Salesforce. All they provide you is the bare metal in easily-purchasable quantities. Whatever time you saved not having to lug a server up to your datacenter and plug it in will be spent debugging CloudFormation stacks or figuring out how to auction off your reserved instances that you no longer want.
- frandroid 7y agoObviously you'll still need some staff. But suddenly you don't need staff that knows how to handle physical server rooms and data centers, you limit it to AWS which covers a whole range of needs which would need to be addressed by different staff with different skillsets.
- wuliwong 7y ago>The fear of buying a computer has made them billions. Are you saying that AWS service aren't providing value to their customers but it's fear-based marketing?
- 7y ago
- prolificd 7y agoIt does appear like price gouging. Once you're in, you're locked in. Any chance some of those high costs are due to movement of data due to GDPR compliance? Maybe Apple did all its prep in 2017. Also there are some misconceptions on inter-AZ data transfer as well: (https://www.lastweekinaws.com/blog/aws-cross-az-data-transfer-costs-more-than-aws-says/ https://www.lastweekinaws.com/blog/aws-cross-az-data-transfe...)
- mitchs 7y agoI assure you there is nothing sinister about the price asymmetry. Fiber and routers have as much bandwidth coming as going. The inbound traffic is mostly composed of little http(s) requests. The outbound is full of images and mountains of JavaScript. Cloud providers don't charge for ingress because they got it for free when they grew their egress capability to meet demand.
- ec109685 7y agoWhen was the last time AWS reduced egress rates? And they could negotiate the best rates on the planet given their scale.
- riking 7y agoI get the impression cloud providers don't "negotiate rates" but rather "build infrastructure" most of the time.
- dx034 7y agoThey don't own many of their data centers. Locations were leaked a while ago, most are colocated in major data centers. They won't dig new trenches and lay cables between those but rather negotiate rates for existing fiber.
- jedberg 7y agoMaybe I missed it but where did they get this info? Was it stolen from AWS? As far as I know, Apple would never share cost data like this, nor would a lot of others on this list. Apple doesn’t even publicly acknowledge that they are customers.
- ghaff 7y ago>"The chart, which is based on internal AWS sales figures obtained by The Information," I'm not sure that "stolen" is the right word but, yes, it certainly appears based on what The Information wrote that either someone at AWS or a third-party with access to the info (probably not too likely) leaked the confidential numbers. Disgruntled former employee or... Who knows.
- balozi 7y agoThe only thing worse than the data transfer costs is explaining them to your non-cloud-savvy stakeholders. Go ahead, try explain this sh*t to your board.
- mattbillenstein 7y agoFairly large optimization if you're smaller and a large amount of your data out is cachable is to run a varnish cache on some of the clouds that give you "free" bandwidth. ie a $20/mo instance on linode gets you 4TB of transfer -- $0.005 per gig. Scale enough of these in various DCs around the world and you have a pretty cheap self-hosted CDN.
- buboard 7y agoa $20/mo old server from hetzner gives you unlimited 1Gbps - thats 324TB /mo
- mattbillenstein 7y agoGood deal - I do most of my work in the US, I don't know of a similar deal here.
- deleted 7y ago[deleted]
- bufferoverflow 7y agoFor $1.67/mo, you can get unlimited 1Gbps here: https://servercheap.net/crm/cart.php?gid=10 https://servercheap.net/crm/cart.php?gid=10 Or if you don't trust "unlimited", Hetzner sells CX11 with 20TB/mo for 2.96 eur/mo https://www.hetzner.com/cloud https://www.hetzner.com/cloud
- mlyle 7y agoLots of places that have 1gbps bare metal unmetered boxes for ~$200/mo, too. At 10% utilization that's about $0.005/gig.
- mattbillenstein 7y agoany in the US?
- yeldarb 7y agoWow, does it really cost 4-5x more to download a 1 GB file from S3 or Google Cloud Storage than it does to store it for a month? That is mind-blowing.
- qvrjuec 7y agoBandwidth is expensive, storage is cheap.
- asdfman123 7y agoAmazon Snowball, if I understand it correctly, is a box full of hard drives that they mail you to transfer data. So no bandwidth there. It still costs $0.03/GB, which is on top of the $200 they charge you to use the service.
- throwaway2048 7y agocloud providers charge a lot more for bandwidth yes, but in reality bandwidth is much, much cheaper than storage for them to provide.
- dx034 7y agoNot with smaller cloud providers or colocation centers.
- eric_b 7y agoIt's nickel and diming all the way. I always thought the S3 pricing was sort of ridiculous. Pay for the amount you store (fair). Pay for the egress data (less fair but ok...). Pay for the number of API calls! (lol!)
- spullara 7y agoIt wasn't true at first but people abused the hell out of it, for example, by storing data only in the filename. The API cost prevents very inefficient use of S3.
- notyourday 7y agoCross AZ traffic: https://www.lastweekinaws.com/blog/aws-cross-az-data-transfer-costs-more-than-aws-says/ https://www.lastweekinaws.com/blog/aws-cross-az-data-transfe... No connection to the blog.
- chickenpotpie 7y agoI've actually been working on a library to help mitigate cloud storage lock-in. The idea is to treat cloud storage providers like disks are treated in RAID. For example, you have 3 separate cloud providers. Cloud providers 1 and 2 have every other byte of data striped across them. Cloud provider 3 has parity data. To pull a file you can only need 2 of the 3 cloud providers. If you don't like how a cloud storage provider is treating you or charging you just pull from the other 2 providers and use them as a backup in case one goes down. You can also just remove them entirely from the equation, but then you have no redundancy if one of the others goes down. It gives you a lot of negotiating power to lower egress costs because you can just pull them out of the equation at any time and reinstate them once you get better pricing.
- planteen 7y agoAre you doing this to replace a CDN? There are already 3rd party CDNs like CloudFlare. If you are doing it as a replacement for traffic within an AWS region and availability zone, it seems like you will be both more expensive and have much higher latency. Or is the application something else entirely?
- chickenpotpie 7y agoIt's something else entirely. It's a mixed cloud approach combining the storage offerings of Azure, Google Cloud, and S3 providers. The idea is not to trust one cloud provider to provide fair pricing and proper redundancy. Right now I'm mirroring RAID 0,1, and 3. Applying RAID 3 to the cloud is going to give you higher latency and more processor and memory usage because the file has to be reassembled on the client machine. However, if you apply RAID 1 to the cloud your latency is similar because each cloud provider has the full file. In the case of RAID 1 the library will upload a full copy to each cloud provider and will download files by trying providers until one succeeds. If you only use two providers your pricing is usually the same because geo-redundancy in storage providers is often twice the cost and you're getting geo-redundancy built in by having multiple providers in different regions. RAID 3 is actually cheaper because you have geo-redundancy, but you're only storing 1.5 times as much data.
- Merrill 7y agoMy GP's practice went to a SaaS provider for electronic health records. Never mind transmission costs -- there is no standard way to export their patient records in order to move to a different provider. It requires a custom consulting services contract with their existing provider to retrieve their patient records, at an exorbitant price tag.
- cabaalis 7y agoThis is quite typical. Even with MIPS PI measures and the push for interoperability. Getting patient data in bulk out of a hosted EHR can be a frustrating exercise. What they are doing is holding the providers' data hostage, in the name of HIPAA and security.
- lazyant 7y agowhere's Corey Quinn
- lacker 7y agoEven the huge-looking bill for Apple is only 6.5% of their total bill. Moving data into AWS is free, so it's not totally surprising that they pay for that by making other stuff more expensive. I found it more interesting just to see a list of their top ten customers. In particular I didn't realize that Capital One had so much infrastructure.
- notyourday 7y ago> Even the huge-looking bill for Apple is only 6.5% of their total bill. Moving data into AWS is free, so it's not totally surprising that they pay for that by making other stuff more expensive Moving data into any network that is outbound heavy is free because both paid peering and transit is settled based on a peak percentile traffic (unless it is flat rate). That's why the "gansta" position is to have a balanced in/out for any network as in that case you get to effectively double charge for the same pipes -- your eyeball heavy customers pay for incoming and your web farm customers pay for outgoing.
- lacker 7y agoThis doesn't deserve to be downvoted, it makes a good point! I had not thought about the likely-to-be-outbound-heavy nature of cloud providers and how that affects things.
- wodenokoto 7y agoWhat's up with Apple's spend? Did they move most of their data out of AWS?
- outworlder 7y agoMy take is also that they were moving out. The article does acknowledge that they spent a fraction of that on the next year.
- ec109685 7y agoOr they optimized via direct connect.
- vetrom 7y agoI've worked at at least one company where we found using Direct Connect to a data center somewhere and egressing via that data center instead of amazon saved us significant bandwidth costs.
- js4ever 7y agoI have recently talked with a friend about a very cheap way to egress from AWS, at $2.5 per TB with the AWS lightsail $5 instance which include 2TB of egress per instance. This friend have extracted 200TB of backups for $500 instead of $17,000!
- FlyingAvatar 7y agoReading this, I immediately thought: the cost has to go way down with 200TB of volume. It goes down, but only to $14,131.11! Nice find. I will file this away if I ever need to do a full remote recovery.
- Elect2 7y agoSo where is your friend's data in? s3? ebs? I'm curious what is the cost to transfer data from s3/ec2 to Lightsail.
- js4ever 7y agoMostly in S3 but S3/EC2 to lightsail is free within the same region :)
- Elect2 7y agoI tried but seems aws has closed the door. To connect ec2 to Lightsail via private IP I have to create a "ec2-classic VPC" in ec2. But aws has disabled "ec2-classic VPC" in all regions
- rbdeveloper 7y agoFrom what I know is that you can peer the Lightsail (AWS-managed) VPC with your normal, default EC2 VPC via the Lightsail console. Then you launch EC2 instances in the default VPC for the transfer
- crazysim 7y agoThat's against Lightsail TOS, if you care.
- pmoriarty 7y agoI recently saw a talk by a couple of former Google employees who have a business helping cloud customers save money, and they were saying that they see a lot of money being wasted by companies in overprovisioning and neglecting to shut down or delete unused resources like VMs or virtual disks. Some of their advice for saving money was to keep track of who created each resource and why, so there's less reason to doubt whether an apparently unused resource can be deleted, and to make some limits regarding how many resources can be automatically created (especially in dev environments). Some other ideas were to look for signs of inactivity like low CPU or bandwidth use, and consider deleting such little used resources. There was much more to the talk, but those were some of the highlights that I can remember without digging out my notes. It was a good talk.
- spike021 7y agoMakes sense. If you're not properly tagging your resources then it can be very hard to track down if/where it's been used, how often, or when it was used last. You can automate/template as much orchestration as you want with stuff like Terraform to bring up well-defined resources, but there will always be outliers without tagging.
- 3wolf 7y agoThe $0.01/GB inter-availability-zone data transfer cost can be a killer for misconfigured workloads. It's a shame AWS doesn't make this more clear on their pricing page. I've seen people run Spark clusters across multiple AZs, incurring a huge costs whenever a shuffle happens.
- Elect2 7y agoData transfer fee is the biggest reason stopped me from using AWS/GCP/Azure. Anyone using Oracle Cloud? I found their bandwidth is really cheap.
- satanspastaroll 7y agoThe thing about that is that it's Oracle. You might survive stuffing your hand in a hornet's nest, but it's unlikely.
- lykr0n 7y agoThe cloud is like the Hotel California. Can checkout anytime but never leave.
- jrochkind1 7y agoWhere is the article getting these numbers from? Did they say and I missed it?
- kresten 7y agoAWS Azure or Google cloud data transfer: 9 cents per GB Digital Ocean: 1 cent per GB The really weird thing is this should be the absolute lead on all Digital Ocean marketing but they don't even mention it. It's their single biggest selling point.
- mostlystatic 7y agoI recently started running more Google Cloud VMs in the UK instead of the US. It was costing me £2 a day to run the VMs, but somehow I also paid £12 a day for "GCP Storage egress between NA and EU”. Turns out that by default Google's Docker container registry only stores the Docker images in the US. So each time I launched a VM the Docker image was downloaded from the US. I wrote more about it here: https://www.mattzeunert.com/2019/10/13/reducing-docker-image-size-and-cutting-gcp-cost.html https://www.mattzeunert.com/2019/10/13/reducing-docker-image... The billing interface didn't show that the Cloud Storage cost was related to the Docker images. I was investigating my normal Cloud Storage use, but it didn't explain why I was being charged so much. Only after a few days did I get the idea that it might be the Docker images that were causing it.
- merb 7y agoactually there is a eu mirror for the gcp registry eu.gcr.io
- chatmasta 7y agoThat seems like the kind of thing that should be configured by default on servers in the EU.
- sieabahlpark 7y agoIt's in their docs
- freehunter 7y agoTheir docs say it’s configured by default on EU servers?
- skj 7y agoYou say what image to use. There is no default.
- 7y ago
- minitoar 7y agoWe mostly deal with this by keeping the data inside an AWS/Azure region as much as possible. You pay to get it in there, and you pay for storage, but you can access it for free within the same region.
- geggam 7y ago>Because this is what the market agrees to bear. Spending points but this joke has to be said Difference between wife and a prostitute ? You know up front how much it costs to get rid of the prostitute. :)
- dang 7y agoI didn't see this earlier, but need to let you know that comments like this will get you banned here. We want thoughtful conversation on HN. Please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and don't post like this on HN.
- united893 7y agoJust curious, how on earth did Kevin and Amir get such incredibly detailed of AWS spending of these accounts. Either they put it together from public resources, or (more worrysome) someone at AWS leaked them line-item based expenses of their top customers. This has got to be incredibly sensitive data for AWS, not something they'd want leaked. If I'm (say) AirBnB or Snap, I'd worry that this data leaks information that would be detrimental to their ability to negotiate for cloud computing with Google, Microsoft etc.
- milesskorpen 7y agoThe article specifically says its based on internal data.
- DoctorOetker 7y agodoes anyone know of a cheap service to order large amounts of data by mail on physical media? I am often interested in some dataset for which I can afford the storage in the form of a hard drive but not in the form of a download through my home connection. If a service existed that simply offered the following: * customer provides URL (and optionally hash checksum) * customer pays, and later receives hard disk drive / SSD drive with the download contained * democratic pricing for the media, or alternatively send your own media (hence at twice media shipping cost...) * possibly eventually local brick and mortar locations / affiliate locations to drop off and pick up media, in the larger cities * preferably without account, although an account is not a large impediment * definitely not coupled to a financial account in a credit fashion, i.e. no qualms with topping up the account, but the service should not be able to withdraw money from my financial account. i.e. debit only (like the typical european bank cards, yes I know credit cards are available in europe as well...) This would seem like a profitable side business for many programmer types who have high bandwidth connections (or have access to them and are allowed to use the connection for this purpouse). If someone builds the software stack for a main portal such that affiliates can advertise their physical location, and their pricing for media, for download and for copying to media, then customers could compare and choose on the basis of price.
- chillfox 7y agoI don't think it would be a cheap service if you set a modest goal of earning a cinema trip with popcorn and a drink for every trip to the post office. 1TB HDD: 59 AUD International Shipping (assuming we can keep it under 1kg with packaging): 38 AUD That is 97 AUD before considering any profit you might want to make (35 AUD for the movie + small popcorn and drink), fixed setup costs like a NAS to cache data sets or the bandwidth costs.
- DoctorOetker 7y agoIf someone hails a cab on uber, one doesn't hail a cab from a different continent... there seems to be plenty of opportunities here, like buy or rent an old bank building with the individually lockable drawers, put your drive in the locker which has a USB cable or ethernet cable, close the locker, use some app to set the URL / hash, pay, and you get a ETA, download complete notification, and a deadline to pick it up (or else incur a fee to unlock the drawer proportional to overtime). I guess the idea could be pitched to those operating rentable local PO boxes, using similar lockers, but with internet connectivity.
- rhizome 7y agoCloud providers can launch and advertise and advocate every little product they're able to refactor out of the prevailing development and systems engineering practices, but transit remains expensive, and basically consistently so. Not only that, but AWS transit is like 20x the price of boring old hosting providers, providers who don't also charge for egress.
- sbhn 7y agoUk gov spends more on AWS than it does for EU membership.
- chriselles 7y agoIt sounds very similar to physical documents/records storage service “roach motel”-like fee structures. Make it nearly free to move documents in. Charge monthly rent per box of documents. But charge like a wounded bull if customers try to permanently remove documents.