12 ms·
A $1k AWS mistake
- fragmede 11mo agoJust $1,000? Thems rookie numbers, keep it up, you'll get there (my wallet won't, ow).
- thecodemonkey 11mo agoHaha, yep we were lucky to catch this early! It could easily have gotten lost with everything else in the monthly AWS bill.
- bravetraveler 11mo agoCame here to say the same, take my vote - DevOops
- harel 11mo agoYou probably saved me a future grand++. Thanks
- thecodemonkey 11mo agoThat was truly my hope with this post! Glad to hear that
- nrhrjrjrjtntbt 11mo agoNAT gateway probably cheap as fuck for Bezos & co to run but nice little earner. The parking meter or exit ramp toll of cloud infra. Cheap beers in our bar but $1000 curb usage fee to pull up in your uber.
- tecleandor 11mo agoI think it's been calculated that data transfer is the biggest margin product in all AWS catalog by a huge difference. A 2021 calculation done by Cloudflare [0] estimated almost 8000% price markup in EU and US regions. And I can see how, in very big accounts, small mistakes on your data source when you're doing data crunching, or wrong routing, can put thousands and thousands of dollars on your bill in less than an hour. -- 0: https://blog.cloudflare.com/aws-egregious-egress/
- wiether 11mo ago> can put thousands and thousands of dollars on your bill in less than an hour By default a NGW is limited to 5Gbps https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway... A GB transferred through a NGW is billed 0.05 USD So, at continuous max transfer speed, it would take almost 9 hours to reach $1000 Assuming a setup in multi-AZ with three AZs, it's still 3 hours if you have messed so much that you can manage to max your three NGWs I get your point but the scale is a bit more nuanced than "thousands and thousands of dollars on your bill in less than an hour" The default limitations won't allow this.
- tecleandor 11mo agoThat's a NAT gateway, but if you're pulling data for analysis from S3 buckets you don't have those limitations. Let's say they decide to recalculate or test a algorithm: they do parallel data loading from the bucket(s), and they're pulling from the wrong endpoint or region, and off they go. And maybe they're sending data back, so they double the transfer price. RDS Egress. EC2 Egress. Better keep good track of your cross region data!
- ukoki 11mo agoI don't think its about profits, its about incentivising using as many AWS products as possible. Consider it an 'anti-lock-in fee'
- deleted 11mo ago[deleted]
- CjHuber 11mo agoDoes Amazon refund you for mistakes, or do you have to land on HN frontpage for that to happen?
- Aeolun 11mo agoI presume it depends on your ability to pay for your mistakes. A $20/month client is probably not going to pony up $1000, a $3000/month client will not care as much.
- viraptor 11mo agoThey do sometimes if you ask. Probably depends on each case though.
- thecodemonkey 11mo agoHahaha. I'll update the post once I hear back from them. One could hope that they might consider an account credit.
- Dunedan 11mo agoDepends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them. Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measures we implemented to ensure that such a situation doesn't happen again.
- rwmj 11mo agoWait, what measures you implemented? How about AWS implements a hard cap, like everyone has been asking for forever?
- Dunedan 11mo agoThe measures were related to the specific cause of the unintended charges, not to never incur any unintended charges again. I agree AWS needs to provide better tooling to enable its customers to avoid such situations.
- viraptor 11mo agoThe service gateways are such a weird thing in AWS. There seems to be no reason not to use them and it's like they only exist as a trap for the unaware.
- wiether 11mo agoReading all the posts about people who got bitten by some policies on AWS, I think they should create two modes: - raw - click-ops Because, when you build your infra from scratch on AWS, you absolutely don't want the service gateways to exist by default. You want to have full control on everything, and that's how it works now. You don't want AWS to insert routes in your route tables on your behalf. Or worse, having hidden routes that are used by default. But I fully understand that some people don't want to be bothered but those technicalities and want something that work and is optimized following the Well-Architected Framework pillars. IIRC they already provide some CloudFormation Stacks that can do some of this for you, but it's still too technical and obscure. Currently they probably rely on their partner network to help onboard new customers, but for small customers it doesn't make sense.
- viraptor 11mo ago> you absolutely don't want the service gateways to exist by default. Why? My work life is in terraform and cloudformation and I can't think of a reason you wouldn't want to have those by default. I mean I can come up with some crazy excuses, but not any realistic scenario. Have you got any? (I'm assuming here that they'd make the performance impact ~0 for the vpc setup since everyone would depend on it)
- wiether 11mo agoBecause I want my TF to reflect exactly my infra. If I declare two aws_route resources for my route table, I don't want a third route existing and being invisible. I agree that there is no logical reason to not want a service gateway, but it doesn't mean that it should be here by default. The same way you need to provision an Internet Gateway, you should create your services gateways by yourself. TF modules are here to make it easier. Everything that comes by default won't appear in your TF, so it becomes invisible and the only way to know that it exists is to remember that it's here by default.
- merpkz 11mo ago> AWS charges $0.09 per GB for data transfer out to the internet from most regions, which adds up fast when you're moving terabytes of data. How does this actually work? So you upload your data to AWS S3 and then if you wish to get it back, you pay per GB of what you stored there?
- hexbin010 11mo agoYes uploading into AWS is free/cheap. You pay per GB of data downloaded, which is not cheap. You can see why, from a sales perspective: AWS' customers generally charge their customers for data they download - so they are extracting a % off that. And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.
- belter 11mo ago> And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances. Please get some training...and stop spreading disinformation. And to think on this thread only my posts are getting downvoted.... "Free data transfer out to internet when moving out of AWS" - https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...
- hexbin010 11mo agoI don't appreciate your disinformation accusation nor your tone. People are trying to tell you something with the downvotes. They're right.
- array_key_first 11mo agoIt's not disinformation at all, there's a lot of hurdles to this. In the link you posted, it even says Amazon can't actually tell if you're leaving AWS or not so they're going to charge you the regular rate. You need explicit approval from them to get this 'free' data transfer.
- 11mo ago
- dabiged 11mo agoI made the same mistake and blew $60k. I have never understood why the S3 endpoint isn't deployed by default, except to catch people making this exact mistake.
- rikafurude21 11mo agoThats a year salary but hey think about how much more complicated your work would be if you had to learn to self-host your infra!
- sixtyj 11mo agoText je srozumitelný, ale angličtina je neuhlazená. Funkční verze: Cloud cult was successfully promoted by all major players, and people have completely forgotten about the possibilities of traditional hosting. But when I see a setup form for an AWS service or the never-ending list of AWS offerings, I get stuck almost immediately.
- antonvs 11mo agoThis is a non sequitur. I know how to self host my infra, but I’ve been using cloud services for the last 15 years because it means I don’t have to deal with self hosting my infra. It runs completely by itself (mostly managed services, including k8s) and the only time I need to deal with it is when I want to change something.
- 11mo ago
- krystalgamer 11mo agoAh, the good old VPC NAT Gateway. I was lucky to have experienced all of the same mistakes for free (ex-Amazon employee). My manager just got an email saying the costs had gone through the roof and asked me to look into it. Feel bad for anyone that actually needs to cough up money for these dark patterns.
- mgaunard 11mo agoPersonally I don't even understand why NAT gateways are so prevalent. What you want most of the time is just an Internet gateway.
- Hikikomori 11mo agoOnly works in public subnets, which isn't what you want most of the time.
- deleted 11mo ago[deleted]
- mgaunard 11mo agoIf you want to avoid any kind of traffic fees, simply don't allow routing outside of your VPC by default.
- belter 11mo ago[flagged]
- wiether 11mo agoThere's nothing to gain in punching down They made a mistake and are sharing it for the whole word to see in order to help others avoid making it. It's brave. Unlike punching down.
- belter 11mo agoThis has nothing about punching down. Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. Its like when Google bought thousands of servers without ECC memory, but felt they were so smart they could not resist telling the world how bad that was and writing a paper about it...Or they could have hired some real hardware engineers from IBM or Sun...
- Nevermark 11mo ago> Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. You realize they didn’t ask you to read their article right? They didn’t put it on your fridge or in your sandwich. Policing who writes what honest personal experience on the Internet is not a job that needs doing. But if you do feel the need to police, don’t critique the writer, but HN for letting interested readers upvote the article here, where it is of course, strictly required reading. I mean, drill down to the real perpetrators of this important “problem”!
- belter 11mo agoThey are doing Enterprise sales and were founded in 2014...These type of technical blogs are normally written to demonstrate the company internal expertise, demonstrate skills of the engineering team to motive hiring etc... Does this inspire confidence?
- 11mo ago
- andrewstuart 11mo agoWhy are people still using AWS? And then writing “I regret it” posts that end up on HN. Why are people not getting the message to not use AWS? There’s SO MANY other faster cheaper less complex more reliable options but people continue to use AWS. It makes no sense.
- chistev 11mo agoExamples?
- andrewstuart 11mo agoOf what?
- wiether 11mo ago> faster cheaper less complex more reliable options
- andrewstuart 11mo agoAllow me to google that for you….. https://www.ionos.com/servers/cloud-vps https://www.ionos.com/servers/cloud-vps $22/month for 18 months with a 3-year term 12 vCores CPU 24 GB RAM 720 GB NVMe Unlimited 1Gbps traffic
- wiether 11mo agoAWS is not just EC2 And even EC2 is not just a VPS If you need a simple VPS, yes, by all means, don't use AWS. For this usecase AWS is definitely not cheaper nor simpler. Nobody said that. Ever.
- andrewstuart 11mo agoThey’re Linux computers. Anything AWS does you can run on Linux computers. It’s naive to think that AWS is some sort of magically special system that transcends other networked computers, out of brand loyalty. That’s the AWS kool aid that makes otherwise clever people think there’s no way any organization can run their own computer systems - only AWS has the skills for that.
- V__ 11mo agoJust curious but if you are already on Hetzner, why not do the processing also there?
- gizzlon 11mo agohttps://news.ycombinator.com/item?id=45978308 https://news.ycombinator.com/item?id=45978308
- Havoc 11mo agoThese sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there being accounts like that in existence already (some azure accounts are hardcapped by amount but ofc that’s not loudly advertised).
- sofixa 11mo agoIt's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate internally, and things will stop working and take forever to fix. It's easier to waive cost overages than deal with any of that.
- ed_elliott_asc 11mo agoLet people take the risk - somethings in production are less important than others.
- arjie 11mo agoThey have all the primitives. I think it's just that people are looking for a less raw version than AWS. In fact, perhaps many of these users should be using some platform that is on AWS, or if they're just playing around with an EC2 they're probably better off with Digital Ocean or something. AWS is less like your garage door and more like the components to build an industrial-grade blast-furnace - which has access doors as part of its design. You are expected to put the interlocks in. Without the analogy, the way you do this on AWS is: 1. Set up an SNS queue 2. Set up AWS budget notifications to post to it 3. Set up a lambda that watches the SNS queue And then in the lambda you can write your own logic which is smart: shut down all instances except for RDS, allow current S3 data to remain there but set the public bucket to now be private, and so on. The obvious reason why "stop all spending" is not a good idea is that it would require things like "delete all my S3 data and my RDS snapshots" and so on which perhaps some hobbyist might be happy with but is more likely a footgun for the majority of AWS users. In the alternative world where the customer's post is "I set up the AWS budget with the stop-all-spending option and it deleted all my data!" you can't really give them back the data. But in this world, you can give them back the money. So this is the safer one than that.
- ryanjshaw 11mo agoAs a bootstrapped dev, reading stories like these gives me so much anxiety. I just can’t bring myself to use AWS even despite its advantages.
- thecodemonkey 11mo agoWe are also 100% customer-funded. AWS makes sense for us for the enterprise version of Geocodio where we are SOC2 audited and HIPAA-compliant. We are primarily using Hetzner for the self-serve version of Geocodio and have been a very happy customer for decades.
- abigail95 11mo agoWhat is a bootstrapped dev?
- jabroni_salad 11mo agoIt means you are self funded and do not have a pile of other people's money to burn.
- abigail95 11mo agoI would guess that's most AWS accounts. I have my 5 personal accounts all on one debit card. I learned AWS the same way most "bootstrapped" people do, with the free tier. Maybe it's more of a minefield than it was a decade ago.
- themafia 11mo agoThe documentation is thick but it has a common theme and format to it. So once you get the hang of finding the "juicy bits" you can usually locate them anywhere. The docs do generally warn you of these cases, or have a whole "best practices" section which highlights them directly. The key is, do not make decisions lightly in the cloud, just because something is easy to enable in the UI does not mean it's recommended. Sit down with the pricing page or calculator and /really/ think over your use case. Get used to thinking about your infrastructure in terms of batch jobs instead of real time and understand the implementation and import of techniques like "circuit breakers." Once you get the hang of it it's actually very easy and somewhat liberating. It's really easy to test solutions out in a limited form and then completely tear them down. Personally I'm very happy that I put the effort in.
- Hikikomori 11mo agoSaved >120k/month by deploying some vpc endpoints and vpc peering (rather than tgw).
- denvrede 11mo agoVPC peering becomes ugly fast, once your network architecture becomes more complex. Because transitive peering doesn't work you're building a mesh of networks.
- Hikikomori 11mo agoCan just use both, tgw by default and add peering where you have heavy traffic. Did this while managing 1k+ VPCs.
- 4gotunameagain 11mo agoI'm still adamant about the fact that the "cloud" is a racket. Sure, it decreases the time necessary to get something up running, but the promises of cheaper/easier to manage/more reliable have turned out to be false. Instead of paying x on sysadmin salaries, you pay 5x to mega corps and you lose ownership of all your data and infrastructure. I think it's bad for the environment, bad for industry practices and bad for wealth accumulation & inequality.
- lan321 11mo agoI'd say it's a racket for enterprise but it makes sense for small things. For example, a friend of mine, who's in a decent bit of debt and hence on the hunt for anything that can make some money, wanted to try making essentially a Replika clone for a local market and being able to rent an H100 for 2$ an hour was very nice. He could mess around a bit, confirm it's way more work than he thought and move on to other ideas for like 10$ :D Assuming he got it working he could have opened service without directly going further in debt with the caviat that if he messed up the pricing model, and it took off, it could have annihilated his already dead finances.
- 4gotunameagain 11mo agoBut that is not what is usually referred to as the cloud. I am not against infrastructure for rent like VPS or an H100 node, but against the behemoths of AWS, Azure and the like
- stef25 11mo agoMade a similar mistake like this once. While just playing around to see what's possible I upload some data to the AWS algo that will recommended products to your users based on everyone's previous purchases. I uploaded a small xls with uid and prodid columns and then kind of forgot about it. A few months later I get a note from bank saying your account is overdrawn. The account is only used for freelancing work which I wasn't doing at the time, so I never checked that account. Looks like AWS was charging me over 1K / month while the algo continuously worked on that bit of data that was uploaded one time. They charged until there was no money left. That was about 5K in weekend earnings gone. Several months worth of salary in my main job. That was a lot of money for me. Few times I've felt so horrible.
- nine_k 11mo agoI worked in a billing department, and learned to be healthily paranoid about such things. I want to regularly check what I'm billed for. I of course check all my bank accounts' balances at least once a day. All billing emails are marked important in my inbox, and I actually open them. And of course I give every online service a separate virtual credit card (via privacy dot com, but your bank may issue them directly) with a spend limit set pretty close to the expected usage.
- auggierose 11mo agoAre there any cloud providers that allow a hard cap on dollars spent per day/week/month? Should there not be a law that they have to?
- torginus 11mo ago> I've been using AWS since around 2007. Back then, EC2 storage was entirely ephemeral and stopping an instance meant losing all your data. The platform has come a long way since then. Personally I miss ephemeral storage - having the knowledge that if you start the server from a known good state, going back to that state is just a reboot away. Way back when I was in college, a lot of out big-box servers worked like this. You can replicate this on AWS with snapshots or formatting the EBS volume into 2 partitions and just clearing the ephemeral part on reboot, but I've found it surprisingly hard to get it working with OverlayFS
- fergie 11mo agoIs it possible for hobbyists to set a hard cut off for spending? Like, "SHUT EVERYTHING DOWN IF COSTS EXCEED $50"
- conception 11mo agoYes, but you have to program it. And there is a little bit of whack so it might be $51 or something like that.
- Raed667 11mo agomy understanding from reading this kind of threads is that there is no real way to enforce it and the provider makes no guarantees, as your usage can outpace the system that is handling the accounting and shutoff
- rileymat2 11mo agoThat sounds like an architecture choice? One that would cause less revenue on the AWS side, with a conflicting incentive there.
- tacker2000 11mo agoto be fair, im not sure its a conscious choice, since its not really easy to couple lets say data transfer bytes directly to billing data in real time, and im sure that would also use up a lot of resources. But of course, the incentive to optimize this is not there.
- wulfstan 11mo agoThis happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat (https://fck-nat.dev/v1.3.0/ https://fck-nat.dev/v1.3.0/) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every old-school Linuxer should be able to do in their sleep.)
- withinboredom 11mo agoOr just run bare metal + garage and call it a day.
- perching_aix 11mo agoI personally prefer to just memorize the data and recite it really quickly on-demand. Only half-joking. When something grossly underperforms, I do often legitimately just pull up calc.exe and compare the throughput to the number of employees we have × 8 kbit/sec [0], see who would win. It is uniquely depressing yet entertaining to see this outperform some applications. [0] spherical cow type back of the envelope estimate, don't take it too seriously; assumes a very fast 200 wpm speech, 5 bytes per word, and everyone being able to independently progress
- luhn 11mo ago8kbit/min, you mean.
- perching_aix 11mo agoOh yeah lol, whoops. Still applies sadly.
- iso1631 11mo agoOr colocate your bare metal in two or three data centres for resilience against environmental issues and single supplier.
- tlaverdure 11mo agoAbolish NAT Gateways. Lean on gateway endpoints, egress only internet gateways with IPv6, and security groups to batten down the hatches. All free.
- agwa 11mo agoNow that AWS charges for public IPv4 addresses, is it still free if you need to access IPv4-only hosts?
- tlaverdure 11mo agoYeah not free if you definitely need IPv4. AWS has been adding a lot more IPv6 support to their services so hopefully the trend continues in AWS and the broader industry. You can probably get pretty far though if your app doesn't have hard requirements to communicate with IPv4 only hots.
- whalesalad 11mo agoWait till you encounter the combo of gcloud parallel composite uploads + versioning + soft-delete + multi-region bucket - and you have 500TB of objects stored.
- deleted 11mo ago[deleted]
- lapcat 11mo ago> AWS's networking can be deceptively complex. Even when you think you've done your research and confirmed the costs, there are layers of configuration that can dramatically change your bill. Unexpected, large AWS charges have been happening for so long, and so egregiously, to so many people, including myself, that we must assume it's by design of Amazon.
- lloydatkinson 11mo agoI can’t see this as anything but on purpose
- AmbroseBierce 11mo agoImagine a world were Amazon was forced to provide a publicly available report were they disclose how many clients have made this error -and similar ones- and how much money they have made from it. I know nothing like this will ever exist but hey, is free to dream.
- siliconc0w 11mo agoIt used to be that you could whine to your account rep and they'd waive sudden accidental charges like this. Which we did regularly due to all the sharp edges. These days I gather it's a bit harder.
- cobolcomesback 11mo agoThis wouldn’t have specifically helped in this situation (EC2 reading from S3), but on the general topic of preventing unexpected charges from AWS: AWS just yesterday launched flat rate pricing for their CDN (including a flat rate allowance for bandwidth and S3 storage), including a guaranteed $0 tier. It’s just the CDN for now, but hopefully it gets expanded to other services as well. https://news.ycombinator.com/item?id=45975411 https://news.ycombinator.com/item?id=45975411
- joshtbradley 11mo agoI did this when I was ~22 messing with infra for the first time. A $300 bill in two days when I had $2000 in the bank really stung. I love AWS for many things, but I really wish they made the cost calculations transparent for beginners.
- kevmo 11mo agoI wonder why they don't...
- mooreds 11mo agoAlways always set up budget alarms. Make sure they go to an list with multiple people on it. Make sure someone pays attention to that email list. It's free and will save your bacon. I've also had good luck asking for forgiveness. One time I scaled up some servers for an event and left them running for an extra week. I think the damage was in the 4 figures, so not horrendous, but not nothing. An email to AWS support led to them forgiving a chunk of that bill. Doesn't hurt to ask.
- StratusBen 11mo agoEvergreen relevant blog post: "Save by Using Anything Other Than a NAT Gateway" https://www.vantage.sh/blog/nat-gateway-vpc-endpoint-savings https://www.vantage.sh/blog/nat-gateway-vpc-endpoint-savings Also as a shameless plug: Vantage covers this is exact type of cost hiccup. If you aren't already using it, we have a very generous free tier: https://www.vantage.sh/ https://www.vantage.sh/
- dylan604 11mo agoHad the exact same thing happen. Only we used a company vetted/recommended by AWS to set this up for us, as we have no AWS experts and we're all too busy tasked doing actual startup things. So we staffed it out. Even the "professionals" get it wrong, and we racked up a huge expense as well. Staffed out company shrugged shoulders, and then just said sorry about your tab. We worked with AWS support to correct situation, and cried to daddy AWS account manager for a negotiated rate.
- maciekkmrk 11mo agoAn entire blog article post to say "read the docs and enable VPC S3 endpoint". It's all in the docs: https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.... >There is another type of VPC endpoint, Gateway, which creates a gateway endpoint to send traffic to Amazon S3 or DynamoDB. Gateway endpoints do not use AWS PrivateLink, unlike the other types of VPC endpoints. For more information, see Gateway endpoints. Even the first page of VPC docs: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html https://docs.aws.amazon.com/vpc/latest/userguide/what-is-ama... >Use a VPC endpoint to connect to AWS services privately, without the use of an internet gateway or NAT device. The author of the blog writes: > When you're using VPCs with a NAT Gateway (which most production AWS setups do), S3 transfers still go through the NAT Gateway by default. Yes, you are using a virtual private network. Where is it supposed to go? It's like being surprised that data in your home network goes through a router.
- jairuhme 11mo ago> An entire blog article post to say "read the docs and enable VPC S3 endpoint". I think it's okay if someone missed something in the docs and wanted to share from their experience. In fact, if you look at the the s3 pricing page [0], under Data Transfer, VPC endpoints are mentioned at all. It simply says data transfer is free between AWS services in the same region. I think that much detail would be enough to reasonably assume you didn't have to set up additional items to accomplish. [0]https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- kidsil 11mo agoGreat write-up, thanks for sharing the numbers. I get pulled into a fair number of "why did my AWS bill explode?" situations, and this exact pattern (NAT + S3 + "I thought same-region EC2→S3 was free") comes up more often than you’d expect. The mental model that seems to stick is: S3 transfer pricing and "how you reach S3" pricing are two different things. You can be right that EC2→S3 is free and still pay a lot because all your traffic goes through a NAT Gateway. The small checklist I give people: 1. If a private subnet talks a lot to S3 or DynamoDB, start by assuming you want a Gateway Endpoint, not the NAT, unless you have a strong security requirement that says otherwise. 2. Put NAT on its own Cost Explorer view / dashboard. If that line moves in a way you didn’t expect, treat it as a bug and go find the job or service that changed. 3. Before you turn on a new sync or batch job that moves a lot of data, sketch (I tend to do this with Mermaid) "from where to where, through what, and who charges me for each leg?" It takes a few minutes and usually catches this kind of trap. Cost Anomaly Detection doing its job here is also the underrated part of the story. A $1k lesson is painful, but finding it at $20k is much worse.
- blutoot 11mo agoRegardless of the AWS tech in question (and yes VPCE for non-compute services is a very common pattern in an enterprise setup using AWS since VPC with NAT is a pretty fundamental requirement), I honestly believe this was the biggest miss from the author: “Always validate your assumptions. I thought "EC2 to S3 is free" was enough. I should have tested with a small amount of data and monitored the costs before scaling up to terabytes.” To me this is a symptom of DevOps/infra engineers being too much in love with infra automation without actually testing the full end to end flow.
- citizenpaul 11mo agoIts staggering to me that after all this time there are somehow still people in potions like this that are working without basic cost monitoring alerts on cloud/SaaS services It really shows the Silicon Vally disconnect with the real world, where money matters.
- abujazar 11mo ago$1000 for 20 TB of data transfer sounds like fraud. You can get a VM instance with 20 TB included INTERNET traffic at Hetzner for €4.15.
- lowbloodsugar 11mo agoI’m sure NAT gateways exist purely to keep uninformed security “experts” at companies happy. I worked at a Fortune 500 company but we were a dedicated group building a cloud product on AWS. Security people demanded a NAT gateway. Why? “Because you need address translation and a way to prevent incoming connections”. Ok. That’s what an Internet Gateway is. In the end we deployed a NAT gateway and just didn’t setup routes to it. Then just used security groups and public IPs.
- knowitnone3 11mo agoThat's a loophole AWS needs to close
- Fokamul 11mo agoThe lesson: Don't use AWS
- throwawayffffas 11mo ago> The solution is to create a VPC Gateway Endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT Gateway entirely. The solution is to move your processing infrastructure to Hetzner.
- bpiroman 11mo agoSo happy I don't use AWS
- mikesickler 11mo agoGot killed by AWS Macie. The default 5K cap is brutal
- deleted 11mo ago[deleted]
- sprybear 11mo agoI'm not telling anyone to stay away from AWS, but I've heard far too many similar stories to feel comfortable recommending it.