28 ms·

As AWS Use Soars, Companies Surprised by Cloud Bills

155 points by kaboro 8y ago

jak92 8y ago
Paywall, cannot read article past third paragraph.
- deleted 8y ago
  [deleted]
- mitchdoogle 8y ago
  Seriously, anything that is not fully readable without going through additional hoops should be removed, or not even posted in the first place.
- vernie 8y ago
  Gotta recoup those cloud hosting costs somehow!
- okbake 8y ago
  I'm curious if all of these comments here are people who paid to read it or just extrapolating off a the headline.
  - dang 8y ago
    This comment breaks the site guidelines, which ask you not to insinuate astroturfing without evidence. Please don't post like that here; it's a toxic trope that people are nearly always imagining and projecting onto one another. Mind you, your post was far from the worst example of this—but it still needs nipping in the bud. In this case, the history of the users commenting is enough to make it overwhelmingly unlikely that they were paid, and the well-known propensity of users to comment just on a title clinches it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - okbake 8y ago
      Sorry, I didn't mean to imply astroturfing. My comment is asking how many people "paid to read the article" (because it's behind a paywall), and not "were paid to comment here". Still not a great comment I admit, but with the amount of discussion going on I would be surprised if everyone is subscriber or if they are going off of the headline alone.
twunde 8y ago
I think that everyone who's used AWS or a competitor knows that it's very easy to rack up a large bill by accident. For companies, the main issue isn't the expense but the unpredictability.
- WrtCdEvrydy 8y ago
  The use is that most people use AWS like if it was a datacenter they don't own. If you're going for AWS, you should consider rewriting a lot of your services to use AWS features well. Auto Scaling Groups and Fargate EC2 are some common components I see few companies using. If your instances are the same size through the day, you are doing it wrong!* *Exception: If you provide the same level of service and traffic 24/7.
  - exabrial 8y ago
    This is [unfortunately] correct. If you marry your company's destiny to AWS, their managed products are far more cost-effective than using them as a datacenter [no doubt this is intentional by Amazon]. If you're just using the EC2 services, much like owning the hardware, unused CPU and RAM is wasted money.
  - breckenedge 8y ago
    I recently gave Fargate a try and was very unimpressed with the costs. Anecdata is terrible, but my experience running 1 Fargate container 24x7 on the lowest specs (0.25 CPU and 512 MB RAM) to handle baseline was was going to cost as much as a no-contract T3.micro EC2 instance with much more capacity (2 CPU and 1 GB RAM). AWS Kubernetes was also a bust at $120/mo just to get started (that's the cost before an EC2 server is provisioned).
    - WrtCdEvrydy 8y ago
      Fargate has two flavors, ECS or EC2. EC2 is just EC2 with scaling groups while ECS is a fully non-managed solution that's for small items.
      - 8y ago
SEJeff 8y ago
Just wanted to make a shoutout to Corey Quinn, who has a consultancy which literally works on lowering your excessive AWS bills. He also runs the most excellent Last Week in AWS newsletter https://lastweekinaws.com/ https://lastweekinaws.com/ and podcast Screaming in the Cloud: https://www.screaminginthecloud.com https://www.screaminginthecloud.com I've personally known Corey for 10+ years and he's a good dude, so if anyone is looking to lower their AWS bills, talk to Corey.
- crescentfresh 8y ago
  > literally works on lowering your excessive AWS bills these companies are a dime a dozen, no personal offense intended to yourself or this company.
  - lathiat 8y ago
    The fabulous level of trolling and entertainment he offers on his twitter account however is priceless.
    - SEJeff 8y ago
      And both the newsletter + podcast are quite fantastic. He really is unrivaled in the space. If you use AWS, you should be keeping tabs on those things. Bonus points that they're free and amusing. I know him from back when I was helping maintain the Saltstack python config management tool[1]. He was a user and we were doing training on contributing code to salt. It turned out that I did teach him a thing or two and he did several contributions to salt[2]. That's literally it. [1] https://github.com/saltstack/salt https://github.com/saltstack/salt [2] https://github.com/saltstack/salt/commits?author=quinnypig https://github.com/saltstack/salt/commits?author=quinnypig
- mychael 8y ago
  Your comment doesn't contribute to the discussion. It's more of an advertisement than a comment.
  - 8y ago
brootstrap 8y ago
After years of working with AWS this is not surprising. You can have developers with minimal sys admin / dev ops building instances and running jobs. Back in early days of my startup I recall my team launching a big model run and leaving for weekend. We can back on monday to a 10k bill and thousands of compute hours... That did not go well, 10k on a few days of compute for a small startup haha! Still to this day we've got tons of EC2 instances lying around racking up charges. We dont have the devops support to build proper auto-scaling, making things really fit AWS. We just needed a server to run 24/7 and this was easy. Can only imagine the insanity that goes on at huge Pinterest level scale. How many EC2 instances do they have? Thousands? Hundreds of thousands?
- wolfi1 8y ago
  if you just have webservers, api gateway + lambda is the way to go Edit: to make myself clear, I meant the backend, frontend, of course, should be served like below
  - WrtCdEvrydy 8y ago
    Cloudfront + S3 for static sites. I've seen so many people just create an EC2 instance for serving static content. If you have a banging new React app that generates static files, you don't need anything more.
    - wolfi1 8y ago
      of course, I meant it backend-wise
  - meritt 8y ago
    lambda gets incredibly expensive for anything that approaches even modest usage levels. Once you exceed about 10 requests/second, you'd be better off with the lowest tier virtualized instances ($2 - $5/mo from most providers including AWS)
    - RhodesianHunter 8y ago
smallgovt 8y ago
I was always incredulous at how many free credits AWS hands out to startups (up to $100K per startup), but then every time I see an article like this I'm reminded how large the recurring infrastructure bill can be for a successful business. Such a great business for Amazon.
- ceejayoz 8y ago
  It's quite clever, too, as it likely encourages setups that are pretty inefficient on long-term AWS spend.
  - ams6110 8y ago
    It's like the apocryphal drug dealer who gives out free doses to unwitting kids until they are hooked.
    - genericone 8y ago
      This article is one of the first shots fired in the 'War on Clouds'.
- tome 8y ago
  Hmm, it's like early stage funding and then reaping the benefits from the small number who make it to unicorn level ...
- h1d 8y ago
  I thought they make more than the shopping division.
cobookman 8y ago
A lot of people don't realize that Reserved Instnaces don't show up on your monthly invoice. So once every 3 years you're hit by a large bill for RI renewal.
- neuronflux 8y ago
  You can choose how to setup your billing for reserved instances. > You can choose to pay for your Reserved Instance upfront, partially upfront, or monthly source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts-reserved-instances-application.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts...
  - cobookman 8y ago
    You get the best discount by paying fully upfront. If you do this, the AWS bill doesn't show the RI on your monthly bill what-so-ever. Leading to a suprise 3 years down the road.
    - user5994461 8y ago
      And more importantly you're engaged to pay for the full year no matter which option you pick, with (almost) no possibility to cancel. Long story short, only play 1 year full upfront.
- dba7dba 8y ago
  "Reserved Instances" is a poison that every AWS user must be aware of. Figuring out how/if it's used is really convoluted. And getting out early is a pain.
jdblair 8y ago
When I joined a particular startup 4 years ago I was surprised that they owned their own servers mounted in racks in a rented colo. I was all, how retro! But then the head of IT showed me the cost savings over AWS and how he orchestrated it all using VMWare images, and I saw he had a point. He saved more than enough money to pay for his full time salary to keep it all going, even dealing with hardware maintenance. What was unique was that the company had a predictable load that didn't change much. If we had needed to change capacity level drastically over a regular cycle then AWS would have started to make sense.
- lykr0n 8y ago
  This. If you can predict your workload with some accuracy, and have someone who knows how to design systems- you can save a lot of money. AWS' value is that "we can do everything except write your code for you" which you end up paying a lot for.
  - chomp 8y ago
    Even if you can't predict the workload too well, you can go hybrid cloud - colo for the baseline and then AWS for the unpredictable stuff. ManageIQ is great for this.
  - ekovarski 8y ago
    ... isn't that the hardest part, predicting workload, for many companies. Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. If you can predict workload then you could also reduce your cloud bills, in theory, by moving spinning up/down instances. Many are going the cloud route because the cost of entry is much easier to start developing but eventually, yes, the cost skyrockets and you do end up paying for putting little forethought into the real workload requirements.
    - toomuchtodo 8y ago
      > Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. One senior sysadmin/infrastructure engineer with on-prem experience is enough to scale up your colo presence. You can either assemble your own servers or custom order and have them shipped directly to the colo. Remote console and power management (PDUs) is a solved problem. Out of band management, also solved. Virtualization, solved. Push your VM snapshots, content, artifacts, and database backups into your object store of choice for offsite backups. Ideally, your colo facility is within driving distance of your office if you absolutely need physical access. Otherwise, you pay someone at the colo a few hundred bucks a month for remote hands services. EDIT: k8s arguably makes this easier than in the past. You want to run on your own cluster? You can. You need to burst your containers to spot instances in someone else's cloud? You can.
sasasassy 8y ago
I personally feel many things in AWS are purposely made in order to hide the real costs from you. For instance how you are charged for S3 traffic your buckets receive even if you don't have any files there. Or how traffic between zones that is outside of your control is still charged. Or how things like AZ replication cost a ton and you have no metric to show if it even makes sense to enable them. Heck, even usage alerts cost money.
- paulfurtado 8y ago
  > For instance how you are charged for S3 traffic your buckets receive even if you don't have any files there. Or how traffic between zones that is outside of your control is still charged. Can you say a bit more about these two things? When do you get charged for s3 traffic on a bucket with no files? What do you define as uncontrollable cross zone traffic here?
  - extra88 8y ago
    Maybe they mean something else but I expect a GET request is $0.0004 per 1,000 requests (i.e. $0.40 per million requests) regardless of whether it returns a file (200) or not (404). Sounds more like a nuisance than a financial risk.
- hartator 8y ago
  Yeah S3 traffic bills are always surprising. It’s probably 2x or even 3x more than the actual storage cost for our use case. And we have caching mechanisms on our end.
- Dunedan 8y ago
  S3 traffic cost is a funny topic. I remember ~3 years ago we had an issue with AWS, where they kept charging us for web traffic to an S3 bucket we deleted. When we complained, their response was like: Well, then stop sending traffic to that (non-existing) bucket. That of course wasn't going to happen as the URL was out in the wild and got requested quite often automatically by third-party services.
  - 8y ago
ham_sandwich 8y ago
With the rise of lambda+managed services, I think we’ll start seeing finance and development start to blend and merge. Besides giving businesses more legibility into what specific parts of their business logic cost to operate vs. the value they generate, you can start building higher-order financial systems based on flows of capital+information within businesses. From there you can implement all sorts of financial engineering like insurance and options+derivatives that could allow businesses to do things like dial up leverage against these flows. Certainly half-baked ideas, but fun to think about the possibilities.
- jacques_chester 8y ago
  I'm moderately hopeful that the basic idea (close association between activity and cost) will improve, but I have some caveats. One is that not everything will fit. Lots and lots and lots of workloads exist in their current form and are not going to switch overnight. Whatever bookkeeping method that gets developed needs to deal with the mix of costing models. Another is that you still need to assign costs. In a lot of companies this is the subject of intense corporate politics. Whether such a method is adopted will depend on who is wielding what cudgel. Related to which: there will be cases of premature optimisation on the visible. Optimising for cost is fine, but costs often include estimates that can be overlooked. It's one thing to optimise for "least dollars spent per invocation". Another thing to optimise for "fewest pissed off customers". The latter is harder to measure but in many cases more important. But overall? Yes. I think it could be a step forward.
dalbasal 8y ago
Pay-for-what-you-use pricing has the ability to escalate. You don't buy a server for $30k, and go through whatever procurement paperwork is set up for that. You just increase average usage by here and there. There's no procurement hurdle, so it can be a blank check. ..just like it's easier to get I to debt with a credit card than a personal loan. Credit card debt mounts one purchase at a time. Personal loans require you to decide to borrow €X.
- dba7dba 8y ago
  Reminds me of reading how US Armed forces in Iraq being forced to buy NEW printers because their procurement system didn't allow buying toners to replace used up toners in practically new printers. MBAs/Accountants...
spectre256 8y ago
The two most insidious things I've found in AWS billing: - If you want the AWS dashboard metrics to have 1 minute instead of 5 minute granularity, that's $2.10 per instance per month. 5 minutes is pretty useless, so either you set up other monitoring or you pay a tax on the number of instances you have. - If you use AMIs (which you probably should) to launch EC2 instances with all your software baked in already, you will probably end up with dozens or hundreds of old, unused AMIs. Furthermore each of these AMIs is linked to a snapshot, which is stored on S3. S3 pricing is very cheap but it's a significant amount of work to determine which AMIs are no longer in use and to delete both the AMI and the corresponding snapshot. Every 100 AMIs you have at the standard 8GB root volume size costs you $18/month.
- Someone1234 8y ago
  > If you use AMIs (which you probably should) to launch EC2 instances with all your software baked in already, you will probably end up with dozens or hundreds of old, unused AMIs. We previously had this issue. What we do now is instead of leaving old ones on S3, we download and archive them. We just tag them on the dashboard and a nightly script does the rest. In general tagging is an extremely powerful way to organize your AMIs. This article[0] has good examples of tagging you can follow. We use something akin to the "Business Tag" strategy, with an Excel document. Definitely requires some internal organisation but the cost savings speak for themselves. [0] https://aws.amazon.com/answers/configuration-management/aws-ami-design/ https://aws.amazon.com/answers/configuration-management/aws-...
EGreg 8y ago
“The Cloud” is a euphemism for extreme centralization of web and other hosting.
- mbrumlow 8y ago
  I joined this company to build out a software stack for being able to do disaster recovery of sites off site. This resulted in building a system that takes incremental backups of client systems and uploading them to our servers. In a DR event we would spin up VMs backed by the images we had copied to our servers. Throughout the processes the word "cloud" was brought up. Who's cloud would we use? Over and over again I tried to explain we were actually building a "cloud" service. And by that I mean we are just offering a service that runs on our servers. Due to the feature set and cost we ended up settling on bare metal servers from SoftLayer (IBM). The entire solutions has been made to run on commodity hardware. For the longest time the company kept marking that we were using "IMBs Cloud". Every now and then I get a request to ask what it would take to move to AWS. My response is always the same. More operating budget, and less features. We spin up DR environments for many servers2 in seconds -- and can do so because we have full control over the hypervisor, so moving to anything like AWS, Azure, or what not means we give up the ability for near instance restore. Owning the entire stack has its own set of problems, but in the long run we should be able to move much faster. "The Cloud" -> "The Internet"
pavelmark 8y ago
Paywall.
josh_carterPDX 8y ago
I just joined a new company who is doing nothing but helping companies figure out how to save more money on their cloud costs. This is such a timely article. The same costs that are being used to justify the value of the cloud are now being used to justify going back to buying servers. Which is the wrong solution, by the way. With anything it just comes down to education and best practices. People need to utilize the incredible tools out there along with understanding better how the cost structures work.
- NightlyDev 8y ago
  Servers the wrong solution? Maybe in a few situations. AWS is just too expensive. Eg. 600 TB data out and 100 TB in is $37300 on AWS, and by going with colocation a price of < $500 is very achieveable. That's ~1/75th of the cost!!! There is nothing to understand here, except that AWS in insanely expensive and colocation is a way better option in this situation.
  - rstupek 8y ago
    I agree that the price is high but you're not comparing the same things. Is the $500 a month DDOS protected bandwidth with as much redundancy as AWS is providing? I definitely shift as much bandwidth away from AWS as I can with any solution I make but they do provide more than a dumb pipe
    - josh_carterPDX 8y ago
      Ditto. The other point to make here is that AWS is not the only game in town. I agree they're not an inexpensive option, but there are even some small "mom and pop" cloud providers that are cheaper and easier to use than AWS. However, until they stop gobbling up market share there will always be ancillary markets that will work to support those that invest in their services.
    - JoshTriplett 8y ago
      > Is the $500 a month DDOS protected bandwidth with as much redundancy as AWS is providing? Sometimes you don't need that. AWS S3 provides reduced-redundancy options for less cost. It'd be nice if the same applied to other services.
jacques_chester 8y ago
Where I work there's a monthly report sent out that shows what each team is spending. Periodically teams poke around their infrastructure and remove anything they don't need any more. A few times there have been company-wide "spring cleaning" days. That's all it has taken to save a lot of money.
- user5994461 8y ago
  Same. It's pretty amazing to be able to see where the money is going and adjust anytime.
incomplete 8y ago
i honestly cannot believe that this is still a thing! well, actually, i can. :( i run a large AWS consolidated billing family for a sizeable computer science graduate research lab and tracking our AWS spends are an absolute nightmare. if you think about it for a minute, why would it be in the best interests of amazon/AWS to allow us granular, easy to parse and (most importantly) easy to customize and use billing reports? they're in the business of making money, and providing mechanisms for folks to limit their spends will limit profits. another thing to consider is that the billing subsystems for all of these cloud providers are literally one of the first things to be engineered, and after release, one of the last to ever be updated. for instance, it took amazon two years to release OUs (organizational units): one year in closed beta, one in open. while these are great for organizing accounts, you still can't see how much an OU spent w/o a bunch of python/boto gymnastics! i was on a call w/some of the leads of the aws billing subsystem about a year back, and asked about what the roadmap for billing features and the response was "2020 at the earliest".
- user5994461 8y ago
  What are you having troubles with? I find the billing page to be pretty good, once the tags are configured for billing. Far better than any internal tool I had the chance to work with.
  - incomplete 8y ago
    we have a lot of projects, and i use OUs to track the N-ary tree of lab->project->people and their spends. since the aws billing subsystem can't map OU->$$$, i wrote a thing: https://github.com/ucbrise/aws-audit https://github.com/ucbrise/aws-audit it'd be also super awesome to have OU-based budgets. we use teevity ice when we really need to dig deep (https://github.com/Teevity/ice https://github.com/Teevity/ice) but it's pretty much abandonware at this point. i've also explored a couple of commercial offerings (ronin & teevity) but they don't work well in our AWS research credit-based model.
    - user5994461 8y ago
deleted 8y ago
[deleted]
dba7dba 8y ago
AWS is really expensive. Even for running a small personal websites. I thought I could gain AWS experience AND run a few personal websites on them. What a mistake. I've moved off all (well like 2) my personal sites to other VPS provider, that starts at $5/month for 512MB VPS. The worst annoyance is 'Reserved instances'. You are basically signing up for a long term contract that you can't get out of easily.
- gshakir 8y ago
  S3 website hosting for static pages will be cheaper than VPS unless you are running an app.
- phonon 8y ago
  Lightsail is even cheaper, AND AWS https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/
tyingq 8y ago
Part of this is that AWS is often more expensive than on-prem. The other part, though, is that dev teams in large companies often aren't held accountable for on-prem costs. The on-prem costs are often in a big baseline number with little visibility into how much of that big number belongs to each team.
waterishail 8y ago
I'm reminded of this article - http://highscalability.com/blog/2017/12/4/the-eternal-cost-savings-of-netflixs-internal-spot-market.html http://highscalability.com/blog/2017/12/4/the-eternal-cost-s... Netflix run a spot market for instances to drive down costs for exactly the reasons mentioned here
AcerbicZero 8y ago
This is one of those predictable things, that is conveniently ignored by certain layers of a business. The more entertaining situation (from my position anyway) is the drama that the rapid Azure adoption by otherwise very non-cloud companies will cause in a year or two, when the free credits Microsoft stuffed into their ELA's go away. I love running stuff in the cloud, but without a conscious decision in an organization to prepare for, and avoid, the various pitfalls, its just a giant cost sink.
pnathan 8y ago
One of the basic tenants of engineering is "why is this feature appropriate"; much of AWS can be sliced away with this razor, much of the time. Appropriate tool for the appropriate time. A lot of the AWS systems simply are special purpose and not good for general application. I'm working on a Kubernetes configuration at home that provides the common & popular features AWS in a consistent & portable manner: blob store, database, etc.
kevinAtStorj 8y ago
I find it interesting that while the price of hard drives has decreased by about 50 percent over the past five years, the price of cloud storage has remained roughly flat. As margin continues to expand, the need for alternative models and competition in 'Cloud' is becoming increasingly apparent. https://storj.io/blog/2018/11/the-high-price-of-traditional-cloud-storage/ https://storj.io/blog/2018/11/the-high-price-of-traditional-...
- riskneutral 8y ago
  I think the end game is probably something resembling electrical grid markets. Electrical power is bought and sold in real-time by traders, as the supply and load fluctuate. There are many players in a competitive market, like nuclear plants, gas plants, solar, wind, etc. There are even complex derivatives (like “swing options”) traded in the electrical markets. Someday computing resources will be bought and sold in this way too, with a large number of players in a competitive market, which will drive down the currently inflated cloud computing costs. We can already see an embryonic version of such a market in the form of AWS “spot instances” and the AWS reserved instance secondary market. AWS is fighting tooth and nail to avoid or delay such a commoditization of computing utilities, which is why they announce 50 new vendor lock-in software products every week instead of focusing only on the hardware resource.
kemitchell 8y ago
Missed opportunity: As AWS Use Soars, Companies Surprised by Soaring Bills
nunez 8y ago
companies that think that they can treat aws like their own colo and dont have any semblance of budgeting will always be surprised by their bills. the cloud can be very expensive...it can also be significantly cheaper (think: a retailer with mostly static websites with some light javascript here and there that peaks on black friday). doing an analysis and being responsible can help here.
deepsun 8y ago
Sometimes I wonder whether AWS makes more money than GCP because average GCP users develop better optimized systems.

155 points by kaboro 8y ago

jak92 8y ago

Paywall, cannot read article past third paragraph.

deleted 8y ago
[deleted]
mitchdoogle 8y ago
Seriously, anything that is not fully readable without going through additional hoops should be removed, or not even posted in the first place.
vernie 8y ago
Gotta recoup those cloud hosting costs somehow!
okbake 8y ago
I'm curious if all of these comments here are people who paid to read it or just extrapolating off a the headline.
- dang 8y ago
  This comment breaks the site guidelines, which ask you not to insinuate astroturfing without evidence. Please don't post like that here; it's a toxic trope that people are nearly always imagining and projecting onto one another. Mind you, your post was far from the worst example of this—but it still needs nipping in the bud. In this case, the history of the users commenting is enough to make it overwhelmingly unlikely that they were paid, and the well-known propensity of users to comment just on a title clinches it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - okbake 8y ago
    Sorry, I didn't mean to imply astroturfing. My comment is asking how many people "paid to read the article" (because it's behind a paywall), and not "were paid to comment here". Still not a great comment I admit, but with the amount of discussion going on I would be surprised if everyone is subscriber or if they are going off of the headline alone.

twunde 8y ago

I think that everyone who's used AWS or a competitor knows that it's very easy to rack up a large bill by accident. For companies, the main issue isn't the expense but the unpredictability.

WrtCdEvrydy 8y ago
The use is that most people use AWS like if it was a datacenter they don't own. If you're going for AWS, you should consider rewriting a lot of your services to use AWS features well. Auto Scaling Groups and Fargate EC2 are some common components I see few companies using. If your instances are the same size through the day, you are doing it wrong!* *Exception: If you provide the same level of service and traffic 24/7.
- exabrial 8y ago
  This is [unfortunately] correct. If you marry your company's destiny to AWS, their managed products are far more cost-effective than using them as a datacenter [no doubt this is intentional by Amazon]. If you're just using the EC2 services, much like owning the hardware, unused CPU and RAM is wasted money.
- breckenedge 8y ago
  I recently gave Fargate a try and was very unimpressed with the costs. Anecdata is terrible, but my experience running 1 Fargate container 24x7 on the lowest specs (0.25 CPU and 512 MB RAM) to handle baseline was was going to cost as much as a no-contract T3.micro EC2 instance with much more capacity (2 CPU and 1 GB RAM). AWS Kubernetes was also a bust at $120/mo just to get started (that's the cost before an EC2 server is provisioned).
  - WrtCdEvrydy 8y ago
    Fargate has two flavors, ECS or EC2. EC2 is just EC2 with scaling groups while ECS is a fully non-managed solution that's for small items.
    - 8y ago

SEJeff 8y ago

Just wanted to make a shoutout to Corey Quinn, who has a consultancy which literally works on lowering your excessive AWS bills. He also runs the most excellent Last Week in AWS newsletter https://lastweekinaws.com/ https://lastweekinaws.com/ and podcast Screaming in the Cloud: https://www.screaminginthecloud.com https://www.screaminginthecloud.com I've personally known Corey for 10+ years and he's a good dude, so if anyone is looking to lower their AWS bills, talk to Corey.

crescentfresh 8y ago
> literally works on lowering your excessive AWS bills these companies are a dime a dozen, no personal offense intended to yourself or this company.
- lathiat 8y ago
  The fabulous level of trolling and entertainment he offers on his twitter account however is priceless.
  - SEJeff 8y ago
    And both the newsletter + podcast are quite fantastic. He really is unrivaled in the space. If you use AWS, you should be keeping tabs on those things. Bonus points that they're free and amusing. I know him from back when I was helping maintain the Saltstack python config management tool[1]. He was a user and we were doing training on contributing code to salt. It turned out that I did teach him a thing or two and he did several contributions to salt[2]. That's literally it. [1] https://github.com/saltstack/salt https://github.com/saltstack/salt [2] https://github.com/saltstack/salt/commits?author=quinnypig https://github.com/saltstack/salt/commits?author=quinnypig
mychael 8y ago
Your comment doesn't contribute to the discussion. It's more of an advertisement than a comment.
- 8y ago

brootstrap 8y ago

After years of working with AWS this is not surprising. You can have developers with minimal sys admin / dev ops building instances and running jobs. Back in early days of my startup I recall my team launching a big model run and leaving for weekend. We can back on monday to a 10k bill and thousands of compute hours... That did not go well, 10k on a few days of compute for a small startup haha! Still to this day we've got tons of EC2 instances lying around racking up charges. We dont have the devops support to build proper auto-scaling, making things really fit AWS. We just needed a server to run 24/7 and this was easy. Can only imagine the insanity that goes on at huge Pinterest level scale. How many EC2 instances do they have? Thousands? Hundreds of thousands?

wolfi1 8y ago
if you just have webservers, api gateway + lambda is the way to go Edit: to make myself clear, I meant the backend, frontend, of course, should be served like below
- WrtCdEvrydy 8y ago
  Cloudfront + S3 for static sites. I've seen so many people just create an EC2 instance for serving static content. If you have a banging new React app that generates static files, you don't need anything more.
  - wolfi1 8y ago
    of course, I meant it backend-wise
- meritt 8y ago
  lambda gets incredibly expensive for anything that approaches even modest usage levels. Once you exceed about 10 requests/second, you'd be better off with the lowest tier virtualized instances ($2 - $5/mo from most providers including AWS)
  - RhodesianHunter 8y ago

smallgovt 8y ago

I was always incredulous at how many free credits AWS hands out to startups (up to $100K per startup), but then every time I see an article like this I'm reminded how large the recurring infrastructure bill can be for a successful business. Such a great business for Amazon.

ceejayoz 8y ago
It's quite clever, too, as it likely encourages setups that are pretty inefficient on long-term AWS spend.
- ams6110 8y ago
  It's like the apocryphal drug dealer who gives out free doses to unwitting kids until they are hooked.
  - genericone 8y ago
    This article is one of the first shots fired in the 'War on Clouds'.
tome 8y ago
Hmm, it's like early stage funding and then reaping the benefits from the small number who make it to unicorn level ...
h1d 8y ago
I thought they make more than the shopping division.

cobookman 8y ago

A lot of people don't realize that Reserved Instnaces don't show up on your monthly invoice. So once every 3 years you're hit by a large bill for RI renewal.

neuronflux 8y ago
You can choose how to setup your billing for reserved instances. > You can choose to pay for your Reserved Instance upfront, partially upfront, or monthly source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts-reserved-instances-application.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts...
- cobookman 8y ago
  You get the best discount by paying fully upfront. If you do this, the AWS bill doesn't show the RI on your monthly bill what-so-ever. Leading to a suprise 3 years down the road.
  - user5994461 8y ago
    And more importantly you're engaged to pay for the full year no matter which option you pick, with (almost) no possibility to cancel. Long story short, only play 1 year full upfront.
dba7dba 8y ago
"Reserved Instances" is a poison that every AWS user must be aware of. Figuring out how/if it's used is really convoluted. And getting out early is a pain.

jdblair 8y ago

When I joined a particular startup 4 years ago I was surprised that they owned their own servers mounted in racks in a rented colo. I was all, how retro! But then the head of IT showed me the cost savings over AWS and how he orchestrated it all using VMWare images, and I saw he had a point. He saved more than enough money to pay for his full time salary to keep it all going, even dealing with hardware maintenance. What was unique was that the company had a predictable load that didn't change much. If we had needed to change capacity level drastically over a regular cycle then AWS would have started to make sense.

lykr0n 8y ago
This. If you can predict your workload with some accuracy, and have someone who knows how to design systems- you can save a lot of money. AWS' value is that "we can do everything except write your code for you" which you end up paying a lot for.
- chomp 8y ago
  Even if you can't predict the workload too well, you can go hybrid cloud - colo for the baseline and then AWS for the unpredictable stuff. ManageIQ is great for this.
- ekovarski 8y ago
  ... isn't that the hardest part, predicting workload, for many companies. Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. If you can predict workload then you could also reduce your cloud bills, in theory, by moving spinning up/down instances. Many are going the cloud route because the cost of entry is much easier to start developing but eventually, yes, the cost skyrockets and you do end up paying for putting little forethought into the real workload requirements.
  - toomuchtodo 8y ago
    > Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. One senior sysadmin/infrastructure engineer with on-prem experience is enough to scale up your colo presence. You can either assemble your own servers or custom order and have them shipped directly to the colo. Remote console and power management (PDUs) is a solved problem. Out of band management, also solved. Virtualization, solved. Push your VM snapshots, content, artifacts, and database backups into your object store of choice for offsite backups. Ideally, your colo facility is within driving distance of your office if you absolutely need physical access. Otherwise, you pay someone at the colo a few hundred bucks a month for remote hands services. EDIT: k8s arguably makes this easier than in the past. You want to run on your own cluster? You can. You need to burst your containers to spot instances in someone else's cloud? You can.

sasasassy 8y ago

I personally feel many things in AWS are purposely made in order to hide the real costs from you. For instance how you are charged for S3 traffic your buckets receive even if you don't have any files there. Or how traffic between zones that is outside of your control is still charged. Or how things like AZ replication cost a ton and you have no metric to show if it even makes sense to enable them. Heck, even usage alerts cost money.

paulfurtado 8y ago
> For instance how you are charged for S3 traffic your buckets receive even if you don't have any files there. Or how traffic between zones that is outside of your control is still charged. Can you say a bit more about these two things? When do you get charged for s3 traffic on a bucket with no files? What do you define as uncontrollable cross zone traffic here?
- extra88 8y ago
  Maybe they mean something else but I expect a GET request is $0.0004 per 1,000 requests (i.e. $0.40 per million requests) regardless of whether it returns a file (200) or not (404). Sounds more like a nuisance than a financial risk.
hartator 8y ago
Yeah S3 traffic bills are always surprising. It’s probably 2x or even 3x more than the actual storage cost for our use case. And we have caching mechanisms on our end.
Dunedan 8y ago
S3 traffic cost is a funny topic. I remember ~3 years ago we had an issue with AWS, where they kept charging us for web traffic to an S3 bucket we deleted. When we complained, their response was like: Well, then stop sending traffic to that (non-existing) bucket. That of course wasn't going to happen as the URL was out in the wild and got requested quite often automatically by third-party services.
- 8y ago

ham_sandwich 8y ago

With the rise of lambda+managed services, I think we’ll start seeing finance and development start to blend and merge. Besides giving businesses more legibility into what specific parts of their business logic cost to operate vs. the value they generate, you can start building higher-order financial systems based on flows of capital+information within businesses. From there you can implement all sorts of financial engineering like insurance and options+derivatives that could allow businesses to do things like dial up leverage against these flows. Certainly half-baked ideas, but fun to think about the possibilities.

jacques_chester 8y ago
I'm moderately hopeful that the basic idea (close association between activity and cost) will improve, but I have some caveats. One is that not everything will fit. Lots and lots and lots of workloads exist in their current form and are not going to switch overnight. Whatever bookkeeping method that gets developed needs to deal with the mix of costing models. Another is that you still need to assign costs. In a lot of companies this is the subject of intense corporate politics. Whether such a method is adopted will depend on who is wielding what cudgel. Related to which: there will be cases of premature optimisation on the visible. Optimising for cost is fine, but costs often include estimates that can be overlooked. It's one thing to optimise for "least dollars spent per invocation". Another thing to optimise for "fewest pissed off customers". The latter is harder to measure but in many cases more important. But overall? Yes. I think it could be a step forward.

dalbasal 8y ago

Pay-for-what-you-use pricing has the ability to escalate. You don't buy a server for $30k, and go through whatever procurement paperwork is set up for that. You just increase average usage by here and there. There's no procurement hurdle, so it can be a blank check. ..just like it's easier to get I to debt with a credit card than a personal loan. Credit card debt mounts one purchase at a time. Personal loans require you to decide to borrow €X.

dba7dba 8y ago
Reminds me of reading how US Armed forces in Iraq being forced to buy NEW printers because their procurement system didn't allow buying toners to replace used up toners in practically new printers. MBAs/Accountants...

spectre256 8y ago

The two most insidious things I've found in AWS billing: - If you want the AWS dashboard metrics to have 1 minute instead of 5 minute granularity, that's $2.10 per instance per month. 5 minutes is pretty useless, so either you set up other monitoring or you pay a tax on the number of instances you have. - If you use AMIs (which you probably should) to launch EC2 instances with all your software baked in already, you will probably end up with dozens or hundreds of old, unused AMIs. Furthermore each of these AMIs is linked to a snapshot, which is stored on S3. S3 pricing is very cheap but it's a significant amount of work to determine which AMIs are no longer in use and to delete both the AMI and the corresponding snapshot. Every 100 AMIs you have at the standard 8GB root volume size costs you $18/month.

Someone1234 8y ago
> If you use AMIs (which you probably should) to launch EC2 instances with all your software baked in already, you will probably end up with dozens or hundreds of old, unused AMIs. We previously had this issue. What we do now is instead of leaving old ones on S3, we download and archive them. We just tag them on the dashboard and a nightly script does the rest. In general tagging is an extremely powerful way to organize your AMIs. This article[0] has good examples of tagging you can follow. We use something akin to the "Business Tag" strategy, with an Excel document. Definitely requires some internal organisation but the cost savings speak for themselves. [0] https://aws.amazon.com/answers/configuration-management/aws-ami-design/ https://aws.amazon.com/answers/configuration-management/aws-...

EGreg 8y ago

“The Cloud” is a euphemism for extreme centralization of web and other hosting.

mbrumlow 8y ago
I joined this company to build out a software stack for being able to do disaster recovery of sites off site. This resulted in building a system that takes incremental backups of client systems and uploading them to our servers. In a DR event we would spin up VMs backed by the images we had copied to our servers. Throughout the processes the word "cloud" was brought up. Who's cloud would we use? Over and over again I tried to explain we were actually building a "cloud" service. And by that I mean we are just offering a service that runs on our servers. Due to the feature set and cost we ended up settling on bare metal servers from SoftLayer (IBM). The entire solutions has been made to run on commodity hardware. For the longest time the company kept marking that we were using "IMBs Cloud". Every now and then I get a request to ask what it would take to move to AWS. My response is always the same. More operating budget, and less features. We spin up DR environments for many servers2 in seconds -- and can do so because we have full control over the hypervisor, so moving to anything like AWS, Azure, or what not means we give up the ability for near instance restore. Owning the entire stack has its own set of problems, but in the long run we should be able to move much faster. "The Cloud" -> "The Internet"

pavelmark 8y ago

Paywall.

josh_carterPDX 8y ago

I just joined a new company who is doing nothing but helping companies figure out how to save more money on their cloud costs. This is such a timely article. The same costs that are being used to justify the value of the cloud are now being used to justify going back to buying servers. Which is the wrong solution, by the way. With anything it just comes down to education and best practices. People need to utilize the incredible tools out there along with understanding better how the cost structures work.

NightlyDev 8y ago
Servers the wrong solution? Maybe in a few situations. AWS is just too expensive. Eg. 600 TB data out and 100 TB in is $37300 on AWS, and by going with colocation a price of < $500 is very achieveable. That's ~1/75th of the cost!!! There is nothing to understand here, except that AWS in insanely expensive and colocation is a way better option in this situation.
- rstupek 8y ago
  I agree that the price is high but you're not comparing the same things. Is the $500 a month DDOS protected bandwidth with as much redundancy as AWS is providing? I definitely shift as much bandwidth away from AWS as I can with any solution I make but they do provide more than a dumb pipe
  - josh_carterPDX 8y ago
    Ditto. The other point to make here is that AWS is not the only game in town. I agree they're not an inexpensive option, but there are even some small "mom and pop" cloud providers that are cheaper and easier to use than AWS. However, until they stop gobbling up market share there will always be ancillary markets that will work to support those that invest in their services.
  - JoshTriplett 8y ago
    > Is the $500 a month DDOS protected bandwidth with as much redundancy as AWS is providing? Sometimes you don't need that. AWS S3 provides reduced-redundancy options for less cost. It'd be nice if the same applied to other services.

jacques_chester 8y ago

Where I work there's a monthly report sent out that shows what each team is spending. Periodically teams poke around their infrastructure and remove anything they don't need any more. A few times there have been company-wide "spring cleaning" days. That's all it has taken to save a lot of money.

user5994461 8y ago
Same. It's pretty amazing to be able to see where the money is going and adjust anytime.

incomplete 8y ago

i honestly cannot believe that this is still a thing! well, actually, i can. :( i run a large AWS consolidated billing family for a sizeable computer science graduate research lab and tracking our AWS spends are an absolute nightmare. if you think about it for a minute, why would it be in the best interests of amazon/AWS to allow us granular, easy to parse and (most importantly) easy to customize and use billing reports? they're in the business of making money, and providing mechanisms for folks to limit their spends will limit profits. another thing to consider is that the billing subsystems for all of these cloud providers are literally one of the first things to be engineered, and after release, one of the last to ever be updated. for instance, it took amazon two years to release OUs (organizational units): one year in closed beta, one in open. while these are great for organizing accounts, you still can't see how much an OU spent w/o a bunch of python/boto gymnastics! i was on a call w/some of the leads of the aws billing subsystem about a year back, and asked about what the roadmap for billing features and the response was "2020 at the earliest".

user5994461 8y ago
What are you having troubles with? I find the billing page to be pretty good, once the tags are configured for billing. Far better than any internal tool I had the chance to work with.
- incomplete 8y ago
  we have a lot of projects, and i use OUs to track the N-ary tree of lab->project->people and their spends. since the aws billing subsystem can't map OU->$$$, i wrote a thing: https://github.com/ucbrise/aws-audit https://github.com/ucbrise/aws-audit it'd be also super awesome to have OU-based budgets. we use teevity ice when we really need to dig deep (https://github.com/Teevity/ice https://github.com/Teevity/ice) but it's pretty much abandonware at this point. i've also explored a couple of commercial offerings (ronin & teevity) but they don't work well in our AWS research credit-based model.
  - user5994461 8y ago

deleted 8y ago

[deleted]

dba7dba 8y ago

AWS is really expensive. Even for running a small personal websites. I thought I could gain AWS experience AND run a few personal websites on them. What a mistake. I've moved off all (well like 2) my personal sites to other VPS provider, that starts at $5/month for 512MB VPS. The worst annoyance is 'Reserved instances'. You are basically signing up for a long term contract that you can't get out of easily.

gshakir 8y ago
S3 website hosting for static pages will be cheaper than VPS unless you are running an app.
phonon 8y ago
Lightsail is even cheaper, AND AWS https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/

tyingq 8y ago

Part of this is that AWS is often more expensive than on-prem. The other part, though, is that dev teams in large companies often aren't held accountable for on-prem costs. The on-prem costs are often in a big baseline number with little visibility into how much of that big number belongs to each team.

waterishail 8y ago

I'm reminded of this article - http://highscalability.com/blog/2017/12/4/the-eternal-cost-savings-of-netflixs-internal-spot-market.html http://highscalability.com/blog/2017/12/4/the-eternal-cost-s... Netflix run a spot market for instances to drive down costs for exactly the reasons mentioned here

AcerbicZero 8y ago

This is one of those predictable things, that is conveniently ignored by certain layers of a business. The more entertaining situation (from my position anyway) is the drama that the rapid Azure adoption by otherwise very non-cloud companies will cause in a year or two, when the free credits Microsoft stuffed into their ELA's go away. I love running stuff in the cloud, but without a conscious decision in an organization to prepare for, and avoid, the various pitfalls, its just a giant cost sink.

pnathan 8y ago

One of the basic tenants of engineering is "why is this feature appropriate"; much of AWS can be sliced away with this razor, much of the time. Appropriate tool for the appropriate time. A lot of the AWS systems simply are special purpose and not good for general application. I'm working on a Kubernetes configuration at home that provides the common & popular features AWS in a consistent & portable manner: blob store, database, etc.

kevinAtStorj 8y ago

I find it interesting that while the price of hard drives has decreased by about 50 percent over the past five years, the price of cloud storage has remained roughly flat. As margin continues to expand, the need for alternative models and competition in 'Cloud' is becoming increasingly apparent. https://storj.io/blog/2018/11/the-high-price-of-traditional-cloud-storage/ https://storj.io/blog/2018/11/the-high-price-of-traditional-...

riskneutral 8y ago
I think the end game is probably something resembling electrical grid markets. Electrical power is bought and sold in real-time by traders, as the supply and load fluctuate. There are many players in a competitive market, like nuclear plants, gas plants, solar, wind, etc. There are even complex derivatives (like “swing options”) traded in the electrical markets. Someday computing resources will be bought and sold in this way too, with a large number of players in a competitive market, which will drive down the currently inflated cloud computing costs. We can already see an embryonic version of such a market in the form of AWS “spot instances” and the AWS reserved instance secondary market. AWS is fighting tooth and nail to avoid or delay such a commoditization of computing utilities, which is why they announce 50 new vendor lock-in software products every week instead of focusing only on the hardware resource.

kemitchell 8y ago

Missed opportunity: As AWS Use Soars, Companies Surprised by Soaring Bills

nunez 8y ago

companies that think that they can treat aws like their own colo and dont have any semblance of budgeting will always be surprised by their bills. the cloud can be very expensive...it can also be significantly cheaper (think: a retailer with mostly static websites with some light javascript here and there that peaks on black friday). doing an analysis and being responsible can help here.

deepsun 8y ago

Sometimes I wonder whether AWS makes more money than GCP because average GCP users develop better optimized systems.