5 ms·

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't going to address it constructively.

qeternity 10y ago
I am curious your thoughts on the following. It seems like there is a trough of benefit to AWS. In the mid-size world, especially where computing needs are relatively stable/predictable, you are much better leasing dedicated servers from the usual suspects. Below that size, the linear scaling allows small scrappy startups to really scale/burst with their needs. And at large scale, but below Google/Facebook et al scale, it's cheaper than building your own datacenter yourself. At least this has been my casual observations. And fwiw, the startup I have just launched fits into this trough perfectly. So hopefully I am not just biased. I love the cloud for the spot market. But not for our base load. Spot is the nat gas of the server world, and OVH is the coal.
- QuinnyPig 10y ago
  I agree with a lot of what you're saying-- but that trough is a lot wider than most people think. At the single instance very small scale, it simply doesn't matter all that much. "Our business failed because our infrastructure bill was $400 instead of $150 every month" isn't that common of a story. Base load can be effectively migrated to AWS, depending upon how it's built / structured. It's hard to lose money on infrastructure by converting worker servers to spot instances or lambda functions, as a for-instance-- but it's VERY easy to lose months of engineering effort in doing so. Remember as well: done right you're not just buying services from the cloud, but flexibility as well. Resizing on the fly, bursting, duplicating your infrastructure for a test-- these things add value.
  - rhizome 10y ago
    Are bursting and resizing an issue in the trough, though? GP's premise is that these companies have platform and load stability. There can be marginal gains in having a dynamic architecture, sure, but when it also costs "months of engineering," I don't see where the savings would be compelling, or even guaranteed.
    - QuinnyPig 10y ago
      There are companies "in the trough" who are contently spending tens of millions a year on AWS. It's hard to give generic advice for all situations-- as with so much else, context matters heavily. Those marginal gains could potentially pay for a dozen engineers...
      - rhizome 10y ago
        My point was based on how dynamic the architectures are (or aren't) of those in the trough, not their level of spending.
        QuinnyPig 10y ago
        Ah, my apologies! It depends. I have a toy service that exists entirely as an autoscaling group. That's pretty dynamic! Counterpoint: Mainframes.
        rhizome 10y ago
        A toy service is not what I'm thinking of as being in "the trough." The trough is where scaling and bursting aren't generally necessary.
  - metafunctor 10y ago
    Resizing, bursting, and duplicating on the fly aren't exclusive to AWS, though. A lot of teams rent space from places like OVH or Hetzner because the raw power is so much cheaper, and they already have the talent and tools to scale and manage a computing fleet (people like to think it's harder than it actually is).
alexkon 10y ago
How do you approach a horrifying AWS bill? Can you recommend any reading? I’m going to need to optimize a tiny fleet on GCE pretty soon.
- QuinnyPig 10y ago
  It gets into the specifics of "what makes it horrifying" pretty quickly. It usually boils down to "nobody's responsible for cleaning things up," a misunderstanding of the billing model, and by the time anyone cares to fix it the billing report is multiple gigabytes. Step 1 is always "figure out what you've got, and why it is this way."
  - brazzledazzle 10y ago
    So if the best way to avoid the mess is accountability for keeping things clean, how do you do that from the start? Would something as simple as tagging resources to specific teams/departments be a good place to start? I'm helping with setting up a completely new account/environment and I know once the buzzword fever passes and the annual costs are reviewed there will be frantic requests to cut down the bill. Getting ahead of that inevitability with a plan and some preparation will save us a headache.
    - QuinnyPig 10y ago
      Propagate tags to secondary resources, like EBS volumes. Separate out production from your dev environment-- preferably with a second AWS account with consolidated billing. Buy in to something like Terraform or CloudFormation for building the non-instance parts of your stack. As you grow, start adding project tags to things. Automatically tag resources with the IAM user who creates them. Have an "untagged resources report" that goes out on a schedule. Build with an eye towards "when someone freaks out in a year about the bill, what questions will they have, and how can I best position Future Me to answer them?"
kafkaesq 10y ago
A blanket "never use AWS" isn't going to address it constructively. But that's not what the commenter was saying, of course. More like (paraphrasing) "before you go full-hog on AWS, do some simple math first." Which an amazingly high percentage of people neglect to do, these days.
- EnFinlay 10y ago
  > At this point simply not using AWS is a competitive advantage.
  - logfromblammo 10y ago
    I read this as suggesting that a simple arbitrary rule to not use AWS is likely to save both money and also the time that would otherwise be spent tuning your use of AWS to cost less money. Your time is valuable too. It could be spent developing something novel instead of messing around with server configuration.
- hinkley 10y ago
  Weeks of work can save hours of meetings.
  - thecrazyone 10y ago
    interesting thing to say, but did you plan for it to mean something in this context? :)
    - kafkaesq 10y ago
      Nothing special - just the whole "Yeah, saw your email† ... look, I'm so busy, did you get the site up and running again? I'll get back to you on this after my vacation" management style which seems to drive a great many "decisions" that are made in this realm. And on which AWS, implicitly, banks a great deal of its business model on. † About the monthly you-know-what bill now well into 5 figures, and climbing.
- 10y ago
tyingq 10y ago
I would guess the network egress costs, specifically, might justify a blanket statement like... "If you have content where you can't put a cache in front of it, and it drives significant bandwidth use, then AWS is not a great option"
- stemuk 10y ago
  This. I really feel like egress traffic costs is the most overlooked issue with popular cloud services like AWS or Google Cloud. Charging 9 or 12 cents for a mere GB of regular traffic is just insane, keeping in mind that even most CDNs are cheaper. Data driven applications on AWS are already expensive because of the high traffic costs, but storing/serving media on S3 single handedly lets your bill explode. For everyone who feels like the statements above are exaggerations, just check out their actual prices (https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/) in comparison with a regular CDN (https://www.stackpath.com/pricing/ https://www.stackpath.com/pricing/).
  - tyingq 10y ago
    A quick and dirty comparison would be 3TB out in a month. That's what's typically included at popular VPS services for a decent sized VPS that's $20/month. On an EC2 instance, just the bandwidth for that month would be $269.91.
rdtsc 10y ago
> I consult full time on AWS cost optimization / reduction / understanding. Curious, have you ever consulted a client not to use AWS? As in "your cost optimization is going to be not using AWS and using $x instead"?
- mrlatinos 10y ago
  Curious about this as well. Otherwise, it's like a car salesman saying you shouldn't buy a bike, rather, understand mpg before buying a car.
  - QuinnyPig 10y ago
    Sure-- but I'm usually not retained until the environment is built out. The client's starting point is relevant to what comes next.
- QuinnyPig 10y ago
  Absolutely. Great example: container workloads without high data volumes. Slap those into GCP if you can-- or arbitrage between different cloud providers / regions / on-prem if you have them. That said, clients who come to me already have a significant investment with AWS. Getting rid of that is a high bar in terms of engineering effort. That said, I've no financial incentive to recommend one provider over another. My clients pay me, the vendors do not.
  - rdtsc 10y ago
    Makes total sense. Thank you for responding! We've pushed clients to competitors' products a few times, I was young and thought that was silly. However those clients spread the word about us and we didn't even have much of a marketing department, it was all word of mouth. Gaining their trust was gold and a really good long term investment.
fpp 10y ago
Running / monthly cloud cost and comparison to other options (e.g. run on a rented dedicated machine / co-location) should already been looked at when you do even the most basic business plan. Generally most of the time the best approach is to create solutions that can scale out independent from the HW / infrastructure they run on. With frameworks like K8s / Docker / e.a. this is now relatively easy. With that for example you can start of with a few small cloud machines / docker containers. When you get continuous load - migrate some of them to 1-2 dedicated machines. Same with peak load - just spin up more images. All that of course depends on what kind of application you are providing. If you are running a data intensive apps you might start directly with a dedicated machine. Easy in, expensive out. One cost with cloud services many forget is the cost to get out again. Have a look at the AWS / Azure price-lists.

qeternity 10y ago

I am curious your thoughts on the following. It seems like there is a trough of benefit to AWS. In the mid-size world, especially where computing needs are relatively stable/predictable, you are much better leasing dedicated servers from the usual suspects. Below that size, the linear scaling allows small scrappy startups to really scale/burst with their needs. And at large scale, but below Google/Facebook et al scale, it's cheaper than building your own datacenter yourself. At least this has been my casual observations. And fwiw, the startup I have just launched fits into this trough perfectly. So hopefully I am not just biased. I love the cloud for the spot market. But not for our base load. Spot is the nat gas of the server world, and OVH is the coal.

QuinnyPig 10y ago
I agree with a lot of what you're saying-- but that trough is a lot wider than most people think. At the single instance very small scale, it simply doesn't matter all that much. "Our business failed because our infrastructure bill was $400 instead of $150 every month" isn't that common of a story. Base load can be effectively migrated to AWS, depending upon how it's built / structured. It's hard to lose money on infrastructure by converting worker servers to spot instances or lambda functions, as a for-instance-- but it's VERY easy to lose months of engineering effort in doing so. Remember as well: done right you're not just buying services from the cloud, but flexibility as well. Resizing on the fly, bursting, duplicating your infrastructure for a test-- these things add value.
- rhizome 10y ago
  Are bursting and resizing an issue in the trough, though? GP's premise is that these companies have platform and load stability. There can be marginal gains in having a dynamic architecture, sure, but when it also costs "months of engineering," I don't see where the savings would be compelling, or even guaranteed.
  - QuinnyPig 10y ago
    There are companies "in the trough" who are contently spending tens of millions a year on AWS. It's hard to give generic advice for all situations-- as with so much else, context matters heavily. Those marginal gains could potentially pay for a dozen engineers...
    - rhizome 10y ago
      My point was based on how dynamic the architectures are (or aren't) of those in the trough, not their level of spending.
      - QuinnyPig 10y ago
        Ah, my apologies! It depends. I have a toy service that exists entirely as an autoscaling group. That's pretty dynamic! Counterpoint: Mainframes.
        rhizome 10y ago
        A toy service is not what I'm thinking of as being in "the trough." The trough is where scaling and bursting aren't generally necessary.
- metafunctor 10y ago
  Resizing, bursting, and duplicating on the fly aren't exclusive to AWS, though. A lot of teams rent space from places like OVH or Hetzner because the raw power is so much cheaper, and they already have the talent and tools to scale and manage a computing fleet (people like to think it's harder than it actually is).

alexkon 10y ago

How do you approach a horrifying AWS bill? Can you recommend any reading? I’m going to need to optimize a tiny fleet on GCE pretty soon.

QuinnyPig 10y ago
It gets into the specifics of "what makes it horrifying" pretty quickly. It usually boils down to "nobody's responsible for cleaning things up," a misunderstanding of the billing model, and by the time anyone cares to fix it the billing report is multiple gigabytes. Step 1 is always "figure out what you've got, and why it is this way."
- brazzledazzle 10y ago
  So if the best way to avoid the mess is accountability for keeping things clean, how do you do that from the start? Would something as simple as tagging resources to specific teams/departments be a good place to start? I'm helping with setting up a completely new account/environment and I know once the buzzword fever passes and the annual costs are reviewed there will be frantic requests to cut down the bill. Getting ahead of that inevitability with a plan and some preparation will save us a headache.
  - QuinnyPig 10y ago
    Propagate tags to secondary resources, like EBS volumes. Separate out production from your dev environment-- preferably with a second AWS account with consolidated billing. Buy in to something like Terraform or CloudFormation for building the non-instance parts of your stack. As you grow, start adding project tags to things. Automatically tag resources with the IAM user who creates them. Have an "untagged resources report" that goes out on a schedule. Build with an eye towards "when someone freaks out in a year about the bill, what questions will they have, and how can I best position Future Me to answer them?"

kafkaesq 10y ago

A blanket "never use AWS" isn't going to address it constructively. But that's not what the commenter was saying, of course. More like (paraphrasing) "before you go full-hog on AWS, do some simple math first." Which an amazingly high percentage of people neglect to do, these days.

EnFinlay 10y ago
> At this point simply not using AWS is a competitive advantage.
- logfromblammo 10y ago
  I read this as suggesting that a simple arbitrary rule to not use AWS is likely to save both money and also the time that would otherwise be spent tuning your use of AWS to cost less money. Your time is valuable too. It could be spent developing something novel instead of messing around with server configuration.
hinkley 10y ago
Weeks of work can save hours of meetings.
- thecrazyone 10y ago
  interesting thing to say, but did you plan for it to mean something in this context? :)
  - kafkaesq 10y ago
    Nothing special - just the whole "Yeah, saw your email† ... look, I'm so busy, did you get the site up and running again? I'll get back to you on this after my vacation" management style which seems to drive a great many "decisions" that are made in this realm. And on which AWS, implicitly, banks a great deal of its business model on. † About the monthly you-know-what bill now well into 5 figures, and climbing.
10y ago

tyingq 10y ago

I would guess the network egress costs, specifically, might justify a blanket statement like... "If you have content where you can't put a cache in front of it, and it drives significant bandwidth use, then AWS is not a great option"

stemuk 10y ago
This. I really feel like egress traffic costs is the most overlooked issue with popular cloud services like AWS or Google Cloud. Charging 9 or 12 cents for a mere GB of regular traffic is just insane, keeping in mind that even most CDNs are cheaper. Data driven applications on AWS are already expensive because of the high traffic costs, but storing/serving media on S3 single handedly lets your bill explode. For everyone who feels like the statements above are exaggerations, just check out their actual prices (https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/) in comparison with a regular CDN (https://www.stackpath.com/pricing/ https://www.stackpath.com/pricing/).
- tyingq 10y ago
  A quick and dirty comparison would be 3TB out in a month. That's what's typically included at popular VPS services for a decent sized VPS that's $20/month. On an EC2 instance, just the bandwidth for that month would be $269.91.

rdtsc 10y ago

> I consult full time on AWS cost optimization / reduction / understanding. Curious, have you ever consulted a client not to use AWS? As in "your cost optimization is going to be not using AWS and using $x instead"?

mrlatinos 10y ago
Curious about this as well. Otherwise, it's like a car salesman saying you shouldn't buy a bike, rather, understand mpg before buying a car.
- QuinnyPig 10y ago
  Sure-- but I'm usually not retained until the environment is built out. The client's starting point is relevant to what comes next.
QuinnyPig 10y ago
Absolutely. Great example: container workloads without high data volumes. Slap those into GCP if you can-- or arbitrage between different cloud providers / regions / on-prem if you have them. That said, clients who come to me already have a significant investment with AWS. Getting rid of that is a high bar in terms of engineering effort. That said, I've no financial incentive to recommend one provider over another. My clients pay me, the vendors do not.
- rdtsc 10y ago
  Makes total sense. Thank you for responding! We've pushed clients to competitors' products a few times, I was young and thought that was silly. However those clients spread the word about us and we didn't even have much of a marketing department, it was all word of mouth. Gaining their trust was gold and a really good long term investment.

fpp 10y ago

Running / monthly cloud cost and comparison to other options (e.g. run on a rented dedicated machine / co-location) should already been looked at when you do even the most basic business plan. Generally most of the time the best approach is to create solutions that can scale out independent from the HW / infrastructure they run on. With frameworks like K8s / Docker / e.a. this is now relatively easy. With that for example you can start of with a few small cloud machines / docker containers. When you get continuous load - migrate some of them to 1-2 dedicated machines. Same with peak load - just spin up more images. All that of course depends on what kind of application you are providing. If you are running a data intensive apps you might start directly with a dedicated machine. Easy in, expensive out. One cost with cloud services many forget is the cost to get out again. Have a look at the AWS / Azure price-lists.