5 ms·

How many would say this normally? I.e., what's the baseline? Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analyt

How many would say this normally? I.e., what's the baseline? Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. At one point we literally pulled all the billing data into a SQLite database. (And that was a great idea, as it made queries against it much faster.) Further, most higher ups seem to have a sense of "it won't be invented here", and so a lot of stuff ends up in vendor land; need more people then to pull those separate bills into one spot, and start attributing costs against the revenue they might help generate. I've also worked a fair number of places where people were cagey about telling eng how much they're spending where. If you want your IT dept. to cut costs, they have to know where the money is going. And then there's the yak shaving of it all: we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Cool storage should be (price * amount stored) and the "how much are we storing" graph is not proportional to the cost graph. The only gotcha is an early delete penalty, and we are also pretty confident we're not hitting that. (We know what's being deleted when, and none of it should be subject to it.) But how does it take a over week to answer the question?

jamesfinlayson 4y ago
> Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? I don't get time for that - I've got access to Cloudability but no one has ever showed me how to use it.
noduerme 4y ago
I manage one midsized company's set of cloud resources that I'm pretty sure could be reduced by 40% in cost or more if they were willing to take a 3 year advanced reserve. The problem is I can't get a handle on how much their infrastructure will or won't need to scale in the next three years, since they themselves don't know. There's always some new deal around the corner that would require scaling up. On top of that, I'm not sure what we have now will even be sufficient for the existing size of the business a year from now, as much depends on new features in the pipeline and how they end up being used. The result is they end up overspending by probably $3k a year, but the upshot is the flexibility. As far as getting completely off the cloud, I migrated them away from dedicated boxes years ago and I don't miss dealing with maintaining those. The extra time for that alone was worth at least $3k a year. Even with a smaller dedicated data center, there are still issues with getting tickets responded to promptly. And there's way less in the way of failover unless you build it yourself.
- abofh 4y ago
  So there are three choices, essentially to argue over: - run on demand instances (maybe mix in spot) - run fargate instances (maybe mix in spot) - reserve them. Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
  - noduerme 4y ago
    Really the issue is swapability. The way the system is constructed, there's basically one monolith for customer-facing services and another for internal ops, and some smaller shared databases, services and a bunch of backup. This hews to the company's security profile, and the lack of scattered microservices (which I now consider fortunate) is also a result of that years-ago migration from dedicated servers where these two interconnected systems evolved along separate trajectories [edit: to the point where the customer facing system still only writes to its own mark/sweep DB that's picked up by the internal system, or in rare cases accesses internal APIs by cUrl ... so either could still be anywhere]. Going up a tier on one or both of those monoliths would be more difficult and costly if they're reserved. So in a sense, a long reservation would just be good insurance against the business not growing. But no one wants to pull the trigger on that.
  - eb0la 4y ago
    IMHO Reservations in AWS are good if you have a US Bank Account. Why? Because there is a secondary market (run by AWS) where you can buy and sell reservations. If you have a 1-year project that will have a 1-year extension, then get a 3-year reservation (with some amount upfront) that you can sell. Your second best option is a Savings Plan.
    - ethbr0 4y ago
      Neat! Is this via third party services or Amazon itself? Curious about details on the retrade market.
Aeolun 4y ago
> Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Surprisingly enough we have 3 people working on it right now. Sadly they’ve found that it was extremely easy to save several thousands of dollars a day, and has been so for the past two years.
- jamesfinlayson 4y ago
  Not sure how you did it but I've worked for a company that enabled you to tag resources with operating hours and stuff got shut down outside of the those operating hours - pretty handy way to cut down on usage because most non-live environments don't need everything running 24x7.
  - Aeolun 4y ago
    This was mostly properly scaling down things that didn’t need to be up all the time (as far as I understand it, I wasn’t part of the effort). We do already have a system that shuts down services when they’re not needed (based on tags, so presumably sort of the same thing).
  - gadflyinyoureye 4y ago
    My client does this. Really nice if you’re on the East coast. Ireland might have a few morning hours where nothing works in test. India has large chunks of time with the same issue. Surprisingly this hasn’t lead them to mock service locally. They are just less productive.
    - Aeolun 4y ago
      Why not expand the range? The running of the service can’t be more expensive than the devs right?
      - jamesfinlayson 4y ago
        I agree - you need a dev environment to do your job. That said, I worked at a place where the AWS dev accounts cost hundreds of thousands of dollars a month - I don't know how many employees were outsourced but no one wanted to pay for the extra uptime of the environments for them.
hotpotamus 4y ago
> Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.
- gw99 4y ago
  It's terrible for you every time. we leak $100k a month into Bezos' pockets on a regular basis. Sleep well.
bradleyjg 4y ago
But how does it take a over week to answer the question? In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.
- gw99 4y ago
  If it's any consolation it's not much better on the AWS side. We are currently waiting for them to fix something which they recently broke but some dude is on holiday. I suspect he's the only dude who knows how it works. Even our account managers can't seem to brow beat anyone into fixing it. Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
  - pugz 4y ago
    Out of interest, what support tier are you on? Is it the Enterprise ($15k/month minimum) tier? I've found that AWS Support will resolve any non-critical issue within at most a few hours on that tier. (Our cloud spend was $1M/month)
    - gw99 4y ago
      We have the $15k enterprise contract.
  - bradleyjg 4y ago
    I believe that’s been your experience, but it hasn’t been mine. I’ve had mostly positive support experiences with AWS.
    - gw99 4y ago
      Core services are fine. All the problems are on things they built on their core infrastructure proposition.
- nigma1337 4y ago
mkhpalm 4y ago
> we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.
pid-1 4y ago
> where the money is going AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need. - Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯ - Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket. - CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out. - Many costs are tied with stuff that's really hard to control e.g. egress traffic. - Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... - Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services. The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.
- Hikikomori 4y ago
  >- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... Not if you use shared VPCs.
- darkwater 4y ago
  > - Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
  - raffraffraff 4y ago
    Is this the same for child accounts created via AWS Orgs?
    - darkwater 4y ago
bayraktar 4y ago
But how does it take a over week to answer the question? Sounds like the more basic problem is that we keep ending up working in companies where the "higher ups" are constantly shoving decisions down our throats. Which, ultimately, is probably the root cause of most of the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.
- ethbr0 4y ago
  The silly support tickets that stay open forever and ever and ever is just a company being cheap. If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload. Oh, and for god's sake have a decent "to engineering" escalation path for the support folks. Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.
victorNicollet 4y ago
Hey, we've had our blob storage costs spike up +70% for two days (Oct 4th and Oct 5th), across almost all of our storage accounts, regions and hot/cold/archive tiers. The spike is a consistent ~70% for each of the affected accounts, and this includes accounts where there have been no writes for months (so why is there an increase), as well as accounts where we never delete (so while an increase would be surprising, the decrease on Oct 6th is impossible). Is this similar to what you're seeing ? I was considering writing to Azure support about that... My last billing issue with them took ~120 days to resolve, so I don't have much hope.
- deathanatos 4y ago
  Incredible. You've nailed both the amount (the +70%)¹ and the days (yep, Oct 4th & 5th). You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion. > Is this similar to what you're seeing ? You've nailed it precisely; it sounds exactly like what we're seeing. (What a small world HN is sometimes!) ¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
  - itsdrewmiller 4y ago
    This comment inspired me to check our storage spend and we also have a very unusual spike in storage cost on Oct 4th and 5th. More like 40% increase than 70% though, hot tier.
vidarh 4y ago
As an anecdote on the knowing where money is going front: One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report. My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation. Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes. Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs. And so you get organisations which often have no idea which decisions contribute what to those costs.
marcosdumay 4y ago
> I.e., what's the baseline? I don't believe a baseline exists. The cloud is a new thing, and the fashion on how to use it is still changing constantly. You also can not use this number as a proxy for any other thing. It's just what it is, you can't look for deeper connections.

jamesfinlayson 4y ago

> Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? I don't get time for that - I've got access to Cloudability but no one has ever showed me how to use it.

noduerme 4y ago

I manage one midsized company's set of cloud resources that I'm pretty sure could be reduced by 40% in cost or more if they were willing to take a 3 year advanced reserve. The problem is I can't get a handle on how much their infrastructure will or won't need to scale in the next three years, since they themselves don't know. There's always some new deal around the corner that would require scaling up. On top of that, I'm not sure what we have now will even be sufficient for the existing size of the business a year from now, as much depends on new features in the pipeline and how they end up being used. The result is they end up overspending by probably $3k a year, but the upshot is the flexibility. As far as getting completely off the cloud, I migrated them away from dedicated boxes years ago and I don't miss dealing with maintaining those. The extra time for that alone was worth at least $3k a year. Even with a smaller dedicated data center, there are still issues with getting tickets responded to promptly. And there's way less in the way of failover unless you build it yourself.

abofh 4y ago
So there are three choices, essentially to argue over: - run on demand instances (maybe mix in spot) - run fargate instances (maybe mix in spot) - reserve them. Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
- noduerme 4y ago
  Really the issue is swapability. The way the system is constructed, there's basically one monolith for customer-facing services and another for internal ops, and some smaller shared databases, services and a bunch of backup. This hews to the company's security profile, and the lack of scattered microservices (which I now consider fortunate) is also a result of that years-ago migration from dedicated servers where these two interconnected systems evolved along separate trajectories [edit: to the point where the customer facing system still only writes to its own mark/sweep DB that's picked up by the internal system, or in rare cases accesses internal APIs by cUrl ... so either could still be anywhere]. Going up a tier on one or both of those monoliths would be more difficult and costly if they're reserved. So in a sense, a long reservation would just be good insurance against the business not growing. But no one wants to pull the trigger on that.
- eb0la 4y ago
  IMHO Reservations in AWS are good if you have a US Bank Account. Why? Because there is a secondary market (run by AWS) where you can buy and sell reservations. If you have a 1-year project that will have a 1-year extension, then get a 3-year reservation (with some amount upfront) that you can sell. Your second best option is a Savings Plan.
  - ethbr0 4y ago
    Neat! Is this via third party services or Amazon itself? Curious about details on the retrade market.

Aeolun 4y ago

> Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Surprisingly enough we have 3 people working on it right now. Sadly they’ve found that it was extremely easy to save several thousands of dollars a day, and has been so for the past two years.

jamesfinlayson 4y ago
Not sure how you did it but I've worked for a company that enabled you to tag resources with operating hours and stuff got shut down outside of the those operating hours - pretty handy way to cut down on usage because most non-live environments don't need everything running 24x7.
- Aeolun 4y ago
  This was mostly properly scaling down things that didn’t need to be up all the time (as far as I understand it, I wasn’t part of the effort). We do already have a system that shuts down services when they’re not needed (based on tags, so presumably sort of the same thing).
- gadflyinyoureye 4y ago
  My client does this. Really nice if you’re on the East coast. Ireland might have a few morning hours where nothing works in test. India has large chunks of time with the same issue. Surprisingly this hasn’t lead them to mock service locally. They are just less productive.
  - Aeolun 4y ago
    Why not expand the range? The running of the service can’t be more expensive than the devs right?
    - jamesfinlayson 4y ago
      I agree - you need a dev environment to do your job. That said, I worked at a place where the AWS dev accounts cost hundreds of thousands of dollars a month - I don't know how many employees were outsourced but no one wanted to pay for the extra uptime of the environments for them.

hotpotamus 4y ago

> Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.

gw99 4y ago
It's terrible for you every time. we leak $100k a month into Bezos' pockets on a regular basis. Sleep well.

bradleyjg 4y ago

But how does it take a over week to answer the question? In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.

gw99 4y ago
If it's any consolation it's not much better on the AWS side. We are currently waiting for them to fix something which they recently broke but some dude is on holiday. I suspect he's the only dude who knows how it works. Even our account managers can't seem to brow beat anyone into fixing it. Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
- pugz 4y ago
  Out of interest, what support tier are you on? Is it the Enterprise ($15k/month minimum) tier? I've found that AWS Support will resolve any non-critical issue within at most a few hours on that tier. (Our cloud spend was $1M/month)
  - gw99 4y ago
    We have the $15k enterprise contract.
- bradleyjg 4y ago
  I believe that’s been your experience, but it hasn’t been mine. I’ve had mostly positive support experiences with AWS.
  - gw99 4y ago
    Core services are fine. All the problems are on things they built on their core infrastructure proposition.
nigma1337 4y ago

mkhpalm 4y ago

> we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.

pid-1 4y ago

> where the money is going AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need. - Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯ - Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket. - CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out. - Many costs are tied with stuff that's really hard to control e.g. egress traffic. - Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... - Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services. The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.

Hikikomori 4y ago
>- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... Not if you use shared VPCs.
darkwater 4y ago
> - Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc... And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
- raffraffraff 4y ago
  Is this the same for child accounts created via AWS Orgs?
  - darkwater 4y ago

bayraktar 4y ago

But how does it take a over week to answer the question? Sounds like the more basic problem is that we keep ending up working in companies where the "higher ups" are constantly shoving decisions down our throats. Which, ultimately, is probably the root cause of most of the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.

ethbr0 4y ago
The silly support tickets that stay open forever and ever and ever is just a company being cheap. If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload. Oh, and for god's sake have a decent "to engineering" escalation path for the support folks. Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.

victorNicollet 4y ago

Hey, we've had our blob storage costs spike up +70% for two days (Oct 4th and Oct 5th), across almost all of our storage accounts, regions and hot/cold/archive tiers. The spike is a consistent ~70% for each of the affected accounts, and this includes accounts where there have been no writes for months (so why is there an increase), as well as accounts where we never delete (so while an increase would be surprising, the decrease on Oct 6th is impossible). Is this similar to what you're seeing ? I was considering writing to Azure support about that... My last billing issue with them took ~120 days to resolve, so I don't have much hope.

deathanatos 4y ago
Incredible. You've nailed both the amount (the +70%)¹ and the days (yep, Oct 4th & 5th). You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion. > Is this similar to what you're seeing ? You've nailed it precisely; it sounds exactly like what we're seeing. (What a small world HN is sometimes!) ¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
- itsdrewmiller 4y ago
  This comment inspired me to check our storage spend and we also have a very unusual spike in storage cost on Oct 4th and 5th. More like 40% increase than 70% though, hot tier.

vidarh 4y ago

As an anecdote on the knowing where money is going front: One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report. My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation. Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes. Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs. And so you get organisations which often have no idea which decisions contribute what to those costs.

marcosdumay 4y ago

> I.e., what's the baseline? I don't believe a baseline exists. The cloud is a new thing, and the fashion on how to use it is still changing constantly. You also can not use this number as a proxy for any other thing. It's just what it is, you can't look for deeper connections.