5 ms·

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up

• tuananh 3y ago
  that's one of the main reason we made this feature "opt-out". if you want to keep it up, you have to tag it. once you tag it, it can opt-out 7 days. then you have to extend it (simply chat with our bot)
  • zikduruqe 3y ago
    https://cloudcustodian.io https://cloudcustodian.io Create a rule and shame them on Slack.
• dangus 3y ago
  Well, it’s definitely fathomable. Does my employer have cost control baked into their proceses, tooling, and culture or are they rushing me to get projects out the door leaving barely enough time to make sure they’re production-ready? Most places I’ve worked had no formal production readiness review before launching infrastructure.
• lnxg33k1 3y ago
  I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management
  • pixl97 3y ago
    Which is why your slice of the organization gets a cloud budget. Don't keep your budget under control, well no bonuses for the employees. >Show me the incentive and I will show you the outcome
    • lnxg33k1 3y ago
      >Show me the incentive and I will show you the outcome Would print this phrase on every angle of the offices
    • wizerdrobe 3y ago
      Jokes on corporate, I don’t have ISO, RSU, or a bonus and my raises are always below inflation.
      • lnxg33k1 3y ago
        Oh for the inflation thats awesome, you know how many interviews with engineers Ive had saying they were at a place for 10 years, but the company had a raise cap of 2%, then saying that they wouldn’t hire me because I couldn’t give assurances to be a likeminded clown making corporate rich
    • ponector 3y ago
      But bonuses are not related to the cloud costs. Nobody in engineering cares about spending because there is no benefits on doing so. Even more: most people are on fixed salary and will get paycheck anyway no matter how low their effort is.
      • pixl97 3y ago
        Hence >>Show me the incentive and I will show you the outcome
    • freedom-fries 3y ago
      Look at Mr bonuse-pants here. Y'all get bonus for writing software?
• pjc50 3y ago
  The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is a huge net win! After all, there's another way to waste company money invisibly: design a process which requires meetings and waiting while work is held up.
  • throwup238 3y ago
    We should be getting kickbacks from Amazon for all the work we’ve done for their bottom line.
    • marvin 3y ago
      We kinda do; Amazon puts upwards pressure on US engineering salaries.
      • LunaSea 3y ago
        What? No? They are behind all major tech companies and notoriously one of the worst employers too.
    • teaearlgraycold 3y ago
      I assume all of the unused resources end up subsidizing the rest of them.
  • pyeri 3y ago
    That's the whole ploy with Agile, isn't it? In the classical SDLC or Waterfall paradigm, everything was pre-approved and signed off, not just the cost or billing but even the software design itself. Any change in the process and the designers had to raise a change request. Agile changed all of that and now we know how bad things can get with that.
• onyxringer 3y ago
  Don't you know that "Developer's time is expensive"? :D
• cduzz 3y ago
  I think "engineer" isn't really the correct word to use for the artisans who build much of the tooling used by most companies. An engineer either wears a striped hat and drives a train, or, went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people." It is expected that an engineer can predict with reasonable accuracy the expense and timeline of a project, and how to maintain the resulting thing, without resorting to voodoo like "scrum velocity." In large part that's because engineers stick to doing things that are well understood and predictable, and if there's risk they resolve the risk before undertaking the project. (Is there bedrock over here upon which to build a foundation? I don't know; let's find out first!). Sure, there are engineering disasters even today -- buildings that unexpectedly lean over and door/wall things that unexpectedly fly off the side of airplanes, but those are typically organizational / process problems not "engineering doesn't work" problems.
  • reactordev 3y ago
    “engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to erosion. They can not predict that a steel beam rated for 300T of force would break at 180T. They can not predict a rogue developer removing a package from underneath their dependency tree. You can give estimates all you want but you are still guessing. If engineers were as you say they are, we would never have delays, we would never have traffic jams, we would never have crap software, we would never have flight.
    • practicemaths 3y ago
      “Engineering is the art of modelling materials we do not wholly understand, into shapes we cannot precisely analyse so as to withstand forces we cannot properly assess, in such a way that the public has no reason to suspect the extent of our ignorance.” - Dr. AR Dykes
• happymellon 3y ago
  I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse? It's rhetorical really as I know why. Terrible architecture from "enterprise". Stick everything in a single account so it's hard to figure out how much is your spend. All 3000 databases, and make sure your k8s cluster is 5 8XL boxes so no one can scale down excess capacity. Classic I Burn Money consultancy!
  • darth_avocado 3y ago
    Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.
    • happymellon 3y ago
      We had a battle to get AWS console access in the first place, after that I had to deal with: > You need to request access every 60 days. Luckily it's now added to my permanent role, but even then no billing access? FFS.
  • tuananh 3y ago
    > 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse? This is so true. billing transparency is very important. in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill
  • malfist 3y ago
    Lots of that is just bad design by going the easiest route. It's easy not to grant engineers access to the billing dashboard. It's easy to put everything in the same aws account. Inside Amazon, we're supposed to set up new aws accounts for every service and realm, so we know how much X service's beta environment is costing
• righthand 3y ago
  I had to implement a 2nd deploy for a QA environment, and my first question to the infrastructure team was “won’t this be costly is there a better way to handle this?” They shrugged off the cost and said they would optimize my deploy once I was done with the initial implementation. 6 months later their optimization was to undo all the work not because it wasn’t a good implementation but because it revealed how much non-optimization had went into the QA environment before I even touched it. A lot of cost is probably due to the “we just taped these two things together” strategy for lower environments.
• racl101 3y ago
  I tend to have the opposite problem. I obsesses over the cost of things, and am pretty bashful about bringing it up to my manager, and he's always surprised that scaling some resources doesn't cost more. But I learned the hard way as a contractor about letting these resources run crazy and had to pay out of pocket so I have PTSD about it, which is why I'm vigilant.
• dilyevsky 3y ago
  > It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. That’s because they were explicitly told not to worry about costs for the last 10 years so majority of ICs at this point never had to do it their entire careers
• znpy 3y ago
  This is entirely artificial: I now work at a company where we know very clearly what our infrastructure costs. Yes, we know the exact costs (what was negotiated, not what is on the public pages). And we celebrate costs slashing as much feature delivery and other stuff. But this is entirely a management problem: at my previous job, only one manager (skip-level manager from my point of view) knew what exactly were we paying for infrastructure. That moron wouldn't share that information with us engineers managing infrastructure of course, so there were a lot of infrastructure choices that didn't really made sense according to the public prices but (I guess?) made sense according to a price sheet we didn't know. So we didn't know what we were spending, didn't have the basic data to estimate the price of a new solution or a new service and didn't have the data to determine how much would we be saving by making changes (optimizing stuff etc). I fought that battle for a bit but then i just said "GFYS, i'm not going to have fights with you so that you can save money" and let go. Later i left the company completely. Former colleagues tell me it's even worse now: there are consultants from the cloud provider involved, they know the pricing deals, and whenever the topic comes up the manager shushes the consultant so that the engineers don't hear the prices. tl;dr: it's an entirely artificial problem, and it's most likely a cultural/management problem. edit: and i'm not even talking about incentives, as somebody else has correctly pointed out.
• VoodooJuJu 3y ago
  And that's how insignificant the costs of cloud providers are in the grand scheme of things. It's a lot of money to a bootstrapping startup, but for the vast majority of these cloud providers' customers, it's a rounding error that's easily forgotten for 3 years. And that's precisely why you and your little bootstrapper or indie firm should not be using globocloud: you do not have mountains of cash to piss away. Bare metal is trending again. And in this downturn, it's no wonder why. Smaller companies are getting smarter and more efficient. They've decided to chase money instead of cargo cults. Globocorps burning cash on globocloud is not a signal for small fish to do the same - it's a signal to do the polar opposite. You're not going to become like them by copying what they're doing now. It will not work for you. Globocloud isn't successful because they shovel cash into AWS's shredder, they shovel cash into the shredder because they're successful.
  • hibikir 3y ago
    You'd be surprised. I've seen AWS bills well in the 9 figures. It's just that fixing expensive designs is, in itself, quite expensive, and many of those very large corps have hiring practices that don't allow them to complete for the top of the market. Sometimes there's tens, if not hundreds of millions in savings a year, but corporate sclerosis makes it very difficult for broad cost-saving initiatives to be identified and approved. It's the same issue in any large organization: Large levels of success somewhere allow for large levels of waste somewhere else, but often the waste is not required for the success to exist: The success just makes the organization complacent.
• sotix 3y ago
  I consistently tried to push for cost management at my last job, but the product manager just wanted to push new features he could show off to management above him. We let costs inflate to ridiculous levels despite my constant discussion around the topic. Software engineers ultimately had no say in the matter. I was laid off this week in a mass layoff because the company doesn’t have enough money to pay all of us anymore. It’s disappointing to see, and I wonder how many other teams ignored these optimizations and how much unnecessary total cost it all summed to.
• mr_00ff00 3y ago
  Would be a random coincidence, but if you work at a large bank, I think I may know the team that had that 300k test db lol.
• ExoticPearTree 3y ago
  In the organization that I work in, costs are transparent to everyone involved and most people are aware of the need to keep costs as low as possible. One of the downsides with this approach is that engineers/developers are not very good business people and don't really understand the notion of "the cost of doing business". And from time to time we have issues with "but it costs $70 more per month", and spend $1000 to optimize those $70 :) In the end, even with some of the wrinkles mention above it helps and saves money when costs are transparent and readily available for anyone.
• nlawalker 3y ago
  This topic got a lot of discussion in "I accidentally saved my company half a million dollars": https://news.ycombinator.com/item?id=38069710 https://news.ycombinator.com/item?id=38069710
• myaccountonhn 3y ago
  What I've observed is that people don't really keep track of what they are spending. I like to set up weekly newsletters that show costs and also if there has been a decrease or increase. In bigger corps, you also should have team based tagging of resources so that specific teams get exactly what they are spending. At the very least, managers will look each week and be like "why did costs increase this week? What's going on?" even if the engineers don't care. "What's get measured gets managed" as they say.