7 ms·
The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things t
by fulafel 1y ago
The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments.
("Shall we make the app very resilient to failure? Yes running on multiple regions makes the AWS bill bigger but you'll get much fewer outages, look at all this technobabble that proves it")
And of course AWS lock-in services are priced to look cheaper compared to their overpricing of standard stuff[1] - if you just spend the engineering effort and IaC coding effort to move onto them, this "savings" can be put to more AWS cloud engineering effort which again makes your cloud eng org bigger and more important.
[1] (For example implementing your app off containers to Lambda, or the db off PostgreSQL to DynamoDB etc)
- Hilift 1y ago> The direct cost is the easy part I don't think it is easy. I see most organizations struggle with the fact that everything is throttled in the cloud. CPU, storage, network. Tenants often discover large amounts of activity they were previously unaware of, that contributes to the usage and cost. And there may be individuals or teams creating new usages that are grossly impacting their allocation. Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Then you can start adding in the Cloud value, such as incomprehensible networking diagrams that are probably non-compliant in some way (guess which ones!), and security? What is it?
- m-gasser 1y ago> Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings. Sounds interesting, which setting is that?
- infecto 1y agoWould love to know as well.
- Hilift 1y agoMultiple Active Result Sets (MARS). During large query responses or bulk loads, "full" packets cause an additional packet to be sent over the wire with about five bytes to hold the MARS "wrapper". The net result is one full packet, and one empty packet on the wire, alternating. The performance impact in LAN latency is negligible. However on higher latency between AWS and your premises it has a terrible performance impact. MARS isn't strictly needed for most things. Some features that requires it are ORM (EF) proxies and lazy loading. If you need MARS, there are third party "accelerators" that workaround this madness. "MARS Acceleration significantly improves the performance of connections that use the Multiple Active Result Sets (MARS) connection option." https://documentation.nitrosphere.com/resources/release-notes https://documentation.nitrosphere.com/resources/release-note...
- infecto 1y agoIs that not a client connection flag? MARS does not require a setting change on the server?
- Hilift 1y agoIt is a setting in connection string in the client app.config, "MultipleActiveResultSets=true;".
- Hilift 1y ago"Multiple Active Result Sets=true;"
- anonymars 1y agoI think you may have misinterpreted what he said. I can see why it seems to imply a server setting but that isn't the case > Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings
- _the_inflator 1y agoYes. Cloud sellers new this: Happy path for this flagship project, the shinny new object, and some additional services. After the point of no return what usually happens is, that cloud will be a replica of bare metal development. As an Computer Science dude and former C64/Amiga coder in Senior Management of a large international Bank, I saw first hand, how cost balloon simply due to the fact, that the bank recreates and replicates its bare metal environment in the cloud. So increasing costs while nothing changed. Imagine that: fixed resources, no test environments, because virtualisation was out of the equation in the cloud due to policies and SDLC processes. And it goes on: releases on automation? Nope, request per email and attached scan of a paper document as sign-off. Of course your can buy a Ferrari and use it as a farm tractor. I bet it is possible with a little modification here and there. Another fact is, that lock in plays a huge role. Once you are in it, no matter what you subscribe to, magically everything slows suddenly down, a bit, but since I am a guy who uses a time tracker to test and monitor apps, I could easily draw a line even without utilizing my Math background: enforced throtelling. There is a difference between 100, 300 and 500ms for SaaS websites - people without prior knowledge of peceptual psychology feel it but cannot but their finger in the wound. But since we are in the cloud, suddenly a cloud manager will offer you an speed upgrade - just catered for your needs! Here, have a trial period over 3 month for free and experience the difference for your business! I am a bit of opinionated here and really suppose, that cloud metrics analysed the banks traffic and service usage to willingly slow it down in a way, only professionals could find out. Have you promised to be lightning fast in the first place? No, that's not what the contract says. We fed you with it, but a "normal" speed was agreed upon. It is like getting a Porsche as a rental car for free when you take your VW Beetle to the dealer for a checkup. Hooked, of course. A car is a car after all. How to boil a frog? Slowly. Of course there will be more sales and this is achilles' heel for every business and indifferent customers - easy prey. It is a vicious cycle, almost like taxation. You cannot hide from it, no escape and it is always on the rise.
- franktankbank 1y agoFerrari actually makes tractors.
- deleted 1y ago
- vidarh 1y agoI was about to rage at you over the first sentence, because this is so often how people start trying to argue bare metal setups are expensive. But after reading the rest: 100% this. I see so many people push AWS setups not because it's the best thing - it can be if you're not cost sensitive - but because it is what they know and they push what they know instead of evaluating the actual requirements.
- hibikir 1y agoWell, they aren't wrong about the bare metal either: Every organization ends up tied to their staff, and said staff was hired to work on the stack you are using. People end up in quite the fights because their supposed experts are more fond of uniformity and learning nothing new. Many a company was stuck with a datacenter unit that was unresponsive to the company's needs, and people migrated to AWS to avoid dealing with them. This straight out happened in front of my eyes multiple times. At the same time, you also end up in AWS, or even within AWS, using tools that are extremely expensive, because the cost-benefit analysis for the individuals making the decision, who often don't know very much other than what they use right now, are just wrong for the company. The executive on top is often either not much of a technologist or 20 years out of date, so they have no way to discern the quality of their staff. Technical disagreements? They might only know who they like to hang out with, but that's where it ends. So for path dependent reasons, companies end up making a lot of decisions that in retrospect seem very poor. In startups if often just kills the company. Just don't assume the error is always in one direction.
- baq 1y ago> Many a company was stuck with a datacenter unit that was unresponsive to the company's needs I'd like to +1 here - it's an understated risk if you've got datacenter-scale workloads. But! You can host a lot of compute on a couple racks nowadays, so IMHO it's a problem only if you're too successful and get complacent. In the datacenter, creative destruction is a must and crucially finance must be made to understand this, or they'll give you budget targets which can only mean ossification.
- 1y ago
- anal_reactor 1y agoMy manager wants me to make this silly AWS certification. Let me go on a tangent about trains. In Spain before you board a high-speed train you need to go though full security check, like on an airport. In all other EU countries you just show up and board, but in Spain there's the security check. The problem is that even though the security check is an expensive, inefficient theatre, just in case something does blow up, nobody wants to be the politician that removed the security check. There will be no reward for a politician that makes life marginally easier for lots of people, but there will be severe punishment for a politician that is involved in a potential terrorist attack, even if the chance of that happening is ridiculously small. This is exactly why so many companies love to be balls deep into AWS ecosystem, even if it's expensive.
- mrits 1y agoAWS doesn’t have to be expensive.
- embedding-shape 1y agoSure, but you outgrow the free ("trial") resources in a blink, and then it starts being expensive compared to the alternatives.
- rsav 1y agoNobody gets fired for buying IB^H^H AWS
- kleiba 1y agoHow does Spain deal with trains that come in from a neighboring country?
- hedora 1y agoThe security check has nothing to do with protecting trains or passengers, so your question is irrelevant.
- 1y ago
- torginus 1y agoUnfortunately it's not, and it gets more difficult the more cloud-y your app gets. You can pay for EC2+EBS+network costs, or you can have a fancy cloud native solution where you pay for Lambda, ALBs, CloudWatch, Metrics, Secret Manager, (things you assume they would just give you, like if you eat at a restaurant, you probably won't expect to pay for the parking, toilet, or paying rent for the table and seats). So cloud billing is its own science and art - and in most orgs devs don't even know how much the stuff they're building costs, until finance people start complaining about the monthly bills.
- jmaker 1y agoWe run regular FinOps meetings within departments, so everyone’s aware. I think everyone should. But it’s a lot of overhead of course. So a dev is concerned not only with DevOps anymore but with DevSecFinOps. Not everyone can cope with so many aspects at once. There’s a lot of complexity creep in that.
- torginus 1y agoYeah, AWS has the billing panel, that's where I usually discover that after I make a rough estimate on how much the thing I'm building should cost by studying the relevant tables, I end up with stuff costing twice as much, because on top of the expected items there's always a ton of miscellaneous stuff I never thought about.
- YouAreWRONGtoo 1y ago[dead]
- UltraSane 1y agoI have Claude, ChatGPT, and Gemini analyze our AWS bills and usage metrics once a month and they are surprisingly good at finding savings.
- jrs235 1y agoI have a fair idea of how much our cloud costs. However, others in the org don't want me to actually see how much it costs. Which is bonkers. I can help identify how we could rearchitect or investigate lowering our costs, if I know what they are and where to look first. No point guessing and accidentally optimizing something that will be an order of magnitude or more less beneficial.
- hinkley 1y agoMy last team decided to hand manage a Memcached cluster because it cost half as much as an unmanaged service versus AWS’s alternative. Don’t know how much we really saved versus opportunity cost on dev time though. But it’s close to negative.
- jmaker 1y agoOne of the issues there is that pricing a managed service deprives your people or gaining extra experience. There’s a synergy over time, the more you manage yourself. But it’s totally justified to pick a managed service if it checks out for your budget. The problem I saw often emanate was bad decision making, bad opportunity cost estimation. In other words, there’s an opportunity cost to picking the managed service, too, and they offset each other more or less.
- hinkley 1y agoI wonder if there’s enough space for a Do Well By Doing Good company out there to provide a ladder from cheap self managed up to fully automated rolling upgrades. Because it was mostly fine at first, but later we had some close calls when there were changes that needed to be made on the servers. By the time we managed to mess up our hand managed incremental restart process, we had several layers of cache and so accidentally wiping one didn’t murder our backend, but did throw enough alerts to cause a P2. And because we were doing manual bucketing of caches instead of consistent hashing we hit the OOMKiller a couple times while dialing in. But at this point it was difficult to move back to managed. This feels closest to digital ocean’s business model.
- winrid 1y agoThat should be extremely low maintenance...
- hinkley 1y agoAs long as we didn’t need to touch the machines, nearly zero. But you gotta touch them sometime.
- jmaker 1y agoIt’s a marketing trap. But also a job guarantee since everyone’s in the same trap. You got a couple cloud engineers or "DevOps" that lobby for AWS or any other hyperscaler, NaiveDate managers that write down some decision report littered with logical fallacies, and a few years in the sink cost is so high you can’t get off of it, and instead of doing productivity work you’re sitting in myriads of FinOps meetings, where even fewer understand what’s going on. Engineering mangers are promised cost savings on the HR level. Corporate finance managers are promised OpEx for CapEx trade-off, the books look better immediately. Cloud engineers are embarking on their AWS journey of certification being promised an uptick to their salaries. It’s a win/win for everyone, in isolation, a local optimum for everyone, but the organization now has to pay way more than it—hypothetically—would have been paying for bare metal ops. And hypothetical arguments are futile. And it lends itself well to overengineering and the microservices cargo cult. Your company ends up with a system distributed around the globe across multiple AZs per region of business operations, striving to shave off those 100ms latency off your clients’ RTT. But it’s outgrown your comprehension, and it’s slow anyway, and you can’t scale up because it’s expensive. And instead of having one problem, you now have 99 and your bill is one.
- geodel 1y agoAll great points. I have seen in company of smart people CIO/CTO would freely up admit "Look we know cloud may not be cheap or easier to manage but this is the direction we have taken since we are getting out of owning or managing hardware/datacenter" So it is not like one can dazzle decision makers with any logic or hard data. They are just announcing the decision while calling it a robust discussion over pros and cons of on-prem vs cloud placement.
- jmaker 1y agoYep. I’ve also seen managerial people worship AWS sales reps as oracles, misconstruing ordinary sales meetings with them as something divine, in which they would disclose a lot of company’s IP in awe for them, just to listen to some blabbing superficial truisms. I mean, ChatGPT could tell you more. To add insult to that, the managerial people wouldn’t listen to their own senior, staff, principal engineers, and prefer to follow what the AWS reps told them. It’s really disturbing how the human factor controls decision making in corporations. For my peace of mind, I chose a sane path - if the company as an entity decides to do AWS, I will do my best to meet its goals. I’ve got all Professional and Specialty certs. It’s the human nature. No purpose in tilting at windmills.
- wkat4242 1y agoYeah our company has recently decided to go all cloud (we're always so late with this stuff, especially now that a moment back from cloud has already started emerging). But for me it's totally ruining my job to be honest. I like technology because it enables me to make things. I don't want to become an aws or azure specialist and learn what tickboxes the overlords at Amazon and Microsoft allow me to click. Screw that. That's nothing to do with technical knowledge, it's just about being a corporate drone. In my particular case it's Microsoft. Another problem with this is that they know they own everything in our company now so they're starting to treat us as employees, giving us things to do. Like promoting their features inside our company. I mean, they're a vendor FFS. They should answer to us. A lot of my colleagues are really motivated with this shit and doing all the certs. Many are even becoming Microsoft evangelists and get pissed if I criticise someone. I'm looking for other options in the company now where I can actually do something technical again. I understand there might be no bare metal work left in this company but in that case I'll just want to do something else. I don't want to be some goon that links their entire career to using the products of one big tech company. And also I think Microsoft and Amazon are horrible companies to work with as a customer. So me as a techie I just don't want to work at that anymore. What will remain are a lot of yes men who know how to click boxes. And yeah we're not even doing anything smart or taking advantage of what the cloud offers. We just lifted all the physical stuff to compute instances that run 24/7.