5 ms·
We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce G
by markus_zhang 4y ago
We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce Google Cloud costs.
The savings could be in 7 digits TBH annually. There are so many low hanging fruits no one cares when the day is good. Tables that have no partition expiration that grow into hundreds or even thousands of TBs, buckets with same amount of data that no one looks at, VMs that no service uses, large clusters well above their load (e.g. only 20% used maximum -- and a full replica too), very bad ETL pipelines that cost hundreds of bucks EVERY RUN, very costly queries generated by Looker dashboards EVERY view...there are just too many.
- lazystar 4y agojust messy bread making machines, as joel spoelsky would likely put it https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
- ghaff 4y agoFinOps is definitely becoming a pretty hot topic. And, yeah, a lot of companies are realizing that they have some pretty low hanging fruit where they can cut spend significantly.
- sneak 4y agoFeels like there is a startup in there? Give it your account number, and read only access keys, and it tells you what and where and how to change things.
- deleted 4y ago[deleted]
- ghaff 4y agoThere are companies out there. I expect to see a number of them when I'm at Kubecon in a month or so.
- markus_zhang 4y agoJust a note that Google itself is doing that so one has to better than that. I think the ideal candidate is a senior data architect/ops person who has xtensive hand on experience. Sadly I'm so far from it. Sigh, I'm late for everything in my life.
- sneak 4y agoDisparaging yourself is a good way to make sure you don't achieve anything. You should stop.
- evntdrvn 4y agoFWIW the features that GCP exposes for this is way better/easier than AWS's equivalent. So much better
- markus_zhang 4y agoI never used AWS (and am just a peasant using GCP so don't know everything). What differences did you find to be extremely helpful?
- evntdrvn 4y agoGCP puts automatic analysis recommendations in the UI right next to the resources in the table of VM/etc names, and IMO the recommendations (for both upsizing and downsizing) are dramatically more accurate than what AWS gives you
- mox1 4y agoLots of players in this space. At my work we have: https://www.apptio.com/products/cloudability/ https://www.apptio.com/products/cloudability/
- markus_zhang 4y agoJust curious what do you think about the service? Is it a lot deeper than what Google itself provides? We are just using Google own service to turn down costs.
- mox1 4y agoSo Apptio is a larger application for finance people, with some cloud cost management built in. They are the main users, I just have read only access an login perhaps monthly to view budgets and cloud spend. I just have access to small bits and pieces, but what I have seen I like.
- markus_zhang 4y agoThanks for sharing!
- markus_zhang 4y agoYeah. I can think about consultants making $$ giving full analysis of cloud costs and ways to reduce them. It's actually not very technically challenging TBH. All those low hanging fruits can be grabbed by a junior enginner or even a non tech senior person who is a deep user of such services.
- jerf 4y agoI've wondered how much of the cloud spend in the world is just junk like this. I don't mean that cynically or sarcastically. The cloud makes it legitimately more easy to deploy resources; I've experienced this in my own teams. But the flip side is that it is easier to deploy things without thinking about the costs, or deploy things and forget about them, and management can just shrug and say "yes, it's cloud, it must be cheaper and better". This is by no means a cloud-unique problem. Watch any datacenter a company has been at for a few years get shut down and moved to the cloud. The stream of "hey, who owns this?" and "are we still using this?" and "I just yanked the network cable, has anyone noticed?" is morbidly amusing. Even if you think you're tracking everything, you quite likely aren't, because the owners don't update their own tracking info very well. But cloud makes this worse. So I just find myself wondering how much of AWS profit is on this sort of stuff. I myself have S3 buckets full of crap. As I do watch my spend, said buckets full of crap are in the ~10-50$/month range. At that price point it is difficult to justify engineering time to clean out the crap. But it's not hard to imagine that scaling up by three or more factors of magnitude.
- markus_zhang 4y agoYeah exactly. I feel that recklessly opening new projects and then abandoned them, or refactored multiple times with different teams, quickly added up to the number of rss no one owns. Last couple of weeks the whole data org underwent a forced upgrade of certain components on Google cloud and we got quite of clusters no one knows who owns. I don't know but I'm itchy to start finding ways to build new projects because that is the best way for techy people to grab ownership and promition.
- mprovost 4y agoWe saw this one iteration back with VMWare. Once it became easy to spin up a VM without getting a PO for a server and finding rack space, the number of "servers" in the org exploded. And of course we lost track of what they all did, and who they belonged to... but there was actually a healthy ecosystem of tools built for (and by) VMWare to manage this situation. The cloud providers seem incentivised to make tracking real costs as difficult as possible. At least with VMWare once you paid for it, it was yours.
- tnel77 4y agoI work for a tiny startup and while things are good, we are prioritizing reducing cloud costs as well. Our bill is relatively tiny right now, but as we grow it’ll become a bit more unsightly. Best to do it now and get that over with!
- moonchrome 4y agoAt a tiny startup I'd be more concerned about selling product/delivering to get the next round and keep the company running than saving few 100$ on infra cost.
- delfinom 4y agoNot every startup is funded by VC vultures that set aggressive timelines
- moonchrome 4y agoDeadlines don't matter - if you're in a startup you're trying to test the market fit and iterateing to corner the market. Worrying about micro optimizing infra cost and talking about scaling in this context in my opinion is premature optimization - root of all evil.
- vrosas 4y agoThis, and cloud security, have become my niche at my small company. Keeping these things tidy now will save us so much headache later. It's a bit of a battle fighting product for time but these are hills worth dying on imo.
- markus_zhang 4y agoInteresting. I wonder how one learns it. I guess I first need to jump to a full Cloud admin position to learn on job.
- vrosas 4y ago
- ly3xqhl8g9 4y agoStart by owning the datacenter. You might just as well have the data that no one looks at and the 20% load clusters in one, two, or a few 48U racks [1] in the room next to you. Keep whatever is latency-sensitive still on someone else's computer, i.e. the cloud. [1] Get a "server as they should be", https://oxide.computer https://oxide.computer (no affiliation)
- ehvatum 4y ago> Start by owning the datacenter But then, when uncontrolled inefficiencies need to gobble up more hardware, you would have to buy, install, network, maintain, and understand this new hardware.
- marcosdumay 4y agoUnused resources do not gobble up real hardware. Real inefficiencies do, but then you will have the more detailed data the GP is talking about.
- markus_zhang 4y agoI really want to learn in an on prem env (and learning on my own 32GB 2TB x-handed server is probably just scratching the surface). Sadly I'm a junior so could not make that call. I also have no idea how much cost the transfer to on prem will be.
- marcosdumay 4y agoNo time like today. You don't need to make a large datacenter, or to migrate everything at once. Find some low-risk services, get some consultant help you improve some room on your premises (you'll need to care about security, electricity and cooling), get a pair of network contracts and one hack of servers (or maybe even 4 servers), and move those services. Favor things that are used internally. If you are looking for 7 figures of savings just by picking the easy fruit, you will almost certainly get positive ROI on that. Just do not move critical services if you don't know what you are doing.
- thefourthchime 4y agoYep, we're in the same boat. About 4mo ago we started looking more closely at AWS costs. Tons of low-hanging fruit.
- steveBK123 4y agoI think FAANG centric folks underestimate how many fairly big orgs are in the same situation. Big top down push to move to public cloud, basically no thought to operating expenses.. and now theres a ton of low hanging fruit. Most of these same firms have very little ability to measure their costs in a timely, granular manner, and so put in only very blunt processes to try to limit costs. Last org I was at could probably re-allocate 10% of engineering staff using the cloud to a SWAT team finding cloud cost saves for 2 years before they ran out of stuff that costs 5x their salary to get fix/optimize/kill in the cloud. A lot of the same orgs CTOs sold themselves on the idea that they'd need less infra people in the cloud. This really has not been the case, and more often the cloud infra people are more expensive too. Meanwhile many need to permanently maintain an on-prem footprint for latency/legal/legacy/etc reasons.. so you end up with net more staff. I know a handful of Wall St firms that are moving a lot of "big data" type use cases back out of the cloud because its hideously expensive for their use cases.
- lazystar 4y agowell, its getting into the rearview mirror now, but spectre/meltdown-type vulnerabilities are a big motivator for leaving on-prem. even if you later leave the cloud for on-prem, the period without any phys infra means you can start on a clean slate with better equipment + better DC rental contracts. oh and it really sucks when your brand new SSD's arrive at your main DC, after a 4 month wait, you open the box and the boxes are empty - someone robbed ya. Those expenses are only linked to the infra team, but the effects reverberate through the engineering org.
- sylens 4y agoI wonder if we will also see a dropoff in attendance for big tech conferences like reInvent this year.
- steveBK123 4y agoGiven how many devs were needlessly getting sent at the shops I worked (think 20% of dev team), yes...
- 4y ago
- ryanbrunner 4y agoWe did the same thing (very small company, but outsized cloud hosting costs due to the nature of our business), and frankly the biggest cost savings for us was shifting some resources off AWS and to other providers. For us, Cloudfront was a massive expense that is without exagerration an order of magnitude more expensive than other providers, and even in the world of AWS custom pricing, it's a struggle to get cloudfront costs down to even the retail offering of other providers.
- bazmattaz 4y agoYep I’m in the same size company and we’re doing work to reduce GCP costs. You are so right, our team did a sprint or two of work recently and shaved $100k off our monthly bill which in our company is small potatoes