4 ms·

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just fo

by Aeolun 1y ago

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

bstack 1y ago
Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.
- fock 1y ago
  how large are the clusters then?
- Aeolun 1y ago
  I’m not sure if that makes it better or worse.
  - embedding-shape 1y ago
    I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do. And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people. Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, doesn't immediately notice 1.2TB being used and tries to figure out why, when 120GB ended up being sufficient. It gives much more of a "We're a startup, we're learning as we're running" vibe which again, cool and all, but hardly what people should use for hosting their own stuff on.
    - Anonbrit 1y ago
      If your report for the month is "I saved a terabyte of ram usage across our cluster estate!" and I as a manager do some quick maths and say great, that's our income from 2 median customers. We lost 8 customers because we didn't laugh feature foo in time, which is what you were supposed to be working on, so your contribution for the month is a massive loss to the company... Does that frame things differently? There's are times in your product lifecycle where you doing want your developers looking at things like this, and a time when you do
      - pdimitar 1y ago
        Reading comments like yours makes me realise that I should just leave commercial programming and never come back. Your framing is terrifying. You know why? I am not saying that what you said does not make sense. Of course it does make sense, financially so. But! You the manager one day come to me and my team and say "How could we allow to have 7TB of unused memory sitting around and we paid for it?!" and we'll then have multiple follow-up meetings where we'll be scolded and "trained" how to avoid things like this. We'll get sent articles and told to improve. And believe me when I tell you, _all_ the techies in these meetings want to roll their eyes through it all. Because many of them likely asked "Can I take a closer look at our infra, it seems expensive and we can potentially optimise it?" and were said no by managers like yourself. As an engineer you just can't win. So I don't blame myself or any other techie who sometimes goes cowboy mode to find such problems without asking. Finally, "my contribution for the month" is technological work and nothing else. If I wanted to be a cofounder or have a seat in the board so I have fiduciary duty, I would have said so. It's your job as a manager to put this barrier between stakeholders and front soldiers so the latter can do their thing without disruption, so the organisation can succeed. Are you doing your job well?
    - bastawhiz 1y ago
      7tb in an organization running probably petabytes of ram total is easy to slip under the radar. There's a lot of systems and a lot of moving parts and if it's not broke or triggering alarms, you probably don't care very much.
  - antoniojtorres 1y ago
    It seems realistic to me, commonplace even. Lots to do in a company like this one.
devjab 1y ago
We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our contract with our microsoft partner. This company has some of the thightest control over their cloud infrastructure out of any organisation I've worked with. This is anecdotal, but if my experiences aren't unique then there is a lot of lack of reasonable in DevOps.
- ffsm8 1y ago
  Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as they can get away with, which inevitably means things are being done with borderline malicious compliance... Or just complete incompetence. I'm not even sure I'd blame these devs in particular. The devs just saw it as a quick bonus generator for the MBA in charge of this rebranding while offloading more responsibilities in their shoulders. DevOps made total sense in the work culture where this concept was conceived - Netflix was well known at that point to only ever employ senior Devs. However, in the context of the average 9-5 dev, which often knows a lot less then even some enthusiastic Jrs... Let's just say that it's incredibly dicey wherever it's successful in practice.
  - mustyoshi 1y ago
    I politely disagree. I spent maybe 8 hours over a week rightsizing a handful of heavy deployments from a previous team and reduced their peak resource usage by implementing better scaling policies. Before the new scaling policy the service would scale out and new pods would remain idle and ultimately get terminated without ever responding to a request quite frequently. The service dashboards already existed, all I had to do was a bit of load testing and read the graphs. It's not too much extra work to make sure you're scaling efficiently.
fock 1y ago
we have on-prem with heavy spikes (our batch workload can utilize the 20TB of memory in the cluster easily) and we just don't care much and add 10% every year to the hardware requested. Compared to employing people or paying other vendors (relational databases with many TB-sized tables...) this is just irrelevant. Sadly devs are incentivized by that and going towards the cloud might be a fun story. Given the environment I hope they scrap the effort sooner rather than later, buy some Oxide systems for the people who need to iterate faster than the usual process of getting a VM and replace/reuse the 10% of the company occupied with the cloud (mind you: no real workload runs there yet...) to actually improve local processes...
- g-mork 1y ago
  Somewhat unrelated, but you just tied wasteful software design to high it salaries, and also suggest a reason why Russian programmers might also seem to on the whole be far more effective than we are I wonder if msft simply cut dev salaries by 50% in the 90s, would it have had any measurable effect on windows quality by today
formerly_proven 1y ago
It probably doesn't help that the first line of treatment for any error is to blindly increase memory request/limit and claim it's fixed (preferably without looking at the logs once).

by Aeolun 1y ago

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

bstack 1y ago

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.

fock 1y ago
how large are the clusters then?
Aeolun 1y ago
I’m not sure if that makes it better or worse.
- embedding-shape 1y ago
  I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do. And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people. Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, doesn't immediately notice 1.2TB being used and tries to figure out why, when 120GB ended up being sufficient. It gives much more of a "We're a startup, we're learning as we're running" vibe which again, cool and all, but hardly what people should use for hosting their own stuff on.
  - Anonbrit 1y ago
    If your report for the month is "I saved a terabyte of ram usage across our cluster estate!" and I as a manager do some quick maths and say great, that's our income from 2 median customers. We lost 8 customers because we didn't laugh feature foo in time, which is what you were supposed to be working on, so your contribution for the month is a massive loss to the company... Does that frame things differently? There's are times in your product lifecycle where you doing want your developers looking at things like this, and a time when you do
    - pdimitar 1y ago
      Reading comments like yours makes me realise that I should just leave commercial programming and never come back. Your framing is terrifying. You know why? I am not saying that what you said does not make sense. Of course it does make sense, financially so. But! You the manager one day come to me and my team and say "How could we allow to have 7TB of unused memory sitting around and we paid for it?!" and we'll then have multiple follow-up meetings where we'll be scolded and "trained" how to avoid things like this. We'll get sent articles and told to improve. And believe me when I tell you, _all_ the techies in these meetings want to roll their eyes through it all. Because many of them likely asked "Can I take a closer look at our infra, it seems expensive and we can potentially optimise it?" and were said no by managers like yourself. As an engineer you just can't win. So I don't blame myself or any other techie who sometimes goes cowboy mode to find such problems without asking. Finally, "my contribution for the month" is technological work and nothing else. If I wanted to be a cofounder or have a seat in the board so I have fiduciary duty, I would have said so. It's your job as a manager to put this barrier between stakeholders and front soldiers so the latter can do their thing without disruption, so the organisation can succeed. Are you doing your job well?
  - bastawhiz 1y ago
    7tb in an organization running probably petabytes of ram total is easy to slip under the radar. There's a lot of systems and a lot of moving parts and if it's not broke or triggering alarms, you probably don't care very much.
- antoniojtorres 1y ago
  It seems realistic to me, commonplace even. Lots to do in a company like this one.

devjab 1y ago

We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our contract with our microsoft partner. This company has some of the thightest control over their cloud infrastructure out of any organisation I've worked with. This is anecdotal, but if my experiences aren't unique then there is a lot of lack of reasonable in DevOps.

ffsm8 1y ago
Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as they can get away with, which inevitably means things are being done with borderline malicious compliance... Or just complete incompetence. I'm not even sure I'd blame these devs in particular. The devs just saw it as a quick bonus generator for the MBA in charge of this rebranding while offloading more responsibilities in their shoulders. DevOps made total sense in the work culture where this concept was conceived - Netflix was well known at that point to only ever employ senior Devs. However, in the context of the average 9-5 dev, which often knows a lot less then even some enthusiastic Jrs... Let's just say that it's incredibly dicey wherever it's successful in practice.
- mustyoshi 1y ago
  I politely disagree. I spent maybe 8 hours over a week rightsizing a handful of heavy deployments from a previous team and reduced their peak resource usage by implementing better scaling policies. Before the new scaling policy the service would scale out and new pods would remain idle and ultimately get terminated without ever responding to a request quite frequently. The service dashboards already existed, all I had to do was a bit of load testing and read the graphs. It's not too much extra work to make sure you're scaling efficiently.

fock 1y ago

we have on-prem with heavy spikes (our batch workload can utilize the 20TB of memory in the cluster easily) and we just don't care much and add 10% every year to the hardware requested. Compared to employing people or paying other vendors (relational databases with many TB-sized tables...) this is just irrelevant. Sadly devs are incentivized by that and going towards the cloud might be a fun story. Given the environment I hope they scrap the effort sooner rather than later, buy some Oxide systems for the people who need to iterate faster than the usual process of getting a VM and replace/reuse the 10% of the company occupied with the cloud (mind you: no real workload runs there yet...) to actually improve local processes...

g-mork 1y ago
Somewhat unrelated, but you just tied wasteful software design to high it salaries, and also suggest a reason why Russian programmers might also seem to on the whole be far more effective than we are I wonder if msft simply cut dev salaries by 50% in the 90s, would it have had any measurable effect on windows quality by today

formerly_proven 1y ago

It probably doesn't help that the first line of treatment for any error is to blindly increase memory request/limit and claim it's fixed (preferably without looking at the logs once).