4 ms·

This awakened a memory from last year where a colleague and I were trying to understand where an increase in Linux memory was coming from in machines that hadn’

by tass 2y ago

This awakened a memory from last year where a colleague and I were trying to understand where an increase in Linux memory was coming from in machines that hadn’t been rebooted in a while. This was alerted to by Prometheus metrics. Even after all apps had been restarted, it persisted. Turned out to be a leak of slab memory allocations by a kernel module. That kernel module had since been updated, but all previous versions were still loaded by the kernel so the leak persisted until the next reboot. The leaky kernel module was CrowdStrike’s falcon sensor. It started a discussion where engineering had no option but to run these things for the sake of security, there were no instances where it actually caught anything, but it had the potential to cause incidents and outages.

josephcsible 2y ago
Do you mean "engineering had no option but to run these things for the sake of compliance"?
- tass 2y ago
  Compliance was the reasoning they raised, yes. Whether you require this particular tool to meet compliance, and whether that tool needs to be deployed across the entire stack might be open to interpretation though.
  - atomicnumber3 2y ago
    Open to interpretation but it often ends up something like this: 1. "Do we need to use this shovelware? Can't we achieve compliance via <a few sane steps>? 2. CTO/CEO: We can either use this software and you don't have to think about it, or we can do your thing and I'm going to hang you out to dry the second literally anything goes wrong with it. 3. "Ah. Ok nevermind." [starts updating resume] IMO it's part of this toxic culture against DIYing literally anything. Yes NIH can be bad, but so is the complete opposite where you make insane decisions (technical or money-wise) because of a culture where people are punished for trying to DIY when it does make sense. At a particularly toxic past org of mine, we actually had a "joke" (not the ha-ha kind of joke, the i can't believe we actually do this kind) where for a few key decisions we appointed someone to be the "Jesus" for that decision - someone who was already planning to quit, so they could make a decision that was correct but would get them punished, then "die for our sins", leaving us with the benefits of a correct decision having been made but without any of the political fallout of having done something that offends the sensibilities of leadership. Yes eventually you run out of messiahs to sacrifice and yes it sucked to work where a lot. But damn did they pay really well, so, yknow.
    - znpy 2y ago
      > against DIYing literally anything I get your feeling but I largely disagree: I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all). The nice thing about using an off-the-shelf product is that usually either it comes with support or there’s some kind of community where you can go and ask for help). There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions.
      - kiitos 2y ago
        > I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all) Personally I have seen far more damage done by a reflexive rejection of NIH syndrome, than the costs you're describing here. I've generally found it much easier to maintain a bespoke system built around a narrow set of requirements, than a general purpose system applied too-eagerly.
      - carbotaniuman 2y ago
        > There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions. This applies to buying off the shelf software (and other things) too. Vendor provided software is not immune from bugs or shortcomings or terrible vendors.
- nsguy 2y ago
  As far as I'm aware there's never a requirement to use a specific product/technology for compliance. The standards will require processes or something very general and the way a company complies is their choice. It's certainly possible that a company will buy a certain product because it checks some compliance box but I would expect there are many other ways to check that box.
znpy 2y ago
> CrowdStrike’s falcon sensor Oh god, is that falcon thing from crowdstrike as well?

by tass 2y ago

josephcsible 2y ago

Do you mean "engineering had no option but to run these things for the sake of compliance"?

tass 2y ago
Compliance was the reasoning they raised, yes. Whether you require this particular tool to meet compliance, and whether that tool needs to be deployed across the entire stack might be open to interpretation though.
- atomicnumber3 2y ago
  Open to interpretation but it often ends up something like this: 1. "Do we need to use this shovelware? Can't we achieve compliance via <a few sane steps>? 2. CTO/CEO: We can either use this software and you don't have to think about it, or we can do your thing and I'm going to hang you out to dry the second literally anything goes wrong with it. 3. "Ah. Ok nevermind." [starts updating resume] IMO it's part of this toxic culture against DIYing literally anything. Yes NIH can be bad, but so is the complete opposite where you make insane decisions (technical or money-wise) because of a culture where people are punished for trying to DIY when it does make sense. At a particularly toxic past org of mine, we actually had a "joke" (not the ha-ha kind of joke, the i can't believe we actually do this kind) where for a few key decisions we appointed someone to be the "Jesus" for that decision - someone who was already planning to quit, so they could make a decision that was correct but would get them punished, then "die for our sins", leaving us with the benefits of a correct decision having been made but without any of the political fallout of having done something that offends the sensibilities of leadership. Yes eventually you run out of messiahs to sacrifice and yes it sucked to work where a lot. But damn did they pay really well, so, yknow.
  - znpy 2y ago
    > against DIYing literally anything I get your feeling but I largely disagree: I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all). The nice thing about using an off-the-shelf product is that usually either it comes with support or there’s some kind of community where you can go and ask for help). There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions.
    - kiitos 2y ago
      > I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all) Personally I have seen far more damage done by a reflexive rejection of NIH syndrome, than the costs you're describing here. I've generally found it much easier to maintain a bespoke system built around a narrow set of requirements, than a general purpose system applied too-eagerly.
    - carbotaniuman 2y ago
      > There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions. This applies to buying off the shelf software (and other things) too. Vendor provided software is not immune from bugs or shortcomings or terrible vendors.
nsguy 2y ago
As far as I'm aware there's never a requirement to use a specific product/technology for compliance. The standards will require processes or something very general and the way a company complies is their choice. It's certainly possible that a company will buy a certain product because it checks some compliance box but I would expect there are many other ways to check that box.

znpy 2y ago

> CrowdStrike’s falcon sensor Oh god, is that falcon thing from crowdstrike as well?