4 ms·

I worked for one company where we were super conservative. Every external component was versioned. Nothing was updated without review and usually after it had p

by YZF 5mo ago

I worked for one company where we were super conservative. Every external component was versioned. Nothing was updated without review and usually after it had plenty of soak time. Pretty much everything built from source code (compilers, kernel etc.). Builds [build servers/infra] can't reach the Internet at all and there's process around getting any change in. We reviewed all relevant CVEs as they came out to make a call on if they apply to us or not and how we mitigate or address them. Then I moved to another company where we had builds that access the Internet. We upgrade things as soon as they come out. And people think this is good practice because we're getting the latest bug fixes. CVEs are reviewed by a security team. Then a startup with a mix of other practices. Some very good. But we also had a big CVE debt. e.g. we had secure boots on our servers and encrypted drives. We had a pretty good grasp on securing components talking to each other etc. Everyone seems to think they are doing the right thing. It's impossible to convince the "frequent upgrader" that maybe that's a risk in terms of introducing new issues. We as an industry could really use a better set of practices. Example #1 for me is better in terms of dependency management. In general company #1 had well established security practices and we had really secure products.

dataflow 5mo ago
> Everyone seems to think they are doing the right thing I like to think people would agree more on the appropriate method if they saw the risk as large enough. If you could convince everyone that a nuclear bomb would get dropped on their heads (or a comparably devastating event) if a vulnerability gets in, I highly doubt a company like #2 would still believe they're doing things optimally, for example.
- emodendroket 5mo ago
  Really? You think the alternate mode where you're running 5-year-old versions of stuff with tons of known security flaws is better?
  - coldtea 5mo ago
    What part of "We reviewed all relevant CVEs as they came out to make a call on if they apply to us or not and how we mitigate or address them" gave you that impression?
    - emodendroket 5mo ago
      My experience with how such a strategy typically plays out in reality.
  - HeatrayEnjoyer 5mo ago
    >running 5-year-old versions of stuff with tons of known security flaws No one in this thread proposed that, or anything that could be reasonably assumed to have meant that.
- KronisLV 5mo ago
  > if they saw the risk as large enough. If you expose people to the true risks instead of allowing them to be ignorant, the conclusion that they might come to is that they shouldn’t develop software at all.
  - dataflow 5mo ago
    The assumption was obviously that they have a compelling need to develop the software. For the sake of illustration: you imagine exposing them to whatever the highest level of risk is that still makes them wiling to develop software.
KGunnerud 5mo ago
I would rather work with a company that updates continuously, while also building security into multiple layers so that weaknesses in one layer can be mitigated by others. For example, at one company I worked for, they created an ACL model for applications that essentially enforced rules like: “Application X in namespace A can communicate with me.” This ACL coordinated multiple technologies working together, including Kubernetes NetworkPolicies, Linkerd manifests with mTLS, and Entra ID application permissions. As a user, it was dead simple to use and abstracted away a lot of things i do not know that well. The important part is not the specific implementation, but the mindset behind it. An upgrade can both fix existing issues and introduce new ones. However, avoiding upgrades can create just as many problems — if not more — over time. At the same time, I would argue that using software backed by a large community is even more important today, since bugs and vulnerabilities are more likely to receive attention, scrutiny, and timely fixes.
- 12_throw_away 5mo ago
  > I would rather work with a company that updates continuously, while also building security into multiple layers so that weaknesses in one layer can be mitigated by others. Sorry, if you are updating continuously, that there is at least one layer that you failed to build any security into.
whilenot-dev 5mo ago
You forgot case #4: Worked at a startup where the frontend team thought it was a good idea to use lock files during development, but to do a "fresh" install of all dependecies during the deployment step. And yes, they still thought they were doing the right thing.
- hennell 5mo ago
  To be fair npm makes (made?) it weirdly hard to use lock files so a lot of people did that by mistake. And when you do use lock, it reinstalls every time so a retagged package can just silently update.
  - whilenot-dev 5mo ago
    FYI a retagged package would result in a different SHA512 integrity sum and fail the installation process. It won't "just silently update". Anyway, the point of parent and me wasn't that it was considered to be a "mistake", but people thinking they "are doing the right thing".
  - user34283 5mo ago
    I can’t comment on the behavior of ancient npm versions, but with modern npm I would not even know how to skip using a lockfile. As for the parent comment about not using the lockfile for the production build, that’s just incredibly incompetent. Maybe they should hire someone who knows what they are doing. Contrary to the popular beliefs of backend engineers online, you also need some competency to do frontend properly. In this case what’s needed is „npm ci“ instead of „npm install“ or better „pnpm install —frozen-lockfile“. Pnpm will also do that automatically if the CI environment variable is set.
    - cxr 5mo ago
      > In this case what’s needed is „npm ci“ instead of „npm install“ or better „pnpm install —frozen-lockfile“. The grugbrain developer says, "I can use git-add to keep a version controlled copy of the library in my app's source tree with no extra steps after git-clone." (Pop quiz: what problem were the creators of NPM's lockfile format trying to solve?)
echelon_musk 5mo ago
Do you ride an R1?
- YZF 5mo ago
  I used to ride a 600R ... yes- that's where my handle comes from ;)
ndsipa_pomu 5mo ago
> It's impossible to convince the "frequent upgrader" that maybe that's a risk in terms of introducing new issues I would count myself as a "frequent upgrader" - I admin a bunch of Ubuntu machines and typically set them to auto-update each night. However, I am aware of the risks of introducing new issues, but that's offset by the risks of not upgrading when new bugs are found and patched. There's also the issue of organisations that fall far behind on versions of software which then creates an even bigger problem, though this is more common with Windows/proprietary software as you have less control over that. At least with Linux, you can generally find ways to install e.g. old versions of Java that may be required for specific tools. There's no simple one-size-fits-all and it depends on the organisation's pool of skills as to whether it's better to proactively upgrade or to reluctantly upgrade at a slower pace. In my experience, the bugs introduced by new versions of software are easier to fix/workaround than the various issues of old software versions.
shevy-java 5mo ago
> It's impossible to convince the "frequent upgrader" that maybe that's a risk in terms of introducing new issues. Well, you critisize people who run the latest software here. Two counter-arguments: 1) If you don't upgrade frequently, you end up with super stable debian stuck on ... ancient software. This in turn means that many more recent software, won't work, unless you recompile a lot. I had this issue with mesa for instance, then needing a more recent LLVM, spirv-components and so forth. No chance to have that easily on debian, unless you control what you compile. On my local system here I run gtk2, gtk3 and gtk4 just fine. Good luck having that with debian for recent version; even debian sid is slow compared to, say, gentoo or arch(linux) or void(linux) here. 2) Even debian systems would be vulnerable to copy.fail. So that strategy is also not automatically better. Personally I am among the frequent update folks. I use ruby scripts to automatically update to the latest, in hope that the people who write code are not incompetent. There is no guarantee that newer software is automatically always better; it is a trade-off. I don't have the time and resource for infinite security audits. I need to get things done and this approach, different to the "everything is scary" crowd, works super-well for me. I use a versioned AppDir approach on linux though, so I don't run into many issues of "can not upgrade because of same .so name issue", so I can conveniently switch to other versions as-is, including the kernel. (Excluding ABI differences and glibc, but for about 98% of the programs this works very well. I am also not alone with the get-everything-working approach, see xserver or gtk2-ng: https://github.com/X11Libre/xserver https://github.com/X11Libre/xserver https://git.devuan.org/Daemonratte/gtk2-ng https://git.devuan.org/Daemonratte/gtk2-ng - granted, for the linux kernel this does not work that well ... I think we need better strategies for the linux kernel, things such as copy.fail should not be possible. I have no good solution here, AI will find many more exploits. No clue how we can prevent this or mitigate this more easily. I was surprised when the local instructor showed us how easy it is to use python for gaining superuser access as-is.)
- anthk 5mo ago
  Debian had backports since forever. You could totally upgrade the kernel and MESA since month 1.
  - tremon 5mo ago
JTbane 5mo ago
Honestly the de facto standard is to blame: at my dev job, we vacuum up all the packages we need and get the software deployed to production ASAP, then later go over the SBOM and make sure nothing looks sketchy. I'd imagine this is the default most places; an intensive approval process would slow down CI/CD too much.
bshanks 5mo ago
What's the conservative best practice for a solo founder (a one person company; assume the person spends ~20 hours per week on software development/maintenance and the rest of the time on other stuff)?
pseudohadamard 5mo ago
It's interesting how standards have changed. 25 years ago people screamed about ActiveX despite Microsoft's various attempts to lock it down. Today people use something that pulls a package from a Github repo that was abandoned eight years ago that pulls a package from an FTP server in Denmark that pulls a package from a Raspberry Pi in someone's basement in New Mexico and it's all good practice.

by YZF 5mo ago

dataflow 5mo ago

> Everyone seems to think they are doing the right thing I like to think people would agree more on the appropriate method if they saw the risk as large enough. If you could convince everyone that a nuclear bomb would get dropped on their heads (or a comparably devastating event) if a vulnerability gets in, I highly doubt a company like #2 would still believe they're doing things optimally, for example.

emodendroket 5mo ago
Really? You think the alternate mode where you're running 5-year-old versions of stuff with tons of known security flaws is better?
- coldtea 5mo ago
  What part of "We reviewed all relevant CVEs as they came out to make a call on if they apply to us or not and how we mitigate or address them" gave you that impression?
  - emodendroket 5mo ago
    My experience with how such a strategy typically plays out in reality.
- HeatrayEnjoyer 5mo ago
  >running 5-year-old versions of stuff with tons of known security flaws No one in this thread proposed that, or anything that could be reasonably assumed to have meant that.
KronisLV 5mo ago
> if they saw the risk as large enough. If you expose people to the true risks instead of allowing them to be ignorant, the conclusion that they might come to is that they shouldn’t develop software at all.
- dataflow 5mo ago
  The assumption was obviously that they have a compelling need to develop the software. For the sake of illustration: you imagine exposing them to whatever the highest level of risk is that still makes them wiling to develop software.

KGunnerud 5mo ago

I would rather work with a company that updates continuously, while also building security into multiple layers so that weaknesses in one layer can be mitigated by others. For example, at one company I worked for, they created an ACL model for applications that essentially enforced rules like: “Application X in namespace A can communicate with me.” This ACL coordinated multiple technologies working together, including Kubernetes NetworkPolicies, Linkerd manifests with mTLS, and Entra ID application permissions. As a user, it was dead simple to use and abstracted away a lot of things i do not know that well. The important part is not the specific implementation, but the mindset behind it. An upgrade can both fix existing issues and introduce new ones. However, avoiding upgrades can create just as many problems — if not more — over time. At the same time, I would argue that using software backed by a large community is even more important today, since bugs and vulnerabilities are more likely to receive attention, scrutiny, and timely fixes.

12_throw_away 5mo ago
> I would rather work with a company that updates continuously, while also building security into multiple layers so that weaknesses in one layer can be mitigated by others. Sorry, if you are updating continuously, that there is at least one layer that you failed to build any security into.

whilenot-dev 5mo ago

You forgot case #4: Worked at a startup where the frontend team thought it was a good idea to use lock files during development, but to do a "fresh" install of all dependecies during the deployment step. And yes, they still thought they were doing the right thing.

hennell 5mo ago
To be fair npm makes (made?) it weirdly hard to use lock files so a lot of people did that by mistake. And when you do use lock, it reinstalls every time so a retagged package can just silently update.
- whilenot-dev 5mo ago
  FYI a retagged package would result in a different SHA512 integrity sum and fail the installation process. It won't "just silently update". Anyway, the point of parent and me wasn't that it was considered to be a "mistake", but people thinking they "are doing the right thing".
- user34283 5mo ago
  I can’t comment on the behavior of ancient npm versions, but with modern npm I would not even know how to skip using a lockfile. As for the parent comment about not using the lockfile for the production build, that’s just incredibly incompetent. Maybe they should hire someone who knows what they are doing. Contrary to the popular beliefs of backend engineers online, you also need some competency to do frontend properly. In this case what’s needed is „npm ci“ instead of „npm install“ or better „pnpm install —frozen-lockfile“. Pnpm will also do that automatically if the CI environment variable is set.
  - cxr 5mo ago
    > In this case what’s needed is „npm ci“ instead of „npm install“ or better „pnpm install —frozen-lockfile“. The grugbrain developer says, "I can use git-add to keep a version controlled copy of the library in my app's source tree with no extra steps after git-clone." (Pop quiz: what problem were the creators of NPM's lockfile format trying to solve?)

echelon_musk 5mo ago

Do you ride an R1?

YZF 5mo ago
I used to ride a 600R ... yes- that's where my handle comes from ;)

ndsipa_pomu 5mo ago

> It's impossible to convince the "frequent upgrader" that maybe that's a risk in terms of introducing new issues I would count myself as a "frequent upgrader" - I admin a bunch of Ubuntu machines and typically set them to auto-update each night. However, I am aware of the risks of introducing new issues, but that's offset by the risks of not upgrading when new bugs are found and patched. There's also the issue of organisations that fall far behind on versions of software which then creates an even bigger problem, though this is more common with Windows/proprietary software as you have less control over that. At least with Linux, you can generally find ways to install e.g. old versions of Java that may be required for specific tools. There's no simple one-size-fits-all and it depends on the organisation's pool of skills as to whether it's better to proactively upgrade or to reluctantly upgrade at a slower pace. In my experience, the bugs introduced by new versions of software are easier to fix/workaround than the various issues of old software versions.

shevy-java 5mo ago

> It's impossible to convince the "frequent upgrader" that maybe that's a risk in terms of introducing new issues. Well, you critisize people who run the latest software here. Two counter-arguments: 1) If you don't upgrade frequently, you end up with super stable debian stuck on ... ancient software. This in turn means that many more recent software, won't work, unless you recompile a lot. I had this issue with mesa for instance, then needing a more recent LLVM, spirv-components and so forth. No chance to have that easily on debian, unless you control what you compile. On my local system here I run gtk2, gtk3 and gtk4 just fine. Good luck having that with debian for recent version; even debian sid is slow compared to, say, gentoo or arch(linux) or void(linux) here. 2) Even debian systems would be vulnerable to copy.fail. So that strategy is also not automatically better. Personally I am among the frequent update folks. I use ruby scripts to automatically update to the latest, in hope that the people who write code are not incompetent. There is no guarantee that newer software is automatically always better; it is a trade-off. I don't have the time and resource for infinite security audits. I need to get things done and this approach, different to the "everything is scary" crowd, works super-well for me. I use a versioned AppDir approach on linux though, so I don't run into many issues of "can not upgrade because of same .so name issue", so I can conveniently switch to other versions as-is, including the kernel. (Excluding ABI differences and glibc, but for about 98% of the programs this works very well. I am also not alone with the get-everything-working approach, see xserver or gtk2-ng: https://github.com/X11Libre/xserver https://github.com/X11Libre/xserver https://git.devuan.org/Daemonratte/gtk2-ng https://git.devuan.org/Daemonratte/gtk2-ng - granted, for the linux kernel this does not work that well ... I think we need better strategies for the linux kernel, things such as copy.fail should not be possible. I have no good solution here, AI will find many more exploits. No clue how we can prevent this or mitigate this more easily. I was surprised when the local instructor showed us how easy it is to use python for gaining superuser access as-is.)

anthk 5mo ago
Debian had backports since forever. You could totally upgrade the kernel and MESA since month 1.
- tremon 5mo ago

JTbane 5mo ago

Honestly the de facto standard is to blame: at my dev job, we vacuum up all the packages we need and get the software deployed to production ASAP, then later go over the SBOM and make sure nothing looks sketchy. I'd imagine this is the default most places; an intensive approval process would slow down CI/CD too much.

bshanks 5mo ago

What's the conservative best practice for a solo founder (a one person company; assume the person spends ~20 hours per week on software development/maintenance and the rest of the time on other stuff)?

pseudohadamard 5mo ago

It's interesting how standards have changed. 25 years ago people screamed about ActiveX despite Microsoft's various attempts to lock it down. Today people use something that pulls a package from a Github repo that was abandoned eight years ago that pulls a package from an FTP server in Denmark that pulls a package from a Raspberry Pi in someone's basement in New Mexico and it's all good practice.