7 ms·

I do software supply chain security consulting for several high risk companies and largely agree with this post that we must stop expecting devs to have any res

by lrvick 4y ago

I do software supply chain security consulting for several high risk companies and largely agree with this post that we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it. This will sound pretty harsh, but if your company chooses to use open source code that does not have capable, paid, full time professionals reviewing it for security and quality, then your company is signing up for that responsibility. If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. Broadly normalized negligence is still negligence. This should not be controversial, but it is. Washing hands in hospitals was once controversial too but those advocating for it had irrefutable evidence on their side. The medical industry did not want to pay the labor cost of hygiene, and we are seeing the same in the software industry. https://www.nationalgeographic.com/history/article/handwashing-once-controversial-medical-advice https://www.nationalgeographic.com/history/article/handwashi... Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. Pick one. Both are real options. Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path. Companies must stop expecting others to do their software review job for them. OSS devs already wrote the code for free because, ostensibly, it was fun. You are an ass if you ask them to do anything that is not fun, for free, to make your company safer or more money. Such actions make it not fun anymore, and make them stop entirely. I do not know why companies have code review policies for code written by peers, but if the code is 2 million lines of NPM dependencies essentially copy/pasted from randos on the internet it is suddenly okay to ship straight to prod and give said randos full control of the data or property of millions of people. We need to start calling this out as the negligence that it is.

ahtihn 4y ago
> Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something focused on your needs on stop of the standard library yourself. Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?
- deleted 4y ago
  [deleted]
- lrvick 4y ago
  Those much more often have full time paid people whose actual job is to review them. E.g the standard library of Go has security and quality review by Google. It is reasonable to assume they are investing more in supply chain security than you are in those portions. The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it. If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case. If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.
- willcipriano 4y ago
  He who cashes the check shall be the defendant in the lawsuit. If things went well the people behind the compiler, interpreter and OS wouldn't have received a dime from your successful enterprise, now that it's gone sideways you want to share?
- nomel 4y ago
  The idea is risk reduction, not removal. An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops. If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.
- UncleMeat 4y ago
  I think that a lot of people should be asking themselves about the state of vulns and other bugs in linux. In fact, I'd call the state of correctness of the linux kernel and surrounding systems to be something resembling an industry emergency. It shouldn't be possible for bugs to regress in the kernel, but they do! A lot of very ordinary best practices aren't actually followed in the kernel world.
616c 4y ago
How did you get into this type of consulting?
- lrvick 4y ago
  My software engineering and sysadmin background overlapped a lot with security, and I found flaws and recommended security solutions at every company... so I pivoted my career to full time security engineering for multiple companies. I eventually saw enough massive oversights and targeted attacks to operate as though any person or system that lacks strong accountability for their every action is compromised. Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted. I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission. I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models. Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.
mholt 4y ago
Hey, it sounds like you'd like my essay on this: The Asymmetry of Open Source: https://matt.life/writing/the-asymmetry-of-open-source https://matt.life/writing/the-asymmetry-of-open-source
nradov 4y ago
I agree that the responsibility for quality and security of open-source lies with the consumers of that code. But legal liability for negligence is another thing altogether. If the ultimate customer or user wants their vendor to accept liability for failures or defects then they need to negotiate that up front in a binding contract. If customers don't want to pay for that then that's their problem, they can take whatever garbage their vendors shovel out and then deal with the consequences.
zmmmmm 4y ago
> Companies must stop expecting others to do their job for them. OSS devs already wrote the code for free because, ostensibly, it was fun This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations. EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.
- lrvick 4y ago
  I generally advise my clients to mostly trust open source with lots of well known and documented professional eyes on it like reproducible builds of programming language compilers, standard libraries, and well maintained OS kernels. Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.
- huimang 4y ago
  It doesn't matter if it's written as a hobby project or not. People need to stop making assumptions and just expecting work to be done for free for them. No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.
  - q-big 4y ago
    > random people come by and make demands with zero plans to pay for it. Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).
vngzs 4y ago
I frequently find myself advocating for better supply chain security. But if I asked developers (or their managers) to review/audit 2,000 NPM dependencies, I would almost certainly fail. No other company they know of is doing that, and asking them to start is predicated on the entire industry being wrong. That tends to be a tough sell - even though I agree with you, convincing others is a whole 'nother ball game! What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.
- lrvick 4y ago
  I often suggest that they either review it all, pay a company like mine to review it all, or do tree shaking to remove most of the deps and make it lean enough that they can afford to maintain reviews. Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated. Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.
- HeyLaughingBoy 4y ago
  It's a "thing" in the Medical Device industry. I don't recall if it's now mandated or merely suggested in the recent FDA Cybersecurity Guidelines (still a Draft, not yet a Regulation), but creating a Software Bill of Materials for OTS/SOUP that's in your project is now recommended. Doing this allows you to track and analyze the impact of discovered vulnerabilities that may be in your shipped product. I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.
ralph84 4y ago
Well of course security consultants believe every company should hire security consultants to review every dependency in the stack all the way down. But for the people who sign the checks, do you have ROI numbers to support this approach? How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?
- lrvick 4y ago
  FWIIW I advocated for external review of security critical dependencies and hired external firms for this long before I started my own firm. The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you. As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password. https://nvd.nist.gov/vuln/detail/CVE-2018-9057 https://nvd.nist.gov/vuln/detail/CVE-2018-9057 Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely. We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients. While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-. Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of. https://arxiv.org/abs/2112.10165 https://arxiv.org/abs/2112.10165 Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point. Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk. Reviewing 100% of all deps company-wide is not the only knob one can turn.
lcw 4y ago
Why do you think this is controversial? Whether a company works with another software company that is bonded and/or a person uses OSS if something bad happens to the customer it still is reflective on the company using the software in a negative way. No company would refute that. I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death. Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.
- edgyquant 4y ago
  Do you think CTOs should be held criminally negligent if they lose data that is of national security concern? Because that’s the only way I see us getting audited dependencies as a commonality.
- lrvick 4y ago
  I would agree fintech companies, or any company managing a lot of PII, must take supply chain security much more seriously than say a fashion blog or a video game. The level of negligence is anchored to the potential for harm, but it is IMO a rare case where a successful company manages without a lot of PII or payment details. Even Deezer, who /only/ sells booze, was sued for a data breach and had to give out thousands of $10 checks in a class action. Their negligence hurt them and their users. For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs. I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.
deathanatos 4y ago
I want to first say: I agree with your comment. I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will. (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…) But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work. And in all my years of experience, I still am no closer to understanding how to fix that. > We need to start calling this out as the negligence that it is. People absolutely hate this, IME.
- Kamq 4y ago
  > their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will. The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else. Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned. I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation. edit: me -> be
- bombcar 4y ago
  The third option is to pay someone else to certify things (maybe the developers of the thing, or a third party entirely). The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.
- chrismorgan 4y ago
  > (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…) This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.
politician 4y ago
> on top of the standard library The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.
freeqaz 4y ago
I have two thoughts to share about this from what you've said: > we must stop expecting [Open Source] devs to have any responsibility for code they produce. Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of: "If we just paid these devs for support, then we'd solve the support problem!" Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it. The type of person writing libraries Open Source is not always going to be motivated by money! > Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path. There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!) Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work. Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.) That's the core principle that's been in my head for a while and that's how we're trying to build a company right now. Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important! 0: https://news.ycombinator.com/item?id=32037562 https://news.ycombinator.com/item?id=32037562 1: https://github.com/lunasec-io/lunasec https://github.com/lunasec-io/lunasec
- lrvick 4y ago
  I also am spinning up some projects and tooling to have standardized, cryptographically signed, language agnostic, vcs agnostic, and publicly publishable/searchable code reviews for any open source project. I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration. Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed. Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev https://lance.dev
voxic11 4y ago
> on top of the standard library What makes the standard library so special? Or do you advocate fully developing that in house as well?
- lrvick 4y ago
  See my other comments on this. I tend to have people focus their cycles on easy targets for supply chain attacks such as libraries where there is no evidence -anyone- is doing review. Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.
whatshisface 4y ago
>Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.
msla 4y ago
Even shorter: An entity gets to demand exactly as much from me as we agreed it could demand. My giving it more does not mean it can demand more. Therefore, without an agreement, nobody is allowed to demand anything of me. (Laws are another matter, of course, unless you bring the social contract into it.)
concordDance 4y ago
> If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. You trying to kill FOSS for end users here? > If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world. There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.
- lrvick 4y ago
  FOSS for end users is free and their own personal responsibility so I never see an issue here. When a business runs the code for others, then responsibility and liability kicks in. Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases. If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.
hulitu 4y ago
> we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it. Why talk about the future ? Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "
closeparen 4y ago
When I read these kinds of sentiments about OSS I always think of Windows desktops and Internet Explorer in places where they absolutely don't belong. Yes, it is bad to have an enormously deep and complex stack with a big fat attack surface, almost none of which is relevant to the application. That badness isn't mitigated by the stack being commercial and coming from a single vendor.

by lrvick 4y ago

ahtihn 4y ago

> Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something focused on your needs on stop of the standard library yourself. Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?

deleted 4y ago
[deleted]
lrvick 4y ago
Those much more often have full time paid people whose actual job is to review them. E.g the standard library of Go has security and quality review by Google. It is reasonable to assume they are investing more in supply chain security than you are in those portions. The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it. If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case. If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.
willcipriano 4y ago
He who cashes the check shall be the defendant in the lawsuit. If things went well the people behind the compiler, interpreter and OS wouldn't have received a dime from your successful enterprise, now that it's gone sideways you want to share?
nomel 4y ago
The idea is risk reduction, not removal. An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops. If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.
UncleMeat 4y ago
I think that a lot of people should be asking themselves about the state of vulns and other bugs in linux. In fact, I'd call the state of correctness of the linux kernel and surrounding systems to be something resembling an industry emergency. It shouldn't be possible for bugs to regress in the kernel, but they do! A lot of very ordinary best practices aren't actually followed in the kernel world.

616c 4y ago

How did you get into this type of consulting?

lrvick 4y ago
My software engineering and sysadmin background overlapped a lot with security, and I found flaws and recommended security solutions at every company... so I pivoted my career to full time security engineering for multiple companies. I eventually saw enough massive oversights and targeted attacks to operate as though any person or system that lacks strong accountability for their every action is compromised. Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted. I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission. I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models. Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.

mholt 4y ago

Hey, it sounds like you'd like my essay on this: The Asymmetry of Open Source: https://matt.life/writing/the-asymmetry-of-open-source https://matt.life/writing/the-asymmetry-of-open-source

nradov 4y ago

I agree that the responsibility for quality and security of open-source lies with the consumers of that code. But legal liability for negligence is another thing altogether. If the ultimate customer or user wants their vendor to accept liability for failures or defects then they need to negotiate that up front in a binding contract. If customers don't want to pay for that then that's their problem, they can take whatever garbage their vendors shovel out and then deal with the consequences.

zmmmmm 4y ago

> Companies must stop expecting others to do their job for them. OSS devs already wrote the code for free because, ostensibly, it was fun This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations. EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.

lrvick 4y ago
I generally advise my clients to mostly trust open source with lots of well known and documented professional eyes on it like reproducible builds of programming language compilers, standard libraries, and well maintained OS kernels. Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.
huimang 4y ago
It doesn't matter if it's written as a hobby project or not. People need to stop making assumptions and just expecting work to be done for free for them. No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.
- q-big 4y ago
  > random people come by and make demands with zero plans to pay for it. Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).

vngzs 4y ago

I frequently find myself advocating for better supply chain security. But if I asked developers (or their managers) to review/audit 2,000 NPM dependencies, I would almost certainly fail. No other company they know of is doing that, and asking them to start is predicated on the entire industry being wrong. That tends to be a tough sell - even though I agree with you, convincing others is a whole 'nother ball game! What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.

lrvick 4y ago
I often suggest that they either review it all, pay a company like mine to review it all, or do tree shaking to remove most of the deps and make it lean enough that they can afford to maintain reviews. Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated. Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.
HeyLaughingBoy 4y ago
It's a "thing" in the Medical Device industry. I don't recall if it's now mandated or merely suggested in the recent FDA Cybersecurity Guidelines (still a Draft, not yet a Regulation), but creating a Software Bill of Materials for OTS/SOUP that's in your project is now recommended. Doing this allows you to track and analyze the impact of discovered vulnerabilities that may be in your shipped product. I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.

ralph84 4y ago

Well of course security consultants believe every company should hire security consultants to review every dependency in the stack all the way down. But for the people who sign the checks, do you have ROI numbers to support this approach? How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?

lrvick 4y ago
FWIIW I advocated for external review of security critical dependencies and hired external firms for this long before I started my own firm. The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you. As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password. https://nvd.nist.gov/vuln/detail/CVE-2018-9057 https://nvd.nist.gov/vuln/detail/CVE-2018-9057 Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely. We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients. While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-. Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of. https://arxiv.org/abs/2112.10165 https://arxiv.org/abs/2112.10165 Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point. Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk. Reviewing 100% of all deps company-wide is not the only knob one can turn.

lcw 4y ago

Why do you think this is controversial? Whether a company works with another software company that is bonded and/or a person uses OSS if something bad happens to the customer it still is reflective on the company using the software in a negative way. No company would refute that. I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death. Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.

edgyquant 4y ago
Do you think CTOs should be held criminally negligent if they lose data that is of national security concern? Because that’s the only way I see us getting audited dependencies as a commonality.
lrvick 4y ago
I would agree fintech companies, or any company managing a lot of PII, must take supply chain security much more seriously than say a fashion blog or a video game. The level of negligence is anchored to the potential for harm, but it is IMO a rare case where a successful company manages without a lot of PII or payment details. Even Deezer, who /only/ sells booze, was sued for a data breach and had to give out thousands of $10 checks in a class action. Their negligence hurt them and their users. For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs. I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.

deathanatos 4y ago

I want to first say: I agree with your comment. I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will. (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…) But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work. And in all my years of experience, I still am no closer to understanding how to fix that. > We need to start calling this out as the negligence that it is. People absolutely hate this, IME.

Kamq 4y ago
> their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will. The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else. Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned. I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation. edit: me -> be
bombcar 4y ago
The third option is to pay someone else to certify things (maybe the developers of the thing, or a third party entirely). The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.
chrismorgan 4y ago
> (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…) This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.

politician 4y ago

> on top of the standard library The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.

freeqaz 4y ago

I have two thoughts to share about this from what you've said: > we must stop expecting [Open Source] devs to have any responsibility for code they produce. Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of: "If we just paid these devs for support, then we'd solve the support problem!" Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it. The type of person writing libraries Open Source is not always going to be motivated by money! > Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path. There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!) Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work. Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.) That's the core principle that's been in my head for a while and that's how we're trying to build a company right now. Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important! 0: https://news.ycombinator.com/item?id=32037562 https://news.ycombinator.com/item?id=32037562 1: https://github.com/lunasec-io/lunasec https://github.com/lunasec-io/lunasec

lrvick 4y ago
I also am spinning up some projects and tooling to have standardized, cryptographically signed, language agnostic, vcs agnostic, and publicly publishable/searchable code reviews for any open source project. I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration. Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed. Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev https://lance.dev

voxic11 4y ago

> on top of the standard library What makes the standard library so special? Or do you advocate fully developing that in house as well?

lrvick 4y ago
See my other comments on this. I tend to have people focus their cycles on easy targets for supply chain attacks such as libraries where there is no evidence -anyone- is doing review. Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.

whatshisface 4y ago

>Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.

msla 4y ago

Even shorter: An entity gets to demand exactly as much from me as we agreed it could demand. My giving it more does not mean it can demand more. Therefore, without an agreement, nobody is allowed to demand anything of me. (Laws are another matter, of course, unless you bring the social contract into it.)

concordDance 4y ago

> If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. You trying to kill FOSS for end users here? > If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world. There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.

lrvick 4y ago
FOSS for end users is free and their own personal responsibility so I never see an issue here. When a business runs the code for others, then responsibility and liability kicks in. Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases. If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.

hulitu 4y ago

> we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it. Why talk about the future ? Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "

closeparen 4y ago

When I read these kinds of sentiments about OSS I always think of Windows desktops and Internet Explorer in places where they absolutely don't belong. Yes, it is bad to have an enormously deep and complex stack with a big fat attack surface, almost none of which is relevant to the application. That badness isn't mitigated by the stack being commercial and coming from a single vendor.