4 ms·

My big problem with all the SBOM efforts is that any kind of compliance/accuracy will be best effort and most likely wrong, leading to more problems and blame.

by dlor 6y ago

My big problem with all the SBOM efforts is that any kind of compliance/accuracy will be best effort and most likely wrong, leading to more problems and blame. This is not as simple as writing down your dependencies. Most people don't even know what their full set of transitive dependencies is, or how to even go about finding it. How do you know the SBOM you get is even accurate? You can't just crack open a binary and look at what's inside. If you could, we wouldn't need these giant complicated file formats.

fulafel 6y ago
You can, in fact, crack open the binaries and look at what's inside. The field of tooling for it is called SCA (software composition analysis).
- dlor 6y ago
  Sort of. The quality of the data this tooling generates varies GREATLY among languages, build systems and environments. For packaged software like Solarwinds, sure you can try to run an SCA tool. But is anyone claiming an SBOM or SCA tool could have prevented that attack? The bigger issue is services and hosted software. You can't crack open an API or website that stores your data to see what database they're using. You could ask that they publish an SBOM, but who knows if it's accurate.
  - fulafel 6y ago
    I feel you're moving the goalposts a bit. Perfect is the enemy of good, etc. Also surely the tooling would get a lot of investment and improvement poured into it if the proposal went through. Anyway, if this kind of thing really took off, I could well imagine there being regulation for SaaS products having to do audits involving this, for example.
    - dlor 6y ago
      I sort of see this as a situation where an imperfect SBOM is worse than nothing. It would do nothing but add false confidence. I still haven't seen an example of a single supply-chain attack that an SBOM would have prevented.
      - fulafel 6y ago
        Well, we were discussing tooling that could be used to check if the declared SBOM is correct, not producing the original SBOM. This kind of checking with today's practices is necessarily going to be imperfect, just like the BOMs in the physical manufacturing realm where the idea originates in. But if today's 99% solution turns out to be sufficiently useful, we could start making things in a way that are 100% verifiable (stuff like reproducible builds, etc) In security we've long ago let go of the idea of risk and turst as binary issues, the same thing applies here. Just about every other tool we have to improve security has bigger holes in it than this one.
      - jacques_chester 6y ago
        We already have false confidence problems. Security scanning is a billion-dollar industry based on looking up digests in a table. But because the table is maintained by third parties, their incentives are to always be over-cautious. If they give false positives, the burden falls on their customers or the upstream dependency. But false negatives fall on the vendor. So they create noise. SBOMs from the upstream push the cost back to the upstream and (sorry, investors and founders) vitiate the necessity of those third-party scanning vendors. The incentives change and so too, I expect, would the behaviour.
- ozim 6y ago
  Technically you are right. Question is who is going to pay for that? In my job we dealt with enterprise customers that required list of all libraries we use and what license those have. But they had buckets of money to spend on compliance.
  - fulafel 6y ago
    Are you asking if customers are willing to pay more for products that feature a SBOM? I think this is more a regulation idea.
    - ozim 6y ago
      My main point is that in places where it is needed software is already regulated because there are people who made their risk management and are willing to pay for it. Automotive and banking do a lot of dependency checking, they have a lot of quality gateways. Of course they still fail because usually they have so much software to check that they would not have enough developers in the world to check all (take into account that you also need developers competent in that area). They have to pick their battles and cover most important parts of their operations. So now volume of software to be checked is more than we have man-hours of developers in the world. Saying that you can do that for every software system and do it at least decently is naive. We can put regulation in place but then we have to stop all software development in the world. This way "You can, in fact, crack open the binaries and look at what's inside." is true if you have a single binary but if we speak about organization that depends on thousands of applications problem is exploding to not managable scale.
      - fulafel 6y ago
        I strongly disagree that it's already regulated where needed. Eg cheap IoT devices. The "stop all software development"... No. You just have the sbom as a design requirement and design it in. It's not rocket science. It's regulation, like always there will be a schedule for it, so compliance can be ready when it comes into force.
jmull 6y ago
> Most people don't even know what their full set of transitive dependencies is, or how to even go about finding it. I think that’s the point. Also: you really do know your direct dependencies since you need them to build your software. If the efforts to promote or require SBOM are successful, your dependencies will all have SBOM and your tooling will be update to help you generate yours.
- dlor 6y ago
  I don't think that's true in practice. Try it. I did here: https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 It's basically impossible with today's tooling and practices to come up with a list of dependencies for a moderately complex application.
  - Ericson2314 6y ago
    Not true! We do this with Nix and Guix all the time. Any regulation that tries to allow for Docker or trad distros will, yes, fail. But if it raises the bar so only things with sandboxed build steps will qualify, its perfectly possible. This is why it's really important to stear this conversation so the upset procurers don't make some shoddy thing influenced by the whinging of existing contractors, and stick with their gut instincts.

by dlor 6y ago

fulafel 6y ago

You can, in fact, crack open the binaries and look at what's inside. The field of tooling for it is called SCA (software composition analysis).

dlor 6y ago
Sort of. The quality of the data this tooling generates varies GREATLY among languages, build systems and environments. For packaged software like Solarwinds, sure you can try to run an SCA tool. But is anyone claiming an SBOM or SCA tool could have prevented that attack? The bigger issue is services and hosted software. You can't crack open an API or website that stores your data to see what database they're using. You could ask that they publish an SBOM, but who knows if it's accurate.
- fulafel 6y ago
  I feel you're moving the goalposts a bit. Perfect is the enemy of good, etc. Also surely the tooling would get a lot of investment and improvement poured into it if the proposal went through. Anyway, if this kind of thing really took off, I could well imagine there being regulation for SaaS products having to do audits involving this, for example.
  - dlor 6y ago
    I sort of see this as a situation where an imperfect SBOM is worse than nothing. It would do nothing but add false confidence. I still haven't seen an example of a single supply-chain attack that an SBOM would have prevented.
    - fulafel 6y ago
      Well, we were discussing tooling that could be used to check if the declared SBOM is correct, not producing the original SBOM. This kind of checking with today's practices is necessarily going to be imperfect, just like the BOMs in the physical manufacturing realm where the idea originates in. But if today's 99% solution turns out to be sufficiently useful, we could start making things in a way that are 100% verifiable (stuff like reproducible builds, etc) In security we've long ago let go of the idea of risk and turst as binary issues, the same thing applies here. Just about every other tool we have to improve security has bigger holes in it than this one.
    - jacques_chester 6y ago
      We already have false confidence problems. Security scanning is a billion-dollar industry based on looking up digests in a table. But because the table is maintained by third parties, their incentives are to always be over-cautious. If they give false positives, the burden falls on their customers or the upstream dependency. But false negatives fall on the vendor. So they create noise. SBOMs from the upstream push the cost back to the upstream and (sorry, investors and founders) vitiate the necessity of those third-party scanning vendors. The incentives change and so too, I expect, would the behaviour.
ozim 6y ago
Technically you are right. Question is who is going to pay for that? In my job we dealt with enterprise customers that required list of all libraries we use and what license those have. But they had buckets of money to spend on compliance.
- fulafel 6y ago
  Are you asking if customers are willing to pay more for products that feature a SBOM? I think this is more a regulation idea.
  - ozim 6y ago
    My main point is that in places where it is needed software is already regulated because there are people who made their risk management and are willing to pay for it. Automotive and banking do a lot of dependency checking, they have a lot of quality gateways. Of course they still fail because usually they have so much software to check that they would not have enough developers in the world to check all (take into account that you also need developers competent in that area). They have to pick their battles and cover most important parts of their operations. So now volume of software to be checked is more than we have man-hours of developers in the world. Saying that you can do that for every software system and do it at least decently is naive. We can put regulation in place but then we have to stop all software development in the world. This way "You can, in fact, crack open the binaries and look at what's inside." is true if you have a single binary but if we speak about organization that depends on thousands of applications problem is exploding to not managable scale.
    - fulafel 6y ago
      I strongly disagree that it's already regulated where needed. Eg cheap IoT devices. The "stop all software development"... No. You just have the sbom as a design requirement and design it in. It's not rocket science. It's regulation, like always there will be a schedule for it, so compliance can be ready when it comes into force.

jmull 6y ago

> Most people don't even know what their full set of transitive dependencies is, or how to even go about finding it. I think that’s the point. Also: you really do know your direct dependencies since you need them to build your software. If the efforts to promote or require SBOM are successful, your dependencies will all have SBOM and your tooling will be update to help you generate yours.

dlor 6y ago
I don't think that's true in practice. Try it. I did here: https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 It's basically impossible with today's tooling and practices to come up with a list of dependencies for a moderately complex application.
- Ericson2314 6y ago
  Not true! We do this with Nix and Guix all the time. Any regulation that tries to allow for Docker or trad distros will, yes, fail. But if it raises the bar so only things with sandboxed build steps will qualify, its perfectly possible. This is why it's really important to stear this conversation so the upset procurers don't make some shoddy thing influenced by the whinging of existing contractors, and stick with their gut instincts.