4 ms·

Someone linked to Amber Brown's talk and I thought this point would be important for discussion. I think there is something that isn't being considered here whe

by k4ch0w 7y ago

Someone linked to Amber Brown's talk and I thought this point would be important for discussion. I think there is something that isn't being considered here when it comes to relying on PyPi. 3PP(Third Party Packages) issues are responsible for a lot of application security vulnerabilities today. Every large enterprise organization has no idea what packages a developer is pulling onto their laptop and into their codebase. As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. You have no idea who is on the other end of that push too. It's too hard to keep track of changes and causes us to have to pin packages and versions that have been ok'd for use instead of a new release of Python. You have no guarantee the security engineer who reviewed the code didn't miss anything either.

jahewson 7y ago
There’s really two separate concepts here: review and the standard library. Why not separate concerns and have a review process that isn’t coupled to the standard library? There’s no reason why any PyPI package can’t have meaningful reviews published. The question that you’re ultimately seeking the answer to is “what code has been reviewed and which reviewers do I trust?” - lots of ways to solve that.
- anoncake 7y ago
  One reason for this proposal is to save manpower. Separating concerns doesn't help with that.
  - kgwgk 7y ago
    It does save manpower for Python maintainers if reviewing non-core libraries becomes someone else’s problem.
    - anoncake 7y ago
      Either the reviewer is trusted and might as well review the libraries as part of the standard library or they are not trusted and reviews by them are useless.
  - afiori 7y ago
    But keeping them together does not need a standard library, each python distribution could have a set of blessed packages, those packages could be "unblessed" without breaking applications using them
  - lmm 7y ago
    If the two tasks require different skillsets (and they do) then separating them could easily mean more people are able and willing to help with one or the other.
- Kalium 7y ago
  > There’s no reason why any PyPI package can’t have meaningful reviews published You are completely correct. There's no reason at all why any random PyPI package can't have meaningful reviews published. I would go so far as to say that this is true without any changes to the standard library or current processes at all. With that in mind, I'm thinking about all the various packages my colleages use. I don't think I've seen published reviews for an appreciable fraction of them in any language. This suggests that perhaps enabling reviews might not be the hard problem to solve here. This thought process also hilights to me that the major advantage of a stdlib is that you have a higher degree of assurance that its contents have seen meaningful review by multiple sets of eyes. It's not just the potential for review that matters, it's the degree of assurance.
- alexchamberlain 7y ago
  I wonder if something like Arch Linux's main repository vs AUR would work?
- zwp 7y ago
  > “what code has been reviewed and which reviewers do I trust?” Thank you for putting this so succinctly. There are other supply-chain problems in decentralizing control of the modules though: Modules might only have a few people or one person responsible for them. Consequently suspect commits by a malfeasant or compromised (hacked) team member might not be noticed. Maybe this is a variant of the old saw "with enough eyes all bugs are shallow" but my gut feeling is that commits to Python core will get more inspection than those to a small 3rd party module. Today, modules get some review for free, just by being part of stdlib. Even if I know and trust @jdoe I have less assurances that they didn't get phished and their repo tweaked. Also trusted team members/organizations may change over time. The browser extensions world is the poster child for this, where we've seen not only similarly-named malicious extensions posted to stores but also once-legitimate extensions quietly purchased and subsequently subverted by bad actors. This is one example (adware: could've been much worse): https://www.bleepingcomputer.com/news/security/-particle-chrome-extension-sold-to-new-dev-who-immediately-turns-it-into-adware/ https://www.bleepingcomputer.com/news/security/-particle-chr... I like the idea of a review process but I have a hard time imagining a crowd-sourced system that wouldn't get gamed. We have a "dissolution of responsibility" problem: millions of companies rely on these components but have no explicit responsibility of care. Perhaps that needs to change, somehow.
  - pas 7y ago
    Companies (and anyone interested) could simply pool funds together and use that to ask already trusted and respected auditors to do reviews. And anyone who pays into the fund should be able to vote on what package to review. (And there should be a weighted lottery, so eventually small contributors' wishes have a chance to get fulfilled.) For good measure anyone can put this on the blockchain, make a flattr/patreon thing out of this. (Somehow use github sponsoring...) Who knows.
tialaramex 7y ago
I suspect you're gravely over-estimating how much value there is in the dustier "batteries included" modules over an out of tree component for security. The problem is that security assumptions change. If you're looking at some code in 1995 that accesses an API, it seems reasonable that it disables validation for the SSL library it brings in to access an HTTPS URL. Handling that properly would have been a huge pain, faster is better. But in 2019 you'd be appalled, how is this thing not depending on certifi and requests? As it is it's wide open to a MitM attack and you'd get blind sided. A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() and in 1970s Unix I'm sure this was invaluable. But obviously today "copying strings into fixed width buffers with no terminator" is essentially never what you wanted, but people assume based on the name that strncpy() will do some other thing they do want, and insert a security defect into their code.
- geezerjay 7y ago
  > A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() Arguably, what makes C the perfect example is not necessarily its standard library functions per se but the language's official string data type.
ivoflipse 7y ago
Then maybe those large enterprise organizations should start paying for the long mature process with experienced developers
geofft 7y ago
> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. This is an appealing sentiment, but you will get hacked if you rely on it. Python 2's standard library didn't do SSL certificate verification by default until the fairly controversial PEP 476 from late 2014, which changed the default in a point release of Python 2.7. Through 2014 (and for some time later, given how slow it is to get new upstream releases through distros onto someone's computer), the standard thing to do to use HTTPS securely was to use requests, a module developed by someone who got commit privileges taken away on another of his projects for making multiple releases a day. You were more secure with his code than with the standard library. I think you have no more idea who's on the other end of pushes to the standard library than who's on the other end of pushes to PyPI. (Which is to say, in part, that you can equally well have an idea of both of these if you put effort into it. I've met the maintainers of several of my third-party dependencies at PyCon.) Something being in the standard library doesn't mean it has a higher class of developer behind it. There's more overhead, but it doesn't mean there's more maturity; many third-party module developers are more experienced (either in their field, if it's something like crypto, or just in general as responsible developers) than standard library developers. And if anything it means that security updates are slower and rarer because the process is more painful. Consider the argument of the 2009 paper "Security Impact Ratings Considered Harmful" https://arxiv.org/abs/0904.4058 https://arxiv.org/abs/0904.4058 (disclosure: I'm a coauthor). Whether people find a vulnerability important enough to patch has little correlation in practice with how exploitable it is. And the only thing that gets regular attention is the latest development version of the code. If some module was significantly refactored or reimplemented, no upstream developer is looking for bugs in the old version. So, if you want to be safe and you haven't personally both audited and fuzzed the code you're running, you actually want to be running the latest released version of that code, regardless of whether someone stuck a CVSS on the old version yet. I've met very few companies who can upgrade to the latest Python 3.x minor release promptly when it comes out (and several who are still on Python 2!). I've met many companies who can pip install the latest versions of their dependencies without too much trouble, though.
Scarblac 7y ago
> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, Yes, but do you like funding them? This is all freely downloadable open source software.
- newen 7y ago
  Plenty of people fund the Python Software Foundation. Their revenue is around 3 million dollars a year.

by k4ch0w 7y ago

jahewson 7y ago

There’s really two separate concepts here: review and the standard library. Why not separate concerns and have a review process that isn’t coupled to the standard library? There’s no reason why any PyPI package can’t have meaningful reviews published. The question that you’re ultimately seeking the answer to is “what code has been reviewed and which reviewers do I trust?” - lots of ways to solve that.

anoncake 7y ago
One reason for this proposal is to save manpower. Separating concerns doesn't help with that.
- kgwgk 7y ago
  It does save manpower for Python maintainers if reviewing non-core libraries becomes someone else’s problem.
  - anoncake 7y ago
    Either the reviewer is trusted and might as well review the libraries as part of the standard library or they are not trusted and reviews by them are useless.
- afiori 7y ago
  But keeping them together does not need a standard library, each python distribution could have a set of blessed packages, those packages could be "unblessed" without breaking applications using them
- lmm 7y ago
  If the two tasks require different skillsets (and they do) then separating them could easily mean more people are able and willing to help with one or the other.
Kalium 7y ago
> There’s no reason why any PyPI package can’t have meaningful reviews published You are completely correct. There's no reason at all why any random PyPI package can't have meaningful reviews published. I would go so far as to say that this is true without any changes to the standard library or current processes at all. With that in mind, I'm thinking about all the various packages my colleages use. I don't think I've seen published reviews for an appreciable fraction of them in any language. This suggests that perhaps enabling reviews might not be the hard problem to solve here. This thought process also hilights to me that the major advantage of a stdlib is that you have a higher degree of assurance that its contents have seen meaningful review by multiple sets of eyes. It's not just the potential for review that matters, it's the degree of assurance.
alexchamberlain 7y ago
I wonder if something like Arch Linux's main repository vs AUR would work?
zwp 7y ago
> “what code has been reviewed and which reviewers do I trust?” Thank you for putting this so succinctly. There are other supply-chain problems in decentralizing control of the modules though: Modules might only have a few people or one person responsible for them. Consequently suspect commits by a malfeasant or compromised (hacked) team member might not be noticed. Maybe this is a variant of the old saw "with enough eyes all bugs are shallow" but my gut feeling is that commits to Python core will get more inspection than those to a small 3rd party module. Today, modules get some review for free, just by being part of stdlib. Even if I know and trust @jdoe I have less assurances that they didn't get phished and their repo tweaked. Also trusted team members/organizations may change over time. The browser extensions world is the poster child for this, where we've seen not only similarly-named malicious extensions posted to stores but also once-legitimate extensions quietly purchased and subsequently subverted by bad actors. This is one example (adware: could've been much worse): https://www.bleepingcomputer.com/news/security/-particle-chrome-extension-sold-to-new-dev-who-immediately-turns-it-into-adware/ https://www.bleepingcomputer.com/news/security/-particle-chr... I like the idea of a review process but I have a hard time imagining a crowd-sourced system that wouldn't get gamed. We have a "dissolution of responsibility" problem: millions of companies rely on these components but have no explicit responsibility of care. Perhaps that needs to change, somehow.
- pas 7y ago
  Companies (and anyone interested) could simply pool funds together and use that to ask already trusted and respected auditors to do reviews. And anyone who pays into the fund should be able to vote on what package to review. (And there should be a weighted lottery, so eventually small contributors' wishes have a chance to get fulfilled.) For good measure anyone can put this on the blockchain, make a flattr/patreon thing out of this. (Somehow use github sponsoring...) Who knows.

tialaramex 7y ago

I suspect you're gravely over-estimating how much value there is in the dustier "batteries included" modules over an out of tree component for security. The problem is that security assumptions change. If you're looking at some code in 1995 that accesses an API, it seems reasonable that it disables validation for the SSL library it brings in to access an HTTPS URL. Handling that properly would have been a huge pain, faster is better. But in 2019 you'd be appalled, how is this thing not depending on certifi and requests? As it is it's wide open to a MitM attack and you'd get blind sided. A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() and in 1970s Unix I'm sure this was invaluable. But obviously today "copying strings into fixed width buffers with no terminator" is essentially never what you wanted, but people assume based on the name that strncpy() will do some other thing they do want, and insert a security defect into their code.

geezerjay 7y ago
> A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() Arguably, what makes C the perfect example is not necessarily its standard library functions per se but the language's official string data type.

ivoflipse 7y ago

Then maybe those large enterprise organizations should start paying for the long mature process with experienced developers

geofft 7y ago

> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. This is an appealing sentiment, but you will get hacked if you rely on it. Python 2's standard library didn't do SSL certificate verification by default until the fairly controversial PEP 476 from late 2014, which changed the default in a point release of Python 2.7. Through 2014 (and for some time later, given how slow it is to get new upstream releases through distros onto someone's computer), the standard thing to do to use HTTPS securely was to use requests, a module developed by someone who got commit privileges taken away on another of his projects for making multiple releases a day. You were more secure with his code than with the standard library. I think you have no more idea who's on the other end of pushes to the standard library than who's on the other end of pushes to PyPI. (Which is to say, in part, that you can equally well have an idea of both of these if you put effort into it. I've met the maintainers of several of my third-party dependencies at PyCon.) Something being in the standard library doesn't mean it has a higher class of developer behind it. There's more overhead, but it doesn't mean there's more maturity; many third-party module developers are more experienced (either in their field, if it's something like crypto, or just in general as responsible developers) than standard library developers. And if anything it means that security updates are slower and rarer because the process is more painful. Consider the argument of the 2009 paper "Security Impact Ratings Considered Harmful" https://arxiv.org/abs/0904.4058 https://arxiv.org/abs/0904.4058 (disclosure: I'm a coauthor). Whether people find a vulnerability important enough to patch has little correlation in practice with how exploitable it is. And the only thing that gets regular attention is the latest development version of the code. If some module was significantly refactored or reimplemented, no upstream developer is looking for bugs in the old version. So, if you want to be safe and you haven't personally both audited and fuzzed the code you're running, you actually want to be running the latest released version of that code, regardless of whether someone stuck a CVSS on the old version yet. I've met very few companies who can upgrade to the latest Python 3.x minor release promptly when it comes out (and several who are still on Python 2!). I've met many companies who can pip install the latest versions of their dependencies without too much trouble, though.

Scarblac 7y ago

> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, Yes, but do you like funding them? This is all freely downloadable open source software.

newen 7y ago
Plenty of people fund the Python Software Foundation. Their revenue is around 3 million dollars a year.