5 ms·

I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human b

I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as package maintainers. When you make the next big thing, consider this approach.

drdaeman 9y ago
Maybe you're right, but I see one possible downside that is quite important. I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice. With the intermediate maintainers, that's going to get worse. I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.
- nyrikki 9y ago
  PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression. I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.
  - drdaeman 9y ago
    1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis. 2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH. And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data). --- I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything. To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys. (Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)
    - nyrikki 9y ago
      On point 1) yes so they would have to compile, that is not a huge barrier and it would also result in improved quality of modules just as CI/CD does. On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7) # capsh --print -- -c 'pip list > /dev/null' Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.
zzzcpan 9y ago
I don't think there are any maintainers that verify upstream code, they only manage packages and updates. Which is actually safer to do without maintainers, completely automatically, as it will eliminate a huge attack surface introduced by a maintainer.
- ex_amazon_sde 9y ago
  Debian Developers are responsible of the package quality. Most package only well-known software that they are familiar with or read the code. Some do more thorough security audits. Some security-sensitive packages are maintained by teams to share the workload.
  - zzzcpan 9y ago
    They do a lot that could be and sometimes even is automated, but that doesn't help with security, only weakens it. Which is my point. It's better to automate package generation and trust fewer people, mainly the authors, not introduce maintainers into it.
    - peterwwillis 9y ago
      Authors don't generate distro packages because there's too many distros and each distro needs to make changes that have nothing to do with code. Maintainers are necessary and a package signed by the author is a non-starter. However, source code can be signed and then used to make a package signed by a distro.
      - zzzcpan 9y ago
        Think of it instead of allowing access to the system to maintain packages, we allow people to submit code that generates packages.
        peterwwillis 9y ago
        Of course you can do this - all packages are basically just wrappers around upstream code. But you still need someone to maintain the wrapper, and they have to check every new code release to see if there's something in the wrapper that has to change. And there are multiple distros. There's no getting away from maintainers with traditional linux distros. Code package management is different. The author writes their software specifically to conform to the one code package management system. There's no wrapper glue needed, so you don't need a maintainer. Just release your new code and it fits into the system, and other code/tools/etc can just pick it up and use it. This works if you constantly update all the software you use everywhere, and is pretty much guaranteed to become a nightmare if you don't. CPAN is probably the most mature software package management system in existence and it's still a nightmare if you don't keep a private repo and tightly manage releases, and you absolutely need a maintainer.

drdaeman 9y ago

Maybe you're right, but I see one possible downside that is quite important. I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice. With the intermediate maintainers, that's going to get worse. I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.

nyrikki 9y ago
PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression. I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.
- drdaeman 9y ago
  1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis. 2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH. And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data). --- I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything. To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys. (Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)
  - nyrikki 9y ago
    On point 1) yes so they would have to compile, that is not a huge barrier and it would also result in improved quality of modules just as CI/CD does. On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7) # capsh --print -- -c 'pip list > /dev/null' Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.

zzzcpan 9y ago

I don't think there are any maintainers that verify upstream code, they only manage packages and updates. Which is actually safer to do without maintainers, completely automatically, as it will eliminate a huge attack surface introduced by a maintainer.

ex_amazon_sde 9y ago
Debian Developers are responsible of the package quality. Most package only well-known software that they are familiar with or read the code. Some do more thorough security audits. Some security-sensitive packages are maintained by teams to share the workload.
- zzzcpan 9y ago
  They do a lot that could be and sometimes even is automated, but that doesn't help with security, only weakens it. Which is my point. It's better to automate package generation and trust fewer people, mainly the authors, not introduce maintainers into it.
  - peterwwillis 9y ago
    Authors don't generate distro packages because there's too many distros and each distro needs to make changes that have nothing to do with code. Maintainers are necessary and a package signed by the author is a non-starter. However, source code can be signed and then used to make a package signed by a distro.
    - zzzcpan 9y ago
      Think of it instead of allowing access to the system to maintain packages, we allow people to submit code that generates packages.
      - peterwwillis 9y ago
        Of course you can do this - all packages are basically just wrappers around upstream code. But you still need someone to maintain the wrapper, and they have to check every new code release to see if there's something in the wrapper that has to change. And there are multiple distros. There's no getting away from maintainers with traditional linux distros. Code package management is different. The author writes their software specifically to conform to the one code package management system. There's no wrapper glue needed, so you don't need a maintainer. Just release your new code and it fits into the system, and other code/tools/etc can just pick it up and use it. This works if you constantly update all the software you use everywhere, and is pretty much guaranteed to become a nightmare if you don't. CPAN is probably the most mature software package management system in existence and it's still a nightmare if you don't keep a private repo and tightly manage releases, and you absolutely need a maintainer.