5 ms·

> PyPI is useless Why will it be "useless". Explain your reasoning please.

pletnes 4y ago
Most of the libraries I use include compiled C/C++/Fortran/Rust code. Pandas, scipy, scikit-learn, … if I were limited to pure-python libraries, I would probably rather swap languages, or at least package manager, at great inconvenience. That being said, I don’t think PyPI would be «useless» - this was the state a few years ago, and we had to compile all the libraries ourselves. I don’t want to go back.
- nonrandomstring 4y ago
  > great inconvenience It's a convenience/security trade-off, I see. The only solution I've ever seen to that requires investing trust in an "authority" which then becomes corrupt and censorial. One simply expands the dilemma to a triad; security/freedom/convenience. If I am not mistaken the PyPI "Cheese Shop" is owned by the Python Software Foundation, a 501(c3) nonprofit organisation which constitutionally values Software Freedom highly. It seems natural that convenience would be sacrificed if security is of concern.
  - nine_k 4y ago
    Such an authority in the Linux world used to be a distribution. Installing a binary blob provided by Debian build servers is based on decades of trust. But there is a tradeoff between having things thoroughly vetted and tested, and moving fast.
    - nonrandomstring 4y ago
      Interesting point. So as dimensions we now have - security - freedom - convenience - speed/newness Who can build me a UI with four sliders that selects the packages I can install? Bonus: when I move a slider it highlights all the potential packages that changed status with reasons why they are now included/excluded.
      - dylan604 4y ago
        You're an HN reader, so you should be able to knock this out over a weekend /s
        nonrandomstring 4y ago
        You're right the prototype GUI is a weekend of work. But you also know that's not where the work is :) Now some more intelligent comments are coming in we can talk about the analysis and tagging of thousands of packages, dealing with backward compatibility and what happens when naughty malware just hops to another level of trust. But none of that is a call to give up. We just need to think seriously about the problem we face.
- zzzeek 4y ago
  None of those packages are downloading and running CRAP.EXE within the setup.py process, that's not how native extensions work. It should be possible to flag packages that are downloading things when setup.py runs, much less running exec within setup.py. a python package that really needs you to run a windows installer for its dependencies should have you be doing that separately.
  - pletnes 4y ago
    Sure, I didn’t intend to claim that. It’s just a hassle for me to compile my own C code, which I’d have to do if binaries weren’t bundled. That’s why anaconda python took off on windows - it’s hard work to compile scipy on windows!
    - zzzeek 4y ago
      pypi delivers wheel files for pre-built binaries, and that's the only way one is supposed to distribute pre-built binary executables or shared libraries. the issue of "runs malicious code in setup.py" does not apply in that case because setup.py isn't invoked.
  - BiteCode_dev 4y ago
    Yes but the problem here is the obfuscation of the malware code loading. No need to trigger it in the setup.py process, as long as you have it in the lib, you can always put a call in a .pth somewhere and run your malware as soon as any python is executed.
    - zzzeek 4y ago
      it should be possible to test packages for that also. if you are testing setup.py to see that no network access or exec occurs, you could similarly run the python interpreter after install and ensure no network / exec() happens at that point either, assuming one has not imported the package. or just disallow unfamiliar .pth files from being installed altogether (outside of those generated by setuptools / etc. for normal execution).
- harlanji 4y ago
  Windows S Mode has PyPI restricted to pure Python due to Device Guard. I'm happy to leave it on ($250 laptop). Indeed, Numpy has been a recurring blocker, maybe 3 times now. But with general peace of mind is the only way I've known Python/PyPI, so I'm pretty happy with it. I have a few RasPis that I can use as auxiliary devices as well, which I think is a pretty cool tradeoff, hardware sandbox--not gone there yet, beyond just configuring SSH/xRDP so I'm ready if the day comes. But I've made a ton of web apps and tools anyway, including a little process launcher that plays the role of poor man's Docker. It'd be nice if those popular systems had a pure Python capability anyway, similar analogy being software rendered 3D back in the day.

Most of the libraries I use include compiled C/C++/Fortran/Rust code. Pandas, scipy, scikit-learn, … if I were limited to pure-python libraries, I would probably rather swap languages, or at least package manager, at great inconvenience. That being said, I don’t think PyPI would be «useless» - this was the state a few years ago, and we had to compile all the libraries ourselves. I don’t want to go back.

nonrandomstring 4y ago
> great inconvenience It's a convenience/security trade-off, I see. The only solution I've ever seen to that requires investing trust in an "authority" which then becomes corrupt and censorial. One simply expands the dilemma to a triad; security/freedom/convenience. If I am not mistaken the PyPI "Cheese Shop" is owned by the Python Software Foundation, a 501(c3) nonprofit organisation which constitutionally values Software Freedom highly. It seems natural that convenience would be sacrificed if security is of concern.
- nine_k 4y ago
  Such an authority in the Linux world used to be a distribution. Installing a binary blob provided by Debian build servers is based on decades of trust. But there is a tradeoff between having things thoroughly vetted and tested, and moving fast.
  - nonrandomstring 4y ago
    Interesting point. So as dimensions we now have - security - freedom - convenience - speed/newness Who can build me a UI with four sliders that selects the packages I can install? Bonus: when I move a slider it highlights all the potential packages that changed status with reasons why they are now included/excluded.
    - dylan604 4y ago
      You're an HN reader, so you should be able to knock this out over a weekend /s
      - nonrandomstring 4y ago
        You're right the prototype GUI is a weekend of work. But you also know that's not where the work is :) Now some more intelligent comments are coming in we can talk about the analysis and tagging of thousands of packages, dealing with backward compatibility and what happens when naughty malware just hops to another level of trust. But none of that is a call to give up. We just need to think seriously about the problem we face.
zzzeek 4y ago
None of those packages are downloading and running CRAP.EXE within the setup.py process, that's not how native extensions work. It should be possible to flag packages that are downloading things when setup.py runs, much less running exec within setup.py. a python package that really needs you to run a windows installer for its dependencies should have you be doing that separately.
- pletnes 4y ago
  Sure, I didn’t intend to claim that. It’s just a hassle for me to compile my own C code, which I’d have to do if binaries weren’t bundled. That’s why anaconda python took off on windows - it’s hard work to compile scipy on windows!
  - zzzeek 4y ago
    pypi delivers wheel files for pre-built binaries, and that's the only way one is supposed to distribute pre-built binary executables or shared libraries. the issue of "runs malicious code in setup.py" does not apply in that case because setup.py isn't invoked.
- BiteCode_dev 4y ago
  Yes but the problem here is the obfuscation of the malware code loading. No need to trigger it in the setup.py process, as long as you have it in the lib, you can always put a call in a .pth somewhere and run your malware as soon as any python is executed.
  - zzzeek 4y ago
    it should be possible to test packages for that also. if you are testing setup.py to see that no network access or exec occurs, you could similarly run the python interpreter after install and ensure no network / exec() happens at that point either, assuming one has not imported the package. or just disallow unfamiliar .pth files from being installed altogether (outside of those generated by setuptools / etc. for normal execution).
harlanji 4y ago
Windows S Mode has PyPI restricted to pure Python due to Device Guard. I'm happy to leave it on ($250 laptop). Indeed, Numpy has been a recurring blocker, maybe 3 times now. But with general peace of mind is the only way I've known Python/PyPI, so I'm pretty happy with it. I have a few RasPis that I can use as auxiliary devices as well, which I think is a pretty cool tradeoff, hardware sandbox--not gone there yet, beyond just configuring SSH/xRDP so I'm ready if the day comes. But I've made a ton of web apps and tools anyway, including a little process launcher that plays the role of poor man's Docker. It'd be nice if those popular systems had a pure Python capability anyway, similar analogy being software rendered 3D back in the day.