5 ms·

http://pythonwheels.com/ http://pythonwheels.com/ solves the problem of building c extensions on installation.

by kbar13 11y ago

emidln 11y ago
Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI?
- toomuchtodo 11y ago
  > Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI? Yes. I've seen them, and they've been huge shops.
- vamega 11y ago
  Can you point me to your recommended PyPI-in-a-box system?
  - emidln 11y ago
    I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.
    - vamega 11y ago
      Do I understand this correctly? It only mirrors the packages that are requested from it? So I won't need to download 100GB+ of packages that I am not interested in?
      - emidln 11y ago
        Correct. My team has two main use cases: private packages and guaranteed access to packages we've built with. It's extremely frustrating to come into a codebase after several months or years to find it using a library that no longer seems to exist on the public internet.
    - doki_pen 11y ago
      We migrated off of localshop and onto devpi. Devpi is a much better product and much more actively maintained. localshop was nothing but headaches and constantly breaking.
      - emidln 11y ago
        Maybe you use more esoteric features. The only thing I've done in the last 18 months is patch a bug that prevented uploads from Python 2.7.4-2.7.10. We just run it under circus with chaussette and front it with an elb.
      - kbar13 11y ago
        localshop has significantly improved in the past few months. Before I could barely consider it production-ready.
      - mvantellingen 11y ago
        Author here: I created it to solve an issue I was running into a couple of years ago. I've only recently started using it again myself. I think the development version (not on pypi) is in much better shape with things like multiple repositories and better user management (teams).
  - kbrownlees 11y ago
    I have had good success with http://doc.devpi.net/latest/ http://doc.devpi.net/latest/
    - vamega 11y ago
      That looks good. Might work exceptionally well for my needs. I want to have a PyPI mirror in an environment that doesn't have public internet access. Running devpi in another environment and syncing the resulting repository should allow me to achieve what I want.
  - ronjouch 11y ago
    As a first step, what about trying pip's `-f` option, combined with dumping your wheels in a dumb folder served by apache/nginx: pip --help [...] Install Options: [...] -f, --find-links <url> If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory,then look for archives in the directory listing. EDIT: in the context of deploying some app at work, what's the interest of a full-blown hosted cheeseshop? Users of these solutions, what value does it add to a simple `pip install -f INTERNAL_PKG_URL pkg_a==1.2.3 pkg_b==0.1.2` ? Which features do you frequently use?
- viraptor 11y ago
  > you get to skip the mess of building system packages. Only in cases where you don't have wheels depending on external libraries. If you do, you should still package with the right dependency constraints. Otherwise you can install a wheel which does not work (because of missing .so)
  - emidln 11y ago
    This simply isn't true. You can package .so or .dll (or anything else) with your wheel. An example of doing it with a dll or so is here: http://stackoverflow.com/questions/24071491/how-can-i-make-a-python-wheel-from-an-existing-native-library http://stackoverflow.com/questions/24071491/how-can-i-make-a...
    - viraptor 11y ago
      What you said does not disagree with me. You can have wheels depending on external dynamic libs. Or you can package them together. Both ways have good and bad sides, but if you leave them external, it's very useful to have the right dependencies.

http://pythonwheels.com/ http://pythonwheels.com/ solves the problem of building c extensions on installation.

by kbar13 11y ago

http://pythonwheels.com/ http://pythonwheels.com/ solves the problem of building c extensions on installation.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI?

toomuchtodo 11y ago
> Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems or pay for a hosted PyPI? Yes. I've seen them, and they've been huge shops.
vamega 11y ago
Can you point me to your recommended PyPI-in-a-box system?
- emidln 11y ago
  I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.
  - vamega 11y ago
    Do I understand this correctly? It only mirrors the packages that are requested from it? So I won't need to download 100GB+ of packages that I am not interested in?
    - emidln 11y ago
      Correct. My team has two main use cases: private packages and guaranteed access to packages we've built with. It's extremely frustrating to come into a codebase after several months or years to find it using a library that no longer seems to exist on the public internet.
  - doki_pen 11y ago
    We migrated off of localshop and onto devpi. Devpi is a much better product and much more actively maintained. localshop was nothing but headaches and constantly breaking.
    - emidln 11y ago
      Maybe you use more esoteric features. The only thing I've done in the last 18 months is patch a bug that prevented uploads from Python 2.7.4-2.7.10. We just run it under circus with chaussette and front it with an elb.
    - kbar13 11y ago
      localshop has significantly improved in the past few months. Before I could barely consider it production-ready.
    - mvantellingen 11y ago
      Author here: I created it to solve an issue I was running into a couple of years ago. I've only recently started using it again myself. I think the development version (not on pypi) is in much better shape with things like multiple repositories and better user management (teams).
- kbrownlees 11y ago
  I have had good success with http://doc.devpi.net/latest/ http://doc.devpi.net/latest/
  - vamega 11y ago
    That looks good. Might work exceptionally well for my needs. I want to have a PyPI mirror in an environment that doesn't have public internet access. Running devpi in another environment and syncing the resulting repository should allow me to achieve what I want.
- ronjouch 11y ago
  As a first step, what about trying pip's `-f` option, combined with dumping your wheels in a dumb folder served by apache/nginx: pip --help [...] Install Options: [...] -f, --find-links <url> If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory,then look for archives in the directory listing. EDIT: in the context of deploying some app at work, what's the interest of a full-blown hosted cheeseshop? Users of these solutions, what value does it add to a simple `pip install -f INTERNAL_PKG_URL pkg_a==1.2.3 pkg_b==0.1.2` ? Which features do you frequently use?
viraptor 11y ago
> you get to skip the mess of building system packages. Only in cases where you don't have wheels depending on external libraries. If you do, you should still package with the right dependency constraints. Otherwise you can install a wheel which does not work (because of missing .so)
- emidln 11y ago
  This simply isn't true. You can package .so or .dll (or anything else) with your wheel. An example of doing it with a dll or so is here: http://stackoverflow.com/questions/24071491/how-can-i-make-a-python-wheel-from-an-existing-native-library http://stackoverflow.com/questions/24071491/how-can-i-make-a...
  - viraptor 11y ago
    What you said does not disagree with me. You can have wheels depending on external dynamic libs. Or you can package them together. Both ways have good and bad sides, but if you leave them external, it's very useful to have the right dependencies.