3 ms·

Solved as long as there aren't native dependencies not managed by pip, which there often are.

by foota 9y ago

joshuamorton 9y ago
Wheels solved this problem in 2013. For context, you can install opencv, tensorflow, ROS, matplotlib, and the entire scipy stack in a virtualenv, with no external dependencies, using wheels. This means that you can generate images, train a machine learning algorithm on them, compare the results to conventional CV algorithms, and display them in an ipython notebook all from a venv. There's a huge amount of C++ and even qt integration in that pipeline, all isolated. It won't be maximally performant (ie for massive training), but for that you'd want distributed docker deploys or similar anyway.
- sethammons 9y ago
  Wheels just broke something in our build pipeline. They removed support for Python 2.6 and started tossing errors. I was able to fix it by pinning Wheel which probably should have been done originally by who ever made the build utility, but it would have been a non issue with Go and a binary.
  - joshuamorton 9y ago
    Well, python 2.6 was also EoL'd 4 years ago, so yes, if you're using a no longer supported piece of software and not pinning your versions, I'd argue that you're inviting issues.
    - sethammons 9y ago
      Sure. But if the build utility was just some binary, then it wouldn't matter. If Go was abandoned by all maintainers tomorrow or they broke all the packages, the already built binary will still work. Should someone have changed the Python build tool to be 2.7 or 3? Maybe if they were bored and new it was something that needed work or wanted to be good tech citizen. However, what really happened is that no one even knew what the tool was really doing, just that it was part of a suite of tools in a build process, and no one would have looked twice at it ever again had wheel not removed support for 2.6. /me shrugs.
      - joshuamorton 9y ago
        >But if the build utility was just some binary I mean it totally would if the binary dynamically linked against a file you didn't have on your system, which is exactly what happened with `wheel` (python 2.6 doesn't have OrderedDict, which wheel now uses).
- deathanatos 9y ago
  Wheels doesn't completely solve this problem. For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library. This dependency isn't captured in the wheel metadata, and it's up to you to install the right libblas on the host, or somewhere where it will get loaded. (Though honestly, this isn't usually an unmanageable level of complexity. But virtualenvs are not free from external dependencies always.) IIRC, we also had a problem where a wheel failed to find symbols in a SO it was linked against. It turns out that the wheel worked fine in precise, but failed in trusty, and we ended up having to split our wheel repository because the wheel was specific to the Ubuntu release. (It seemed, at the time, that whatever SO it was linked against appears to have made backwards incompatible changes without changing the major.) There are rare cases like this where the wheel is unique to attributes of the host that the wheel metadata can't capture.
  - viraptor 9y ago
    > For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library And with the wheel packaging, you're free to embed that library in the wheel that depends on it. You can also not do that and rely on the system libraries. The wheel provides you a way to do what you want, but doesn't force you to do it. The are good reasons for either of those approaches, so I'd say wheel does solve the issue.
  - joshuamorton 9y ago
    As another user mentioned, they do, if the library maintainers take the time to do things correctly. The package I'm describing does not link outside of the virtualenv, the matlplotlib, opencv-python, and tensorflow libraries include all necessary dependencies (although you have to use a non-default renderer in matplotlib, because it doesn't bundle all of them). What you say is correct, virtualenvs are not free from external dependencies always, but correctly build wheels are. Wheels and virtualenvs aren't the same thing.
- pwang 9y ago
  Have you actually done this? If so, on what platform(s)?
  - joshuamorton 9y ago
    Linux, If you'd like, I'll post the pipfile for the repository.
- alfla 9y ago
  Could you elaborate on how ROS fits into this framework?
  - joshuamorton 9y ago
    That's a different project, but I also use ros in a virtualenv. Its a bit weird because you end up installing ros both inside and outside of the venv (various ros command line tools only use /opt/ros/... python deps), but your actual nodes run in your virtualenv. And ros doesn't even need to be a wheel fwiw, its pure python, but its also just a painful thing to deal with for many, so it was worth mentioning.
- SmirkingRevenge 9y ago
  You can pip install pyspark now too!

Solved as long as there aren't native dependencies not managed by pip, which there often are.

by foota 9y ago

Solved as long as there aren't native dependencies not managed by pip, which there often are.

Wheels solved this problem in 2013. For context, you can install opencv, tensorflow, ROS, matplotlib, and the entire scipy stack in a virtualenv, with no external dependencies, using wheels. This means that you can generate images, train a machine learning algorithm on them, compare the results to conventional CV algorithms, and display them in an ipython notebook all from a venv. There's a huge amount of C++ and even qt integration in that pipeline, all isolated. It won't be maximally performant (ie for massive training), but for that you'd want distributed docker deploys or similar anyway.

sethammons 9y ago
Wheels just broke something in our build pipeline. They removed support for Python 2.6 and started tossing errors. I was able to fix it by pinning Wheel which probably should have been done originally by who ever made the build utility, but it would have been a non issue with Go and a binary.
- joshuamorton 9y ago
  Well, python 2.6 was also EoL'd 4 years ago, so yes, if you're using a no longer supported piece of software and not pinning your versions, I'd argue that you're inviting issues.
  - sethammons 9y ago
    Sure. But if the build utility was just some binary, then it wouldn't matter. If Go was abandoned by all maintainers tomorrow or they broke all the packages, the already built binary will still work. Should someone have changed the Python build tool to be 2.7 or 3? Maybe if they were bored and new it was something that needed work or wanted to be good tech citizen. However, what really happened is that no one even knew what the tool was really doing, just that it was part of a suite of tools in a build process, and no one would have looked twice at it ever again had wheel not removed support for 2.6. /me shrugs.
    - joshuamorton 9y ago
      >But if the build utility was just some binary I mean it totally would if the binary dynamically linked against a file you didn't have on your system, which is exactly what happened with `wheel` (python 2.6 doesn't have OrderedDict, which wheel now uses).
deathanatos 9y ago
Wheels doesn't completely solve this problem. For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library. This dependency isn't captured in the wheel metadata, and it's up to you to install the right libblas on the host, or somewhere where it will get loaded. (Though honestly, this isn't usually an unmanageable level of complexity. But virtualenvs are not free from external dependencies always.) IIRC, we also had a problem where a wheel failed to find symbols in a SO it was linked against. It turns out that the wheel worked fine in precise, but failed in trusty, and we ended up having to split our wheel repository because the wheel was specific to the Ubuntu release. (It seemed, at the time, that whatever SO it was linked against appears to have made backwards incompatible changes without changing the major.) There are rare cases like this where the wheel is unique to attributes of the host that the wheel metadata can't capture.
- viraptor 9y ago
  > For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library And with the wheel packaging, you're free to embed that library in the wheel that depends on it. You can also not do that and rely on the system libraries. The wheel provides you a way to do what you want, but doesn't force you to do it. The are good reasons for either of those approaches, so I'd say wheel does solve the issue.
- joshuamorton 9y ago
  As another user mentioned, they do, if the library maintainers take the time to do things correctly. The package I'm describing does not link outside of the virtualenv, the matlplotlib, opencv-python, and tensorflow libraries include all necessary dependencies (although you have to use a non-default renderer in matplotlib, because it doesn't bundle all of them). What you say is correct, virtualenvs are not free from external dependencies always, but correctly build wheels are. Wheels and virtualenvs aren't the same thing.
pwang 9y ago
Have you actually done this? If so, on what platform(s)?
- joshuamorton 9y ago
  Linux, If you'd like, I'll post the pipfile for the repository.
alfla 9y ago
Could you elaborate on how ROS fits into this framework?
- joshuamorton 9y ago
  That's a different project, but I also use ros in a virtualenv. Its a bit weird because you end up installing ros both inside and outside of the venv (various ros command line tools only use /opt/ros/... python deps), but your actual nodes run in your virtualenv. And ros doesn't even need to be a wheel fwiw, its pure python, but its also just a painful thing to deal with for many, so it was worth mentioning.
SmirkingRevenge 9y ago
You can pip install pyspark now too!