5 ms·

I do full time scientific python, and this post has some painfully wrong ideas. >The official oletools install instructions talk about using either pip or setu

I do full time scientific python, and this post has some painfully wrong ideas. >The official oletools install instructions talk about using either pip or setup.py. As a general rule, we're very strongly against installing anything system-wide except through Ubuntu's own package management system, and the environment our Python program runs in doesn't really have a home directory to use pip's --user option, so the obvious and simple pip invocations are out. No! The obvious and simple pip invocation is as follows: python3 -m venv ${HOME}/myvenv ${HOME}/myvenv/bin/pip install foo bar baz oletools ${HOME}/myvenv/bin/pin install -e /path/to/my/installable/project and then edit your code in `/path/to/my/installable/project/project-name/*.py` to your heart's content and tell your IDE or editor that `${HOME}/myvenv/bin/python` is the python to use. Activation is strictly optional, but if you like that, `source ${HOME}/bin/activate` (or use direnv to do it for you). I would also recommend export PIP_REQUIRE_VIRTUALENV=1 I divorce what $USER thinks `python` means from the system python as soon as I can with `pyenv global 3.8.5`: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv Of course never, ever `sudo pip ... `. And don't touch $PYTHONPATH unless someone has forced you to use Caffe, who should be shamed on a regular basis for being Bad At Packages.

nurettin 6y ago
The first thing I do when deploying to an existing machine/hard metal: I Grab the pyenv installer, set it up. Then I build python on target machine. MAKE_OPTS="-j8" CONFIGURE_OPTS="--enable-shared --enable-optimizations --with-computed-gotos" CFLAGS="-march=native -O2 -pipe" pyenv install -f 3.8.5 This takes a while but end result is a well enough optimized python. I also replace standard asyncio loop with uvloop to speed things up a little for async tasks.
- hprotagonist 6y ago
  I don't usually go beyond --enable-shared, but yeah, me too.
bap 6y ago
Would love to hear your thoughts on pipenv as a real-world user of python environments. I leave it on the table that your current solution just works and you have never considered pipenv. ( I am not associated with pipenv. I've seemingly organically gravitated to it lately. )
- hprotagonist 6y ago
  my hot take: poetry won, and PEP517/518 build systems are going to be the way forward. Poetry might not last forever, but I think`pyproject.toml` has legs. Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years. https://github.com/python-poetry/poetry https://github.com/python-poetry/poetry
  - bap 6y ago
    Thanks!
- nomel 6y ago
  Last I checked, pipenv doesn't work without an internet connection. That's a requirement, and design mentality, that doesn't work for me. Won't/can't fix issues: https://github.com/pypa/pipenv/issues/1783 https://github.com/pypa/pipenv/issues/1783 https://github.com/pypa/pipenv/issues/408 https://github.com/pypa/pipenv/issues/408
  - bap 6y ago
    Valid point, I hadn’t considered it. Thank you!
hal9000-tng 6y ago
> No! The obvious and simple pip invocation is as follows: > [complex commands to set up a venv follows] "All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!" Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001: https://www.youtube.com/watch?v=Xtah_05BOe8 https://www.youtube.com/watch?v=Xtah_05BOe8
- hprotagonist 6y ago
  I generalized a bit for completeness and to emphasize that contrary to the article's point, you can do all of this in $HOME and never install anything outside your home directory, which was a concern of the author as a usage policy for their server. python -m venv myvenv myvenv/bin/pip install ... works just as well, i just needed a concrete path to talk about in the rest of the text. Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.
  - mumblerino 6y ago
    After we come up with that “one line”, we tend to forget how long it took us to get there. For newbies, writing one line is impossible.
  - pdonis 6y ago
    > you can do all of this in $HOME Which the author explicitly said does not exist for the use case he is discussing: "the environment our Python program runs in doesn't really have a home directory"
    - hprotagonist 6y ago
      so put it wherever! it's just a directory, put it where you have write access.
      - pdonis 6y ago
        > so put it wherever! it's just a directory You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article. What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.
inshadows 6y ago
That's what cool kids use today; just dump complete dependency tree of every project into random location on filesystem. That's not how Debian OS works though, and that's what author, a sysadmin, talks about very clearly. So I don't get why you propose exactly what he dismissed in the 2nd paragraph. > and the environment our Python program runs in doesn't really have a home directory to use pip's --user option
- hprotagonist 6y ago
  I do all of this on debian, literally every day. from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!
  - inshadows 6y ago
    Really? I do `apt-get install foo` and later I in a Python script I write `import foo`. This Python script can be put ANYWHERE on the system. I don't need to mess up with virtualenvs and I have one version across all scripts on the system that's regularly updated on system upgrades (after being tested by community in bleeding edge versions of the OS). To each his own I guess.
    - hprotagonist 6y ago
      apt doesn't have my stack or its dependencies at proper versions, and cannot in general meet the version-locking of dependencies i and my dependencies require. And --system-site-packages is a baaaaad idea.
pdonis 6y ago
> I do full time scientific python Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him. Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)
- pseudalopex 6y ago
  Virtual environments and installing to $HOME are different things. He says using virtual environments would be a lot more to learn but he just reinvented them.
  - pdonis 6y ago
    > Virtual environments and installing to $HOME are different things. I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option. > He says using virtual environments would be a lot more to learn but he just reinvented them. He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.

nurettin 6y ago

The first thing I do when deploying to an existing machine/hard metal: I Grab the pyenv installer, set it up. Then I build python on target machine. MAKE_OPTS="-j8" CONFIGURE_OPTS="--enable-shared --enable-optimizations --with-computed-gotos" CFLAGS="-march=native -O2 -pipe" pyenv install -f 3.8.5 This takes a while but end result is a well enough optimized python. I also replace standard asyncio loop with uvloop to speed things up a little for async tasks.

hprotagonist 6y ago
I don't usually go beyond --enable-shared, but yeah, me too.

bap 6y ago

Would love to hear your thoughts on pipenv as a real-world user of python environments. I leave it on the table that your current solution just works and you have never considered pipenv. ( I am not associated with pipenv. I've seemingly organically gravitated to it lately. )

hprotagonist 6y ago
my hot take: poetry won, and PEP517/518 build systems are going to be the way forward. Poetry might not last forever, but I think`pyproject.toml` has legs. Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years. https://github.com/python-poetry/poetry https://github.com/python-poetry/poetry
- bap 6y ago
  Thanks!
nomel 6y ago
Last I checked, pipenv doesn't work without an internet connection. That's a requirement, and design mentality, that doesn't work for me. Won't/can't fix issues: https://github.com/pypa/pipenv/issues/1783 https://github.com/pypa/pipenv/issues/1783 https://github.com/pypa/pipenv/issues/408 https://github.com/pypa/pipenv/issues/408
- bap 6y ago
  Valid point, I hadn’t considered it. Thank you!

hal9000-tng 6y ago

> No! The obvious and simple pip invocation is as follows: > [complex commands to set up a venv follows] "All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!" Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001: https://www.youtube.com/watch?v=Xtah_05BOe8 https://www.youtube.com/watch?v=Xtah_05BOe8

hprotagonist 6y ago
I generalized a bit for completeness and to emphasize that contrary to the article's point, you can do all of this in $HOME and never install anything outside your home directory, which was a concern of the author as a usage policy for their server. python -m venv myvenv myvenv/bin/pip install ... works just as well, i just needed a concrete path to talk about in the rest of the text. Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.
- mumblerino 6y ago
  After we come up with that “one line”, we tend to forget how long it took us to get there. For newbies, writing one line is impossible.
- pdonis 6y ago
  > you can do all of this in $HOME Which the author explicitly said does not exist for the use case he is discussing: "the environment our Python program runs in doesn't really have a home directory"
  - hprotagonist 6y ago
    so put it wherever! it's just a directory, put it where you have write access.
    - pdonis 6y ago
      > so put it wherever! it's just a directory You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article. What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.

inshadows 6y ago

That's what cool kids use today; just dump complete dependency tree of every project into random location on filesystem. That's not how Debian OS works though, and that's what author, a sysadmin, talks about very clearly. So I don't get why you propose exactly what he dismissed in the 2nd paragraph. > and the environment our Python program runs in doesn't really have a home directory to use pip's --user option

hprotagonist 6y ago
I do all of this on debian, literally every day. from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!
- inshadows 6y ago
  Really? I do `apt-get install foo` and later I in a Python script I write `import foo`. This Python script can be put ANYWHERE on the system. I don't need to mess up with virtualenvs and I have one version across all scripts on the system that's regularly updated on system upgrades (after being tested by community in bleeding edge versions of the OS). To each his own I guess.
  - hprotagonist 6y ago
    apt doesn't have my stack or its dependencies at proper versions, and cannot in general meet the version-locking of dependencies i and my dependencies require. And --system-site-packages is a baaaaad idea.

pdonis 6y ago

> I do full time scientific python Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him. Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)

pseudalopex 6y ago
Virtual environments and installing to $HOME are different things. He says using virtual environments would be a lot more to learn but he just reinvented them.
- pdonis 6y ago
  > Virtual environments and installing to $HOME are different things. I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option. > He says using virtual environments would be a lot more to learn but he just reinvented them. He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.