5 ms·

Trapped in python package; send food.

35 points by blasdel 17y ago

radu_floricica 17y ago
This reminds me of a discussion here a few days ago in which people were happily bashing java's classpath. I don't know anything about python, but my feeling at their comparison of java and python was that something that is way too simple to use can't be at the same time easy to customize. In the light of this article classpaths sound a lot like what they really are: a working compromise everybody can live with.
- jnoller 17y ago
  Not exactly; python already has a PYTHONPATH which acts (sort of) like java's classpath.
moe 17y ago
This is a lot of rambling and a strange approach for what is effectively a 3-liner... ./configure --prefix=/opt/my_python make && make install export PATH="/opt/my_python/bin:$PATH"
- jnoller 17y ago
  No it's not. You missed the fact that python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines. This means you can compile with --prefix all you want, but if you move it, everything breaks in unexpected ways. I agree, all you should need is a --prefix, and that scripts should obey whatever python interpreter is in /usr/bin/env - but that's not the way it works, sadly.
  - moe 17y ago
    python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines. Such scripts are broken. One should fix the scripts instead of mangling the environment. It's a one-liner: find /opt/myproject -type f -exec sed -i 's/#!\/usr\/bin\/python/#!\/usr\/bin\/env python/' {} \; but if you move it, everything breaks in unexpected ways. If you move it then PYTHONPATH is your friend. Once you have a python build for each project (which you want to have anyways) you'll notice that you don't need virtualenv or other kludges anymore either. Yep, it's really that easy. 4 lines of code. How many lines was yours again?
    - jnoller 17y ago
      Did you read the comment above this? Sed is the initial hack to fix it, but it's still a hack; not a fix. Eventually I punted on it for the Linux dist, but I'm probably going to be stuck with the sed fix for OS/X. As for pythonpath - that has nothing to do with moving the entire directory path, including the interpreter, /bin dir, etc.
      - moe 17y ago
        Sorry, I just can't follow where your exact problem is. I have 9 python projects under /opt/foo* right now. Each of them has their own python in /opt/foo*/sys/python. I su to the respective user who has PATH and PYTHONPATH set appropiately in its profile - and everything works as expected. I also don't follow why you consider sed a "hack" here. A hardcoded hashbang line is clearly a Bug - but gladly one that is trivially fixed with that one-liner. On the other side we have your "fix" that involves permanently mangling the environment in strange ways. In my book you are the one proposing the hack here.
        jnoller 17y ago
        I'm not proposing mangling anything here; in fact I dropped the mangled environment (as it was a hack, and added too much space) and swapped to a simpler (and cleaner) virtualenv built in /opt. I still haven't fixed the build something and upload it for people, because of the hardcoded shebang, and yes - sed could fix that.
        moe 17y ago
        Also, you have to run the sed command over the tree every time you install a new package which installs a binary How often do your projects depend on something else, written in python, that installs a binary? For me that number is zero. iirc all of my python dependencies are libraries. And if I stumbled into something that comes with a binary, with a broken hashbang line, then I'd be rather motivated to not only fix it, but to also feed the fix upstream. If the project is useful enough for me to use it then those 10 minutes of my time for a brown-paper-bag micropatch are a no-brainer. Well, to each their own. - But I still think this is a solution looking for a problem, with the potential to create more problems than it solves.
  - jrockway 17y ago
    In Perl, the default build systems change the shebang on installed scripts to point to the perl that "compiled" them. So if you run "/my/weird/perl Makefile.PL && make install", your installed scripts are run by /my/weird/perl. (We also have a generic way to get at installed data files without hard-coded paths; see File::ShareDir.) Not sure why Python can't do this.
Goladus 17y ago
Why can't the hardcoded "#!.../python" lines be fixed with find+sed? I'm not sure I understand exactly what he's trying to do.
- jnoller 17y ago
  I did exactly that at first; the problem is that it shouldn't need to be done, and additionally, other things (such as .pth files) can also include hardcoded paths.
  - Goladus 17y ago
    I don't disagree, I just wouldn't have expected a rant about it and thought I was missing something.
    - jnoller 17y ago
      Yeah, well, it's (mostly) a rant, though I like the kickstart thing I put together. I wouldn't have posted this to hacker news though, fwiw
- nailer 17y ago
  Fixing it is good, but a nicer to start an interpreter, which handles whichever is the preferred location to Python, is to call env, which is always in usr bin. #!/usr/bin/env python Will dynamically start whatever Python is in your $PATH.
  - Goladus 17y ago
    Right, I'm saying that he was faced with a group of files he couldn't tar up, because the #! were all fixed. I'm wondering why he couldn't just replace '#!.*/python' with '#!/usr/bin/env python' for everything in the tree.
    - jnoller 17y ago
      And I can/could, and will probably still end up doing that as much as I might not like it - and any time you install something else which includes a script in a binpath, you need to rerun the sed command.
shadytrees 17y ago
Nit: setuptools does not do this. Proper setuptools packages use the console_scripts entry point, which points to a function will never specify a shebang. A good example is Paste Script [PS]. "paster" is linked the function "paste.script.command:run". When this is pip-installed into a virtualenv, you can `cat` the "paster" executable. It'll look like something like this [pr]; sure enough, the entire thing will have been automatically generated, taking into account the changed Python binary. Slag on setuptools all you want; PEAK got entry points nailed down perfectly. [PS]: http://svn.pythonpaste.org/Paste/Script/trunk/setup.py http://svn.pythonpaste.org/Paste/Script/trunk/setup.py [pr]: http://pastie.org/550423 http://pastie.org/550423 On the other hand, distutils has a `scripts=` keyword that points directly to a file, which then gets copied and pasted into the bin directory. Since the file has be word-for-word the script, the shebang is written directly. It depends on env, which should normally function; but, if it doesn't, as in the OP's case, there's no fallback. A good example of this is Mercurial [hg]. (Note: I actually haven't used raw distutils all that much, so there might be a work-around for this. A few fruitless minutes of Google Code Search hasn't turned up anything interesting.) [hg]: http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg
- jnoller 17y ago
  easy_install, when installed with setuptools, hardcodes the shebang fwiw
- brodie 17y ago
  http://docs.python.org/distutils/setupscript.html#installing-scripts http://docs.python.org/distutils/setupscript.html#installing... "The only clever feature is that if the first line of the script starts with #! and contains the word “python”, the Distutils will adjust the first line to refer to the current interpreter location. By default, it is replaced with the current interpreter location. The --executable (or -e) option will allow the interpreter path to be explicitly overridden."

35 points by blasdel 17y ago

This reminds me of a discussion here a few days ago in which people were happily bashing java's classpath. I don't know anything about python, but my feeling at their comparison of java and python was that something that is way too simple to use can't be at the same time easy to customize. In the light of this article classpaths sound a lot like what they really are: a working compromise everybody can live with.

jnoller 17y ago
Not exactly; python already has a PYTHONPATH which acts (sort of) like java's classpath.

moe 17y ago

This is a lot of rambling and a strange approach for what is effectively a 3-liner... ./configure --prefix=/opt/my_python make && make install export PATH="/opt/my_python/bin:$PATH"

jnoller 17y ago
No it's not. You missed the fact that python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines. This means you can compile with --prefix all you want, but if you move it, everything breaks in unexpected ways. I agree, all you should need is a --prefix, and that scripts should obey whatever python interpreter is in /usr/bin/env - but that's not the way it works, sadly.
- moe 17y ago
  python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines. Such scripts are broken. One should fix the scripts instead of mangling the environment. It's a one-liner: find /opt/myproject -type f -exec sed -i 's/#!\/usr\/bin\/python/#!\/usr\/bin\/env python/' {} \; but if you move it, everything breaks in unexpected ways. If you move it then PYTHONPATH is your friend. Once you have a python build for each project (which you want to have anyways) you'll notice that you don't need virtualenv or other kludges anymore either. Yep, it's really that easy. 4 lines of code. How many lines was yours again?
  - jnoller 17y ago
    Did you read the comment above this? Sed is the initial hack to fix it, but it's still a hack; not a fix. Eventually I punted on it for the Linux dist, but I'm probably going to be stuck with the sed fix for OS/X. As for pythonpath - that has nothing to do with moving the entire directory path, including the interpreter, /bin dir, etc.
    - moe 17y ago
      Sorry, I just can't follow where your exact problem is. I have 9 python projects under /opt/foo* right now. Each of them has their own python in /opt/foo*/sys/python. I su to the respective user who has PATH and PYTHONPATH set appropiately in its profile - and everything works as expected. I also don't follow why you consider sed a "hack" here. A hardcoded hashbang line is clearly a Bug - but gladly one that is trivially fixed with that one-liner. On the other side we have your "fix" that involves permanently mangling the environment in strange ways. In my book you are the one proposing the hack here.
      - jnoller 17y ago
        I'm not proposing mangling anything here; in fact I dropped the mangled environment (as it was a hack, and added too much space) and swapped to a simpler (and cleaner) virtualenv built in /opt. I still haven't fixed the build something and upload it for people, because of the hardcoded shebang, and yes - sed could fix that.
        moe 17y ago
        Also, you have to run the sed command over the tree every time you install a new package which installs a binary How often do your projects depend on something else, written in python, that installs a binary? For me that number is zero. iirc all of my python dependencies are libraries. And if I stumbled into something that comes with a binary, with a broken hashbang line, then I'd be rather motivated to not only fix it, but to also feed the fix upstream. If the project is useful enough for me to use it then those 10 minutes of my time for a brown-paper-bag micropatch are a no-brainer. Well, to each their own. - But I still think this is a solution looking for a problem, with the potential to create more problems than it solves.
- jrockway 17y ago
  In Perl, the default build systems change the shebang on installed scripts to point to the perl that "compiled" them. So if you run "/my/weird/perl Makefile.PL && make install", your installed scripts are run by /my/weird/perl. (We also have a generic way to get at installed data files without hard-coded paths; see File::ShareDir.) Not sure why Python can't do this.

Goladus 17y ago

Why can't the hardcoded "#!.../python" lines be fixed with find+sed? I'm not sure I understand exactly what he's trying to do.

jnoller 17y ago
I did exactly that at first; the problem is that it shouldn't need to be done, and additionally, other things (such as .pth files) can also include hardcoded paths.
- Goladus 17y ago
  I don't disagree, I just wouldn't have expected a rant about it and thought I was missing something.
  - jnoller 17y ago
    Yeah, well, it's (mostly) a rant, though I like the kickstart thing I put together. I wouldn't have posted this to hacker news though, fwiw
nailer 17y ago
Fixing it is good, but a nicer to start an interpreter, which handles whichever is the preferred location to Python, is to call env, which is always in usr bin. #!/usr/bin/env python Will dynamically start whatever Python is in your $PATH.
- Goladus 17y ago
  Right, I'm saying that he was faced with a group of files he couldn't tar up, because the #! were all fixed. I'm wondering why he couldn't just replace '#!.*/python' with '#!/usr/bin/env python' for everything in the tree.
  - jnoller 17y ago
    And I can/could, and will probably still end up doing that as much as I might not like it - and any time you install something else which includes a script in a binpath, you need to rerun the sed command.

shadytrees 17y ago

Nit: setuptools does not do this. Proper setuptools packages use the console_scripts entry point, which points to a function will never specify a shebang. A good example is Paste Script [PS]. "paster" is linked the function "paste.script.command:run". When this is pip-installed into a virtualenv, you can `cat` the "paster" executable. It'll look like something like this [pr]; sure enough, the entire thing will have been automatically generated, taking into account the changed Python binary. Slag on setuptools all you want; PEAK got entry points nailed down perfectly. [PS]: http://svn.pythonpaste.org/Paste/Script/trunk/setup.py http://svn.pythonpaste.org/Paste/Script/trunk/setup.py [pr]: http://pastie.org/550423 http://pastie.org/550423 On the other hand, distutils has a `scripts=` keyword that points directly to a file, which then gets copied and pasted into the bin directory. Since the file has be word-for-word the script, the shebang is written directly. It depends on env, which should normally function; but, if it doesn't, as in the OP's case, there's no fallback. A good example of this is Mercurial [hg]. (Note: I actually haven't used raw distutils all that much, so there might be a work-around for this. A few fruitless minutes of Google Code Search hasn't turned up anything interesting.) [hg]: http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg

jnoller 17y ago
easy_install, when installed with setuptools, hardcodes the shebang fwiw
brodie 17y ago
http://docs.python.org/distutils/setupscript.html#installing-scripts http://docs.python.org/distutils/setupscript.html#installing... "The only clever feature is that if the first line of the script starts with #! and contains the word “python”, the Distutils will adjust the first line to refer to the current interpreter location. By default, it is replaced with the current interpreter location. The --executable (or -e) option will allow the interpreter path to be explicitly overridden."