4 ms·

Weird. The recommendation for a shebang line is to use #!/usr/bin/python That will definitely break stuff. Why not #!/usr/bin/env python?

by yuvadam 15y ago

nikcub 15y ago
env is nothing more than running it in a new shell, so slightly slower for each exec (since it starts a shell and then searches through PATH), may pick the wrong interpreter (because PATH is dependent on the user and a lot of other things outside of your control) and since this is all running on Google infrastructure they know what /usr/bin/python. their goal isn't to write portable code, it is to write fast code that runs on google servers
- etanol 15y ago
  Sorry, but I fail to see where the "env" command spawns a shell: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/env.c?id=v8.13#n153 http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/env.... Could you please explain what am I missing?
  - nikcub 15y ago
    what I mean is that it is just like running 'python' in a shell, not that each instance spawns a new one
    - zwp 15y ago
      > like One significant difference: when running python from a shell there's a fork() and an exec(). env(1) doesn't fork: there are not two processes. (In other words: the shell does not exit when you run a command, env vanishes). There is obviously still some overhead to using env (and I just learnt from the source that env has argument processing). I tried to replicate AncientPC's test but on my machine both invocations take around 0.015s. (Perhaps their username is an indication as to why they see a (2%!) difference...). But okay. What I really came here to say is: security. You can make all the efforts in the world to ensure that all programs get called with full pathnames but then one env shebang and you're suddenly open to running whatever's first in the user's $PATH and happens to call itself "python". EDIT: eg http://portaudit.freebsd.org/d42e5b66-6ea0-11df-9c8d-00e0815b8da8.html http://portaudit.freebsd.org/d42e5b66-6ea0-11df-9c8d-00e0815...
      - nikcub 15y ago
        That last point is important which I forgot to mention, especially in operating systems that have '.' first in PATH. I notice the difference anecdotally without measuring it, but I don't know if that is a conception bias because I know env should be slower. The best way would be an autoconf script in your package and an install run that finds and verifies the local framework. I have to admit that I have never done this though. I have a few Python scripts with decent distribution and just rely on the direct path (and a batch file for win32)
      - AncientPC 15y ago
        I've posted my full test scripts and results here: http://williamting.com/2012/04/19/performance-decrease-using-env/ http://williamting.com/2012/04/19/performance-decrease-using... I've been using this handle since '93 out of inertia. My tests were done on an idle i7-620M, sequentially. If your invocation takes about 0.015s, that means you're not looping enough for differences to appear. A 2% increase from 0.015s is 0.0153s, invisible due to significant digits cut off.
- AncientPC 15y ago
  I wrote a few small scripts to test this out. Each contained a simple print statement, then looped it 5000 times and timed it. There is ~2.1% overhead when using env. This doesn't matter in most cases, but in a Google-sized company with standardized environments it's worth it.
nodata 15y ago
Maybe because it's a hack because to get around the following not working? #!python
- emmelaich 15y ago
  Yes; #!python should be made to work. Not to use the user's PATH, but some system ordained python (via some kernel work). env is a hack.
  - rbanffy 15y ago
    > via some kernel work Kernels shouldn't be aware of the search path shells use.
    - nodata 15y ago
      He may be referring to binfmt.
      - emmelaich 15y ago
        I wasn't referring to implementation really, but the principle. The main python could be in /bin or /usr/bin or possibly /usr/local/bin. The poor user or developer shouldn't be required to anticipate where and should not need to resort to hacks such as env.
    - emmelaich 15y ago
      I agree, as evidenced by the bit you didn't quote. (?!)
- rbanffy 15y ago
  Bad idea. Which python (or program named "python") will run your script?
  - deleted 15y ago
    [deleted]

Weird. The recommendation for a shebang line is to use #!/usr/bin/python That will definitely break stuff. Why not #!/usr/bin/env python?

by yuvadam 15y ago

Weird. The recommendation for a shebang line is to use #!/usr/bin/python That will definitely break stuff. Why not #!/usr/bin/env python?

nikcub 15y ago

env is nothing more than running it in a new shell, so slightly slower for each exec (since it starts a shell and then searches through PATH), may pick the wrong interpreter (because PATH is dependent on the user and a lot of other things outside of your control) and since this is all running on Google infrastructure they know what /usr/bin/python. their goal isn't to write portable code, it is to write fast code that runs on google servers

etanol 15y ago
Sorry, but I fail to see where the "env" command spawns a shell: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/env.c?id=v8.13#n153 http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/env.... Could you please explain what am I missing?
- nikcub 15y ago
  what I mean is that it is just like running 'python' in a shell, not that each instance spawns a new one
  - zwp 15y ago
    > like One significant difference: when running python from a shell there's a fork() and an exec(). env(1) doesn't fork: there are not two processes. (In other words: the shell does not exit when you run a command, env vanishes). There is obviously still some overhead to using env (and I just learnt from the source that env has argument processing). I tried to replicate AncientPC's test but on my machine both invocations take around 0.015s. (Perhaps their username is an indication as to why they see a (2%!) difference...). But okay. What I really came here to say is: security. You can make all the efforts in the world to ensure that all programs get called with full pathnames but then one env shebang and you're suddenly open to running whatever's first in the user's $PATH and happens to call itself "python". EDIT: eg http://portaudit.freebsd.org/d42e5b66-6ea0-11df-9c8d-00e0815b8da8.html http://portaudit.freebsd.org/d42e5b66-6ea0-11df-9c8d-00e0815...
    - nikcub 15y ago
      That last point is important which I forgot to mention, especially in operating systems that have '.' first in PATH. I notice the difference anecdotally without measuring it, but I don't know if that is a conception bias because I know env should be slower. The best way would be an autoconf script in your package and an install run that finds and verifies the local framework. I have to admit that I have never done this though. I have a few Python scripts with decent distribution and just rely on the direct path (and a batch file for win32)
    - AncientPC 15y ago
      I've posted my full test scripts and results here: http://williamting.com/2012/04/19/performance-decrease-using-env/ http://williamting.com/2012/04/19/performance-decrease-using... I've been using this handle since '93 out of inertia. My tests were done on an idle i7-620M, sequentially. If your invocation takes about 0.015s, that means you're not looping enough for differences to appear. A 2% increase from 0.015s is 0.0153s, invisible due to significant digits cut off.
AncientPC 15y ago
I wrote a few small scripts to test this out. Each contained a simple print statement, then looped it 5000 times and timed it. There is ~2.1% overhead when using env. This doesn't matter in most cases, but in a Google-sized company with standardized environments it's worth it.

nodata 15y ago

Maybe because it's a hack because to get around the following not working? #!python

emmelaich 15y ago
Yes; #!python should be made to work. Not to use the user's PATH, but some system ordained python (via some kernel work). env is a hack.
- rbanffy 15y ago
  > via some kernel work Kernels shouldn't be aware of the search path shells use.
  - nodata 15y ago
    He may be referring to binfmt.
    - emmelaich 15y ago
      I wasn't referring to implementation really, but the principle. The main python could be in /bin or /usr/bin or possibly /usr/local/bin. The poor user or developer shouldn't be required to anticipate where and should not need to resort to hacks such as env.
  - emmelaich 15y ago
    I agree, as evidenced by the bit you didn't quote. (?!)
rbanffy 15y ago
Bad idea. Which python (or program named "python") will run your script?
- deleted 15y ago
  [deleted]