8 ms·

Google also makes it difficult to compile Chromium on platforms where /usr/bin/python is python 3, because their shebangs are: #!/usr/bin/env python Rathe

Google also makes it difficult to compile Chromium on platforms where /usr/bin/python is python 3, because their shebangs are: #!/usr/bin/env python Rather than the more portable: #!/usr/bin/env python3 Notably this causes problems for people on Arch Linux, and in the near future, Fedora. This post is presented as a style guide that others should adopt. Google may have bash ubiquiotously available internally, but their approach should not be taken as a model for others. Google's approach has caused real problems for their software's portability in exchange for dubious value-adds. >What about POSIX shell makes it superior for scripting? To address this specifically, I have written a blog article on the subject: https://drewdevault.com/2018/02/05/Introduction-to-POSIX-shell.html https://drewdevault.com/2018/02/05/Introduction-to-POSIX-she... The main points are: - bash is far from as ubiquitous as some people seem to think it is - sh is formally standardized and has multiple competing implementations, whereas bash is defined by its implementation and there is only one - sh is perfectly competent and easy to target, bash's value add is questionable - By the time you need the things bash offers you might as well not use a shell script at all

marcoperaza 8y ago
Competing implementations sounds like a potential downside to me. That’s just more possible configurations where a bug could be hiding. Even if you use standard posix shell, you’d be wise to target a single implementation and stick to it.
- Sir_Cmpwn 8y ago
  So you think that competition between gcc and clang is harmful to the C and C++ ecosystems? If you find bugs in an implementation of POSIX sh, you should report it to that implementation. Multiple competing implementations is a sign of a good standard, it proves the standard's correctness and demonstrates its maturity. There is no standard to hold bash to. Any strange behavior of it might be decided as by design and kept forever (and replicated by anything which tries to be bash compatible) or it could be determined to be a bug and, when fixed, break the scripts which worked around it or depended on it. There's no way to know how this is going to shake out because there is no objective specification for the language. At least when there's a bug in one of these competing sh implementations that you're so worried about, it's objectively a bug and can be fixed.
  - marcoperaza 8y ago
    To be more precise, my point was that you should only be targeting one of the implementations anyway. I don't see how it's a good use of people's time to be testing their scripts against every shell that claims to be sh-compatible, especially at a place like Google where the scripts are almost all being written to run on completely Google-controlled environments. But I didn't think about the bug vs. by-design issue and how that is remedied by a well-defined standard. That does seem to be a factor in favor of using a standardized language. But it doesn’t seem different in principle than depending on any software. It is liable to change and leave you with a difficult decision of forking the library or modifying your code that consumes it. Given bash’s ubiquity, it seems reasonable to expect a good level of stability. If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. Linux, for example, has all sorts of performance and functionality enhancing features over what POSIX provides.
    - Sir_Cmpwn 8y ago
      >If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. In this case, you won't. I addressed this a few comments ago: >By the time you need the things bash offers you might as well not use a shell script at all
  - jhasse 8y ago
    > competition between gcc and clang is harmful to the C and C++ ecosystems? Yes. GCC will die eventually.
- _jal 8y ago
  This is exactly backwards. If you don't have separation of specification and application, you don't have a specification. I mean, standards aren't necessary; interpreters can absolutely be snowflakes. But multiple implementations is a sign of strength, and intentionally targeting the bugs of one implementation is short sighted, if sometimes required.
mort96 8y ago
If you find yourself compiling Google-stuff a lot on Arch, you might find this useful: https://github.com/mortie/nixConf/blob/master/bin/py2 https://github.com/mortie/nixConf/blob/master/bin/py2 I wrote that script exactly because of the issue you described, and I got tired of changing what /usr/bin/python is symlinked to constantly. With that script, I just run `py2 gn gen out/Debug` or whatever, and my /usr/bin/python stays intact so everything else which expects python3 works, and only the `gn` process and all subprocess sees python2. This definitely doesn't excuse Google's behavior, but I don't expect it to be fixed soon. My bug report about it (https://bugs.chromium.org/p/webrtc/issues/detail?id=7376 https://bugs.chromium.org/p/webrtc/issues/detail?id=7376) was closed as wontfix (though I'm not entirely sure the person who closed it understood that I was asking for the python scripts to explicitly use python 2, not for them to port the scripts to python 3).
- moviuro 8y ago
  Clearly the WONTFIX was wrongly issued. The issue should be re-opened.
chrisfinazzo 8y ago
The python v python3 thing is their (Py's implementers) own stupid fault; although the argument could be made that they didn't have any choice. People have been on 2.7 for so long that a breaking change might be the only thing to get them to move over. As Chromium is open source, I'm surprised they haven't really explained how people outside of the Googleplex should handle this. Seems like they're left to their own devices.
joshuamorton 8y ago
The second thing is not more portable than the first (unfortunately).
- Sir_Cmpwn 8y ago
  https://www.python.org/dev/peps/pep-0394/ https://www.python.org/dev/peps/pep-0394/
  - joshuamorton 8y ago
    That, unfortunately, only works if you don't need to maintain backwards compatibility with older systems. `python` will exist and point to python2 on any system. `python2` may not if it predates or ignores the PEP. `python3` didn't even come installed by default on a lot of nixes until recently, making it the worst of option of the three.
    - Sir_Cmpwn 8y ago
      Of course, it will take time for this to become effective. But this is the portable solution. Systems for which this does not work are broken and should be fixed, software should not be updated to accomodate for broken systems.
      - joshuamorton 8y ago
        >software should not be updated to accomodate for broken systems. Its not. The software was written, and maintains, backwards compatibility with systems that predate or partially predate the spec. The spec was written to be backwards compatible with those systems. Using the backwards compatibility option is correct if you wish to maintain backwards compatibility. Consider that Arch also technically violates Pep 394, > for the time being, all distributions should ensure that python, if installed, refers to the same target as python2 We shouldn't update the shebangs in all of our software ever to accommodate a misbehaving, spec-non-compliant distro, should we?
hyperpape 8y ago
I believe your claim that /bin/sh is standardized is incorrect, though I don't know how many actual systems fail to have it present. Source: https://news.ycombinator.com/item?id=17069408 https://news.ycombinator.com/item?id=17069408 (I checked the 2018 version of the spec, and it says the same).
- Sir_Cmpwn 8y ago
  Aye, I was incorrect. I have pushed a correction, but it will take a moment to appear on the site.

marcoperaza 8y ago

Competing implementations sounds like a potential downside to me. That’s just more possible configurations where a bug could be hiding. Even if you use standard posix shell, you’d be wise to target a single implementation and stick to it.

Sir_Cmpwn 8y ago
So you think that competition between gcc and clang is harmful to the C and C++ ecosystems? If you find bugs in an implementation of POSIX sh, you should report it to that implementation. Multiple competing implementations is a sign of a good standard, it proves the standard's correctness and demonstrates its maturity. There is no standard to hold bash to. Any strange behavior of it might be decided as by design and kept forever (and replicated by anything which tries to be bash compatible) or it could be determined to be a bug and, when fixed, break the scripts which worked around it or depended on it. There's no way to know how this is going to shake out because there is no objective specification for the language. At least when there's a bug in one of these competing sh implementations that you're so worried about, it's objectively a bug and can be fixed.
- marcoperaza 8y ago
  To be more precise, my point was that you should only be targeting one of the implementations anyway. I don't see how it's a good use of people's time to be testing their scripts against every shell that claims to be sh-compatible, especially at a place like Google where the scripts are almost all being written to run on completely Google-controlled environments. But I didn't think about the bug vs. by-design issue and how that is remedied by a well-defined standard. That does seem to be a factor in favor of using a standardized language. But it doesn’t seem different in principle than depending on any software. It is liable to change and leave you with a difficult decision of forking the library or modifying your code that consumes it. Given bash’s ubiquity, it seems reasonable to expect a good level of stability. If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. Linux, for example, has all sorts of performance and functionality enhancing features over what POSIX provides.
  - Sir_Cmpwn 8y ago
    >If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. In this case, you won't. I addressed this a few comments ago: >By the time you need the things bash offers you might as well not use a shell script at all
- jhasse 8y ago
  > competition between gcc and clang is harmful to the C and C++ ecosystems? Yes. GCC will die eventually.
_jal 8y ago
This is exactly backwards. If you don't have separation of specification and application, you don't have a specification. I mean, standards aren't necessary; interpreters can absolutely be snowflakes. But multiple implementations is a sign of strength, and intentionally targeting the bugs of one implementation is short sighted, if sometimes required.

mort96 8y ago

If you find yourself compiling Google-stuff a lot on Arch, you might find this useful: https://github.com/mortie/nixConf/blob/master/bin/py2 https://github.com/mortie/nixConf/blob/master/bin/py2 I wrote that script exactly because of the issue you described, and I got tired of changing what /usr/bin/python is symlinked to constantly. With that script, I just run `py2 gn gen out/Debug` or whatever, and my /usr/bin/python stays intact so everything else which expects python3 works, and only the `gn` process and all subprocess sees python2. This definitely doesn't excuse Google's behavior, but I don't expect it to be fixed soon. My bug report about it (https://bugs.chromium.org/p/webrtc/issues/detail?id=7376 https://bugs.chromium.org/p/webrtc/issues/detail?id=7376) was closed as wontfix (though I'm not entirely sure the person who closed it understood that I was asking for the python scripts to explicitly use python 2, not for them to port the scripts to python 3).

moviuro 8y ago
Clearly the WONTFIX was wrongly issued. The issue should be re-opened.

chrisfinazzo 8y ago

The python v python3 thing is their (Py's implementers) own stupid fault; although the argument could be made that they didn't have any choice. People have been on 2.7 for so long that a breaking change might be the only thing to get them to move over. As Chromium is open source, I'm surprised they haven't really explained how people outside of the Googleplex should handle this. Seems like they're left to their own devices.

joshuamorton 8y ago

The second thing is not more portable than the first (unfortunately).

Sir_Cmpwn 8y ago
https://www.python.org/dev/peps/pep-0394/ https://www.python.org/dev/peps/pep-0394/
- joshuamorton 8y ago
  That, unfortunately, only works if you don't need to maintain backwards compatibility with older systems. `python` will exist and point to python2 on any system. `python2` may not if it predates or ignores the PEP. `python3` didn't even come installed by default on a lot of nixes until recently, making it the worst of option of the three.
  - Sir_Cmpwn 8y ago
    Of course, it will take time for this to become effective. But this is the portable solution. Systems for which this does not work are broken and should be fixed, software should not be updated to accomodate for broken systems.
    - joshuamorton 8y ago
      >software should not be updated to accomodate for broken systems. Its not. The software was written, and maintains, backwards compatibility with systems that predate or partially predate the spec. The spec was written to be backwards compatible with those systems. Using the backwards compatibility option is correct if you wish to maintain backwards compatibility. Consider that Arch also technically violates Pep 394, > for the time being, all distributions should ensure that python, if installed, refers to the same target as python2 We shouldn't update the shebangs in all of our software ever to accommodate a misbehaving, spec-non-compliant distro, should we?

hyperpape 8y ago

I believe your claim that /bin/sh is standardized is incorrect, though I don't know how many actual systems fail to have it present. Source: https://news.ycombinator.com/item?id=17069408 https://news.ycombinator.com/item?id=17069408 (I checked the 2018 version of the spec, and it says the same).

Sir_Cmpwn 8y ago
Aye, I was incorrect. I have pushed a correction, but it will take a moment to appear on the site.