7 ms·
A peculiarity of the GNU Coreutils version of 'test' and '['
- yjftsjthsd-h 3y ago> PPS: In theory GNU Coreutils is portable and you might find it on any Unix. In practice I believe it's only really used on Linux. Oh how times have changed:) But even with the demise of most commercial unixen, I was under the impression that Darwin preserved the ancient tradition of installing the system and then immediately replacing its coreutils with GNU?
- monocasa 3y agoDarwin's coreutils are mainly updated from FreeBSD I think. They've got some gnu stuff, but it hasn't been updated since gnu switched to GPLv3.
- yjftsjthsd-h 3y agoRight; the coreutils built in to Darwin are BSD-derived, I was referring to a user installing the GNU versions over top of the OS proper.
- n8henrie 3y agoYes, like I did first with homebrew and now with nix.
- divbzero 3y agoI bet many of us do this due to small differences between BSD and GNU coreutils. One that always gets me is: sed -i '' 's/foo/bar/g' # BSD sed sed -i 's/foo/bar/g' # GNU sed
- jorams 3y agoThat is indeed an incredibly frustrating difference, because there is no version of it that works on both GNU and macOS. In practice I often end up using perl to do an in-place search-replace instead.
- jcelerier 3y agoIt takes a second to install gnu coreutils with homebrew and make Macs barely useable
- emmelaich 3y agoOr just use Perl
- DonHopkins 3y agoAs long as you're using stone knives, might as well throw in some bear skins. https://www.youtube.com/watch?v=F226oWBHvvI https://www.youtube.com/watch?v=F226oWBHvvI
- emmelaich 3y agoFun! But honestly there's a place for Perl. There's far less variance between Perls on different systems than there are for sed/awk/shell. So if you want performant portable code, Perl does better than all of those. I'd never use Perl for a 'big' program these days but it still beats the crap out of the mess of sed/awk/bash/ksh/zsh.
- freedomben 3y agoApple has been threatening to remove perl for ages. One of these days they'll find the "courage" to do so and we'll need a better strategy.
- arp242 3y agoFreeBSD removed Perl from base ages ago, in FreeBSD 5 or 6. That went alright. I don't even have it on my Linux system any more; for better or worse, fewer and fewer things use it.
- xcdzvyn 3y agoUnfortunately not. macOS comes with violently outdated FreeBSD coreutils, for the GPLv3 situation. Though the default shell was recently changed to zsh from Bash 3.2 from 2007.
- yjftsjthsd-h 3y agoRight, which is why I thought it was common for people to install the operating system, then immediately use macports or homebrew to go get the GNU coreutils and a modern version of bash because the BSD versions are less friendly. (Thus continuing and extremely long tradition of layering GNU over the vendor tools; ex. Sun had their own tools, but everyone liked the GNU versions better.)
- stephenr 3y agoPeople do it, but why you would when zsh is right there, is beyond me. Possibly the same type of people that target bash specifically in shell scripts I guess.
- matheusmoreira 3y agoWhat's wrong with bash though? Targeting bash specifically has massively improved my productivity and decreased the incidence of easily avoidable mistakes. Portable POSIX shell scripting is hell on earth but bash scripting with shellcheck can be surprisingly pleasant. I had the same experience with portable makefiles and GNU Make. For example, I managed to create a surprisingly good test suite with bash and the rest of the GNU coreutils: https://github.com/lone-lang/lone/blob/master/scripts/test.bash https://github.com/lone-lang/lone/blob/master/scripts/test.b... It even runs in parallel. Submitted a patch to coreutils to implement the one thing it couldn't test, namely the argv[0] of programs. I should probably go check if they merged it...
- stephenr 3y ago> Portable POSIX shell scripting is hell on earth but bash scripting with shellsheck can be surprisingly pleasant. Assuming you meant `shellcheck`: you know it works for POSIX compatible shell too right? What's wrong with bash is that people invariably end up requiring features that aren't available in some deployed version, and then you've lost a lot of the benefit of writing in shell in the first place. Similar to shellcheck, shunit2 works just fine for running unit tests in posix compatible shell.
- zappb 3y agoThat was the case for a while, but when they were updated to use GPLv3, Apple stopped updating them, probably to avoid licensing problems on iOS. Nowadays you can install them from Homebrew, MacPorts, straight from source, or other methods.
- o11c 3y agoand in case anyone wants to bring up GNU/kFreeBSD - that's officially dead as of July.
- yjftsjthsd-h 3y agoTragically true:( But you can still install https://www.freshports.org/sysutils/coreutils/ https://www.freshports.org/sysutils/coreutils/ on FreeBSD if desired.
- chungy 3y agoI tend to install coreutils on FreeBSD. Besides some minor annoyance with every command being prefixed with "g", some of the programs work a bit nicer than the FreeBSD-shipped versions (or some Linux-centric programs just want the coreutils versions...).
- 1500100900 3y agoI'd be interested in learning which commands you have in mind and what specifically is a bit nicer about their coreutils implementation.
- orra 3y agoIn GNU utilities, option arguments can come after (or between) positional arguments. Personally I find this small convenience invaluable, because I'm used to it.
- usr1106 3y agoOh, I had no idea that GNU utilities allow this. As a Unix graybeard I always place options first. Options last feels like Windows command prompt, so nothing I want to see... I always tell younger colleagues who place options in the end, it might work with some commands, but just don't do it. I did not know that "some" includes all of GNU coreutils? A single common code style is a virtue, even in interactive use if there are onlookers. So I guess I will continue to point it out.
- mike_hock 3y agoEven many command line parsing libraries support it and scan the entire argv for options. You should always terminate the options with "--" if it's in a script and any of the positional arguments are variables that might or might not start with a dash.
- dveeden2 3y agoInstalling GNU coreutils etc was also common on Solaris. However as it usually wasn't in your path before SysV utils it was only used with its full path.
- pixelbeat__ 3y agoYes, macos takes its utils from FreeBSD. An interesting factoid is that FreeBSD/macos sort(1) was using GNU code until recently, since this is quite tricky to implement. Eventually it was reimplemented for GPL avoidance reasons. We do consider macos though, and ensure all tests pass on macos for each release
- freedomben 3y agoDoes Apple compensate or sponsor that work at all?
- pixelbeat__ 3y agoDefinitely not
- nomoreusernames 3y ago[dead]
- hallman76 3y agoThanks hn for doing what you do! I'm so glad to see this here. A few months back I noticed '[' under /bin on my mac. I tried googling to understand what it was, but my google-fu came up short. It felt like one of those ungoogleable things. This link is an excellent starting point for me.
- dylan604 3y agoit is funny that [ is the name of binary but when using it, you can't just type [ without the closing ] by the name, you'd think you could just use [ 0 -gt 1 <enter> after all, we don't have to type grep query perg
- yjftsjthsd-h 3y agoThat does work if you call it as `test`; just a quirk of faking shell syntax in an external binary I think.
- dylan604 3y agoI was halfway expecting /bin/[ -> /bin/test, but that did pass the smell test since the ] is required, so it couldn't just be symlink
- yjftsjthsd-h 3y agoIt could have still been implemented like that in the filesystem, if the binary read the name it was executed under and modified its behavior based on that. As an extreme case, on a stock Alpine Linux system, both test and [ - along with most other core system programs - are all symlinks to a single busybox binary that reads argv[0] and acts like whatever program it's been called as. I'm actually somewhat surprised that GNU didn't do that in this case; I, too, would have expected test and [ to be some manner of link to the same program, either with identical behavior or using invocation name to decide how to behave.
- renewiltord 3y ago
- dang 3y agoRecent and related: test, [, and [[ (2020) - https://news.ycombinator.com/item?id=38387464 https://news.ycombinator.com/item?id=38387464 - Nov 2023 (225 comments)
- 1vuio0pswjnm7 3y agoThere must be reasons to use the standalone programs instead of the shell builtins. But the test builtin will be the same in, e.g., NetBSD sh as it is in Linux or Android dash. That is what I use 100% of the time. It is much faster to use builtins.
- usr1106 3y agoMuch faster in theory, unnoticeable in practice. I assume nobody does number crunching using shell scripts.
- arp242 3y agoYou don't need to do "number crunching"; the performance differences are very real even for simple scripts on modern systems. Launching a process is comparatively expensive (read binary from disk, run linker, allocate new process space in kernel, etc.) whereas a builtin is just "if progname == "[" { .. }". And when you do something like "loop over a few thousand files" the difference really adds up. Consider this example: for i in $(seq 1 10); do for f in /*; do if [ $f = /etc ]; then : # Do nothing. fi done done I have 23 entries in /, so this will run [ 230 times – not a crazy number. % time sh test sh test 0.00s user 0.01s system 91% cpu 0.009 total bash, dash, zsh: they all have roughly the same performance: "fast enough to be practically instantaneous". But if I replace [ with /bin/[: % time sh test sh test 0.11s user 0.38s system 96% cpu 0.509 total Half a second! That's tons slower! And you can keep "fast enough to be practically instantaneous" for thousands of files with the built-in [, whereas it will take many seconds with /bin/[. (Aside: if I statically link [ it's about 0.4 seconds).
- usr1106 3y agoI know what it costs to start a process. My point is just if that script is done in 0.5 seconds that's good enough for me as an occasional human caller. I don't care that it could be much faster. Of course if the script were to handle all files of a filesystem with many small files it could get disturbingly slow. I don't deny that there are cases were it matters. But in over 90% of the scripts I write or use it doesn't.