13 ms·
OpenBSD now enforcing no invalid NUL characters in shell scripts
- enriquto 2y agoGreat. Now forbid spaces in filenames.
- ben_bai 2y agoFunny enough filenames are just byte sequences. So almost anything goes. There was just some patch that added '/' protection, because that's the only character that's not allowed in filenames. https://github.com/openbsd/src/commit/46f7109a9e03df89b66ada65cf07c4da6ddb41a1 https://github.com/openbsd/src/commit/46f7109a9e03df89b66ada...
- klooney 2y agoDoes this break the self extracting tarball trick, where you have a bootstrap shell script with a binary payload appended?
- oguz-ismail 2y agoNo, they still work.
- 2snakes 2y agoSurprised noone has mentioned the Crowdstrike issue, which was due to NUL characters wasn't it?
- amiga386 2y agoIt was not. The Crowdstrike issue was: 1. Their code was calling a 21-parameter "matcher" function with 20 parameters of data. 2. They didn't notice, because all the matcher rules had "allow anything" for the 21st parameter and so never looked at it. 3. They later published the first list of rules with something other than "allow anything" as the 21st parameter, direct to customers. 4. On customer machines, the first rule with a non "match everything" 21st parameter went to look at the 21st element of the 20 element array. It expected a string pointer, but instead there was random stack data. It tried dereferencing this to read the string it was expecting, which caused the kernel driver to segfault during early startup, putting customer machines in a boot loop. https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...
- raverbashing 2y ago> There appears to be one piece of software which is misinterpreting guidance of this, and trying to depend upon embedded NUL. Big oof here. Why? How? > If there is ONE THING the Unix world needs, it is for bash/ksh/sh to stop diverging further by permitting STUPID INPUT that cannot plausibly work in all other shells. We are in a post-Postel world. Amem
- jrockway 2y agoI like the term post-Postel. There are two reliability constraints that all software faces; security and interoperability. The more lax you are about validation, the more likely interoperability is. "That's weird, I'll just do whatever" is doing SOMETHING, and it's often to the end user's liking. But, you also enter a more and more undefined state inside the software on the other side, and that's where weird things happen. Weird things happening typically manifest as security problems. So the more effort you go to to minimize the possibility of entering a weird state, the more confidence you have that your software is working as specified. Postel's Law made a lot of sense to me when developing the early Internet. A lot of people were reading imperfect RFCs, and it was nice when your HP server could communicate with a Sun workstation, even though maybe some bit in the TCP header was set wrong. But now? You just gotta get it right and push a hotfix when you realize you messed something up. (Sadly, I don't think it's possible. Middleboxes are getting more and more popular. At work, we make a product where the CLI talks to the server over HTTP/2. We also install Zscaler on every workstation. Zscaler simply blocks HTTP/2. So you can't use our product. Awkward.)
- Thiez 2y agoThis is also where Google went right with QUIC: encrypt as much as possible to show middleboxes the least possible. This combats ossification. Then again it seems likely middleboxes will just block QUIC (or UDP in general).
- SAI_Peregrinus 2y agoThe Cryptographic Doom Principle (if you have to perform any cryptographic operation before verifying the MAC on a message you’ve received, it will somehow inevitably lead to doom)[1] is a sort of anti-Postel's Law. [1] https://moxie.org/2011/12/13/the-cryptographic-doom-principle.html https://moxie.org/2011/12/13/the-cryptographic-doom-principl...
- 0xbadcafebee 2y ago> If there is ONE THING the Unix world needs, it is for bash/ksh/sh to > stop diverging further by permitting STUPID INPUT that cannot > plausibly work in all other shells. We are in a post-Postel world. > > It remains possible to put arbitrary bytes *AFTER* the parts of the > shell script that get parsed & executed (like some Solaris patch files > do). But you can't put arbirary bytes in the middle, ahead of shell > script parsed lines, because shells can't jump to arbitrary offsets > inside the input file, they go THROUGH all the 'valid shell script > text lines' to get there. So here it is again, an example of OpenBSD making software behavior saner for all of us. I don't consider use of all caps over a minor issue to be sane behavior. At best it's immaturity (trying to force your point rather than persuade), and at worst it's an emotional imbalance that effects judgement. That said, it's ksh, on OpenBSD, so I couldn't care less what they do.
- PufPufPuf 2y agoWhat a weird take. There are just a few emphasized words in the commit message.
- deleted 2y ago[deleted]
- opk 2y agoI've always found the fact that zsh copes with NUL characters in variables etc to be really useful. I can see why this approach makes sense for OpenBSD but they can't prevent NULs appearing in certain places like piped input.
- yesssql 2y ago[dead]
- chmorgan_ 2y agoWow, they still use CVS...
- IcePic 2y agoThis was "answered" in 2013 at the end of this post, https://marc.info/?l=openbsd-misc&m=136724343006024&w=2 https://marc.info/?l=openbsd-misc&m=136724343006024&w=2 I guess it hasn't changed since.
- nubinetwork 2y agoSo I can't bury a tarball inside a shell script anymore?
- josephcsible 2y agoYou still can; it just needs to go at the end: > It remains possible to put arbitrary bytes AFTER the parts of the shell script that get parsed & executed (like some Solaris patch files do).
- volkadav 2y agoLooks like you might be able to at the end of the file, reading the commit message, just not willy-nilly in the middle. :)
- lupusreal 2y agoDoes this break those self-extracting script/tar files? I forget how those are done, I haven't seen one in many years.
- zx2c4 2y agoFrom the article: "It remains possible to put arbitrary bytes AFTER the parts of the shell script that get parsed & executed (like some Solaris patch files do). "
- sneela 2y agoAre you talking about Shar? https://en.wikipedia.org/wiki/Shar_(file_format) https://en.wikipedia.org/wiki/Shar_(file_format)
- ape4 2y agoThat was a neat idea back in the day but should disallowed now. Running downloaded executables considered harmful.
- Joker_vD 2y agoNot in the "Installation: just run `docker run kekw/our-shiny-ai-chatbot` in your shell" world we're living today.
- nucleardog 2y agoI think the better example is the all-too-common: “Installation: Just run `curl -sL http://goo.gl/hsjdiNgtehsn http://goo.gl/hsjdiNgtehsn | sudo bash`”
- osmsucks 2y ago> Running downloaded executables considered harmful Most executables are downloaded. :)
- 73kl4453dz 2y agoThey were generally uuencoded or similar
- bell-cot 2y agoKudos to OpenBSD! Similar to the olde-tyme "-o noexec" and "-o nosuid" options for `mount`, there should be easy, no-exceptions ways to blanket ban other types of simply obvious red-flag activity.
- mcculley 2y ago"We are in a post-Postel world" is a great way to put it. This needs to be repeated by everyone working with file formats or accepting untrusted input.
- deleted 2y ago[deleted]
- nabla9 2y agoAgreed. When every implementation in wide use has their own quirks, you must support them all to make your program widely used. Every special case is yet another potential bug to chase down. It also allows "Embrace, extend, and extinguish" -strategy that Microsoft used so successfully to assfuck the internet over a decade.
- deleted 2y ago[deleted]
- pjmlp 2y agoI think you mean Google.
- nabla9 2y agoNo. The Microsoft. MS invented the term. DOJ found that MS used "Embrace, extend, and extinguish" in internal documents. Younger people don't know how absolutely ruthless and harmful Wintel monopoly was under Gates. Java did not work on purpose. Javascript did not work for purpose. <!--[if IE]> everywhere. They attempted to kill open web in the crib with their blackbird project. Only MSN (The Microsoft Network) for normal people.
- pjmlp 2y agoExcept it Google that morphed the Web into ChromeOS, with the help of EVERYONE that ships it alongside their applications, as they can't be bothered to learn cross-platform frameworks. Many of them people that used to complain about Micro$oft and should know better.
- sneela 2y ago> This was in snapshots for more than 2 months, and only spotted one other program depending on the behaviour (and that test program did not observe that it was therefore depending in incorrect behaviour!!) Fascinating. I wonder what that program is, and why it depends on the NUL character.
- chasil 2y agoI was going to check the status of mksh (the Android system shell), but the project page returns: "Unavailable For Legal Reasons - Sorry, no detailled error message available." http://www.mirbsd.org/mksh.htm http://www.mirbsd.org/mksh.htm The Android system shell is now abandoned? This is also in rhel9 basesos.
- talideon 2y agoFine for me. I just got a HTTP warning and nothing else. ~~I believe Android uses toybox, not mksh.~~ It does use toybox, but toybox doesn't appear to include a shell.
- chaosite 2y agoLooks fine here, maybe they're blocking your IP range for some reason?
- kbolino 2y agoIt's blocked for me too, but only on my home Internet (Xfinity), not my phone (Google Fi/T-Mobile).
- tux3 2y agoWorks from an EU IP, so whatever it is, it's probably not GDPR?
- sph 2y agoIs this in reference to something? Judging from the comments, NUL bytes in shell scripts are a common occurrence that everybody is celebrating this change as if it were ground breaking. I mean, it's a good idea, but I wonder what am I missing here. Also what do they mean by post-Postel?
- semiquaver 2y agoPostel’s Law: https://datatracker.ietf.org/doc/html/rfc761#section-2.10 https://datatracker.ietf.org/doc/html/rfc761#section-2.10
- JimDabell 2y agoPostel’s Law, also known as the Robustness Principle: > be conservative in what you do, be liberal in what you accept from others It’s intended as a way to maximise compatibility, and people have generally followed it when designing protocols and file formats. However it’s led to many security vulnerabilities and has caused a lot of compatibility problems itself. These days a lot of people are realising that it’s more harmful than helpful.
- BlackFly 2y agoEarly spec of TCP had a section on the robustness principle that was generally known as Postel's law (https://datatracker.ietf.org/doc/html/rfc761#section-2.10 https://datatracker.ietf.org/doc/html/rfc761#section-2.10). At the time and until recently this was considered good design. Nowadays people generally want servers to be stricter in what they accept since decades of experience dealing with diverging interpretations of a specification create problems for interoperability.
- eesmith 2y ago"until recently"? More than 10 years just going by HN. https://news.ycombinator.com/item?id=5161214 https://news.ycombinator.com/item?id=5161214 I think HTML showed the problem with Postel's principle. Quoting "Postel’s Law is not for you" at http://trevorjim.com/postels-law-is-not-for-you/ http://trevorjim.com/postels-law-is-not-for-you/ from 2011 > The next version of HTML, HTML5, should considerably reduce the problem of browser incompatibilities. It does this, in part, by rejecting Postel’s Law for browser implementors. Instead of allowing browsers to be liberal when dealing with “flawed” markup, HTML5 requires them to parse it exactly as in the HTML5 specification, and that specification is given much more precisely than before, in the form of a deterministic state machine, in fact. HTML5 is trying to give implementors no leeway at all in this, in the name of browser compatibility.
- saagarjha 2y ago> There appears to be one piece of software which is misinterpreting guidance of this, and trying to depend upon embedded NUL. Curious what this is
- semiquaver 2y agoI wonder if it’s https://justine.lol/ape.html https://justine.lol/ape.html / cosmopolitan libc
- eesmith 2y agoShouldn't be. See the "exit 1" in your link? That's the end of the shell script, and as the OpenBSD link says; > It remains possible to put arbitrary bytes AFTER the parts of the shell script that get parsed & executed (like some Solaris patch files do). But you can't put arbirary bytes in the middle,
- oguz-ismail 2y agoIt is. Binaries generated by cosmocc have NUL in the middle.
- comex 2y agoAh, indeed. Here are the first 16 bytes of one: 4d 5a 71 46 70 44 3d 27 0a 00 00 10 00 f8 00 00 |MZqFpD='........| There are already nul bytes here, and there are a lot more before the single quote gets closed at offset 0x200.
- eesmith 2y agoAnd I can confirm a NUL in 11th byte of my hello.c a.out: >>> s[:11] b"MZqFpD='\n\n\x00" Looking closer, I missed the content of "BIOS BOOT SECTOR".
- chubot 2y agoI'm pretty sure it is, I remember reading something about this Yeah I found it here https://news.ycombinator.com/item?id=41030960 https://news.ycombinator.com/item?id=41030960 2019 bug - https://austingroupbugs.net/view.php?id=1250 https://austingroupbugs.net/view.php?id=1250 https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ > This is an idea whose time has come; POSIX even changed their rules about binary in shell scripts specifically to let us do it. FWIW I agree with this OpenBSD change, which says more pointedly All the shells are written in C, and majority of them use C strings for everything, which means they cannot embed a NUL, so this is not surprising. It is quite unbelievable there are people trying to rewrite history on a lark, and expecting the world to follow alone. i.e. it's not worth it to change a bunch of old code in order to allow making code more esoteric. We want systems code to be more predictable, reliable, and less esoteric ... not more esoteric
- Taikonerd 2y agoOn a similar note, I sometimes think about how newline characters are allowed in filenames, and how that can break simple... for each $filename in `ls` loops -- because in many contexts, UNIX treats newlines as a delimiter. Is there any legitimate use for filenames with newlines?
- xxpor 2y agothis is why things like `find -print0` exist, which is IMO the easiest way to handle this robustly.
- Joker_vD 2y agoSticky notes on the desktop :) Who needs data storage when you can store it all in the metadata?
- IsTom 2y agoYou can also create files named e.g. '--help' (if you're not particularly malicious) and with globbing it'll cause e.g. 'ls *' to print help.
- jasonjayr 2y agotouch -- '-f ..' (If you want to lay an evil trap) Remember that in most option parsing libraries, putting '--' in your arguments stops option parsing, so you can safely run: rm -- '-f ..'
- bityard 2y agoWell, knowing how to deal with wacky input and corner cases are a requirement of learning ANY programming language. Bourne-style shells are no exception. Your example has illegal syntax, but the biggest issue is that you should never parse the output of ls. The shell has built-in globbing. This is how you would loop over all entries (files, dirs, symlinks, etc) in the current directory without getting tripped up by whitespace: for e in *; do echo "got: $e"; done
- dzaima 2y ago
- soupbowl 2y agoI wish FreeBSD replaced /bin/sh with OpenBSDs.
- rollcat 2y agoFreeBSD made many cool moves in the 14.0 release, like finally getting rid of sendmail and adopting DMA (the irony), so perhaps there's a chance? But FreeBSD has always been much less focused on polish/cleanliness than OpenBSD; I mean - they have THREE firewalls, wtf.
- toast0 2y ago> they have THREE firewalls, wtf. I've not used ipf, but ipfw and pf have a different model and different features (although in 14.0, there's more overlap). I have to use them both.
- chrisfinazzo 2y agoRelated: The installer for iTunes 12.2.1 included a bug which might recursively delete a volume if the path given as input included incorrectly escaped spaces.
- NewJazz 2y agoReminds me of this... https://hackaday.com/2024/01/20/how-a-steam-bug-once-deleted-all-of-someones-user-data/ https://hackaday.com/2024/01/20/how-a-steam-bug-once-deleted...
- amiga386 2y agoHere's the actual diff: https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/ksh/shf.c.diff?r1=1.34&r2=1.35 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/ksh/shf.c.... And it looks like that covers all parsed parts of the shell script or history file, including heredocs. I get the feeling it's going to break all shar archives with binary files (not that they're particularly common). It will stop NULs being in the script itself, but it won't stop them coming from other sources, e.g. $ var=$(printf '\0hello') -bash: warning: command substitution: ignored null byte in input $ echo $var hello It remains to be seen if this will be adopted by anyone else, or if it'll be another reason to use OpenBSD only as a restricted environment and not as a general computing platform. > "If there is ONE THING the Unix world needs, it is for bash/ksh/sh to stop diverging further" > OpenBSD ksh: diverges further
- bell-cot 2y ago> Here's the actual diff: Only 8 short, simple lines of c code. Beautiful.
- raverbashing 2y ago> I get the feeling it's going to break all shar archives with binary files (not that they're particularly common) Base64 encode them. This is not diverging further, this is bringing sanity to the table
- jolmg 2y ago> I get the feeling it's going to break all shar archives with binary files shar encodes binary files. Here's what it does with a file that has contents: "foo\0bar\n": sed 's/^X//' << 'SHAR_EOF' | uudecode && begin 600 foo.txt (9F]O`&)A<@K. ` end SHAR_EOF Interestingly, passing that heredoc to uudecode in the shell, it produces no output. However, if I pass the whole shar output to unshar, it does produce the file with the correct content.
- matrix2003 2y agoEh - I actually like developing on OpenBSD first, because of restrictions like this. If it runs on OpenBSD, you are likely to have fewer bugs around things like malloc. OpenBSD is also really good about upstreaming bug fixes, which is a good thing. Firefox used to be a dumpster fire of core dumps on OpenBSD, and many issues were uncovered and fixed that way.
- whiterknight 2y agoSide note: tell your startup to switch its “hardware with Ubuntu Linux inside” to BSD. You will have a much more stable and simple platform that can last a long time.
- quesera 2y agoThe recommendation is solid, but FWIW no one looking for stability would choose Ubuntu, among the Linuxen!
- deleted 2y ago[deleted]
- parasense 2y agoIs this going to murder those fancy shell scripts that self-extract a program appended to the tail, which is really just an encoded blob of some kind, presumably compressed, etc.. ???
- talideon 2y agoNot if it was done competently. Shar files and the likes shouldn't contain NULs, even if they contain compressed data. The appended data should be binary safe.
- Thiez 2y agoAnd in case your data does contain NULs, presumably one could add a layer of base64 encoding. Not nice for the filesize, but also much less likely to upset a text editor when the script is opened (even in the absence of NUL bytes).