17 ms·
Unexpected places you can and can’t use null bytes
- ChrisSD 7y ago> While we’re on the topic, it’s worth noting that the only other restriction on filenames is that that they cannot contain a /, which is the character used to denote directories. Filenames can contain arbitrary other binary data, including spaces and newlines, and there’s no defined character encoding. I've seen this byte people when junk gets written to a filename (either accidentally or maliciously). Especially in shells but also in other programming languages. Issues that aren't always handled well include file names that: * include a newline or some other control characters * start with a `-` * aren't valid UTF-8
- heavenlyblue 7y agoI have recently accidentally created a folder named “~”. Then I tried deleting it through shell.
- NullPrefix 7y agoYou can "safely" delete it with non-recursive rmdir
- bhaak 7y agoDeleting a dir with rmdir is always safe :-) The other trick that makes dangerous operations always more safe is prepending "./" to the path.
- smichel17 7y agoI did similar, once. I had a script where I misquoted and ended up with a directory starting with ~ It was the only directory starting that way, so I typed "rm -rf ~<TAB><ENTER>" Hit control-C a half second later, but the damage was done. Fortunately most of my important files are backed up. Lesson: when deleting files with tricky names, write the command without flags first, then add "-rf" after the path is confirmed correct.
- heavenlyblue 7y agoI’ve got lucky with some directory that is called .asound or similar. It was the first one in the home directory and it didn’t manage to go beyond that.
- naedish 7y agoAren't the filenames . and .. also restricted in Unix? Windows has many more restrictions. [1] [1] https://kizu514.com/blog/forbidden-file-names-on-windows-10/ https://kizu514.com/blog/forbidden-file-names-on-windows-10/
- ChrisSD 7y agoNote that "forbidden" filenames only apply to win32 paths. "Extended-length" paths can be used to create and open such files. This can cause issues with applications that don't know how to handle them. But yes, Windows reserves (16 bit encoded) ASCII control characters and some special symbols like question mark and colon. As for . and .. I believe that on UNIX they are just symlinks, though they may be treated specially by applications? On Windows they don't really exist but some APIs emulate them when resolving paths.
- cesarb 7y ago> As for . and .. I believe that on UNIX they are just symlinks, though they may be treated specially by applications? No, they are treated specially by the kernel and/or the filesystem. They behave similar to hardlinks to the corresponding directory (hardlinks to directories aren't allowed nowadays, these two are an exception). The special treatment by applications is that many applications hide all files and directories starting with a dot, which happens to also apply to these two.
- ChrisSD 7y agoAh, thanks for the correction.
- malkia 7y agoAlso peculiar, in Windows, is handling of files that have spaces at the end :) - I've seen in my gamedev career more than one case, where a tool doesn't do enough trimming on spaces only to bite us later.
- Someone 7y agoThere are two distinct sets of restrictions on file names: those of the OS API (in United, that means no null bytes, / is a directory separator), and those of the file systems in the path, which can be (almost) anything. Quite a few file systems require names to be valid text in a specific encoding, not arbitrary byte sequences. Also, note the use of a plural in “the file systems in the path”. A file systems mounted in another one can change the rules halfway-through. There is not single fixed set of restrictions to file names.
- DonHopkins 7y agoGatorBoxes (an AppleTalk<=>Ethernet Apple File Sharing / NFS bridge) would let Mac users create files in the Unix file system with slashes in their names (presumably because of NFS shenanigans going on in the kernel). Macs let you use "/" in file names, but instead used ":" as the path separation character. It seemed to work fine and dandy at the time, until you ran "restore" and discovered your backups were corrupted! https://en.wikipedia.org/wiki/GatorBox https://en.wikipedia.org/wiki/GatorBox https://news.ycombinator.com/item?id=20007875 https://news.ycombinator.com/item?id=20007875 Just for fun: guess what happens when you create a Mac file name with a slash in it today? It works! Or it seems to work. But behind the scenes, the Finder and Mac user interface libraries actually convert the "/" to a ":", which you can see with "ls". But at least it doesn't corrupt your backups!
- nneonneo 7y agoClassic MacOS used : for the directory separator, so you could freely put / in your filenames. Modern macOS converts / in the UI to : under the hood in order to retain compatibility, while banning : in the UI (the same restriction that classic MacOS had).
- kstenerud 7y agoIt seems a bit odd to call it "unexpected" when any C API that accepts a char* and doesn't include a length parameter is commonly understood to expect a null terminated string, and any API that has a length parameter likely won't have this restriction.
- ferzul 7y agoyou missed the point. it's not that this specific api call can't take embedded \0; it's that there is no alternative api that allows embedded null. there's no way to open a file by using a string containing \0, no matter what api you pick, but you can write data to a file which contains null if you pick the right function. there's no apriori way to know which apis have this duplication and which don't.
- kstenerud 7y agoI don't follow what you're getting at. If there's no length field, it's going to stop scanning on the first null. If there is a length field, it will keep reading until the length is reached. That's the general contract with C APIs. If you actually NEED strings with nulls in them (although I couldn't think of a reason why), you'll need to use/find/create APIs with length fields.
- magicalhippo 7y agoI ran across an issue with some Win32 API call which took a char pointer and a length, but which terminated the string at the first null char. I forgot which one it is, but it was related to text boxes (edits) somehow.
- kstenerud 7y agoPassing strings with embedded NULs would probably break the API in other ways anyway. I'm glad they guarded against that.
- rvnx 7y ago
- cpeterso 7y agoPedantic nit: the ASCII '\0' character is called NUL, not null or NULL. C strings are NUL-terminated or zero-terminated, not null-terminated. https://en.wikipedia.org/wiki/ASCII https://en.wikipedia.org/wiki/ASCII
- taftster 7y agoHonest question (and not being rude). Is that strictly just being pedantic about it? Is the concept of NUL vs "NULL" semantically the same, and so you are just pointing out the ASCII abbreviation being used (three characters being consistent in the spec)? Genuinely curious if the concept of ASCII NUL '\0' can be interpreted differently in other contexts (other languages maybe)? [edit, to clarify the question] Like for example, the ASCII code '\0' is still a valid "thing" (it's a byte sequence, kind of). But in other contexts (other languages maybe), NULL is not a "thing" per se; it's more of a non-thing. How does a C programmer see the difference between NUL and NULL?
- Sharlin 7y agoThe two things referred to by the terms ”NUL byte” and ”null pointer” (aka ”NULL”) are quite distinct and mixing them up is not advisable, but calling the NUL byte a NULL byte, where it’s clear from the context what is meant, is not that confusing.
- thaumasiotes 7y agoIn the context of C, where NUL characters are most likely to be discussed, "NUL byte" and "null pointer" refer to exactly the same thing, that thing being the integer value 0.
- AnimalMuppet 7y agoOnly if pointers are 8 bits. More specifically, "NUL byte" is the character that has value 0, whereas "null pointer" is the pointer that has value 0. OK, I'll try to stop being the pedant. I'll let the real pedants poke holes in what I said...
- cdcarter 7y agoNeat article, but the author doesn't provide an actual reason you'd _need_ to pass a NUL byte into something like a socket address, or command arg. Is there an (perhaps obvious, or not) common usage of NUL byte literals being passed around, not for the purpose of terminating strings? Just terrible ye-olde file formats?
- deleted 7y ago[deleted]
- detaro 7y agoOne common usage: NUL bytes are useful as a separator between file names etc if you want to be prepared for any possible name. E.g. many commandline tools have a "-print0" flag to switch from newlines to NUL-bytes in their output.
- warkdarrior 7y agoFile names in UTF-32 encoding?
- singron 7y agoAll kinds of binary data might have a NUL byte. E.g. if you want to write a NUL byte to a file in the shell, you might try something like echo -n $'\0' > nul This doesn't work for the reason stated in the article. The argument is instead interpreted as a string ended at the NUL byte and the file will be empty. BTW you can get around this with printf since it processes escape sequences internally. printf '\0' > nul
- nneonneo 7y agoThere's also `echo -ne '\0'`, which works similarly (it tells echo to interpret the escape sequences).
- jjnoakes 7y agoThe article does mention one usage, near the end, starting with the text "The “abstract” socket feature..."
- 7y ago
- msarnoff 7y agoOne unexpected place where null bytes are acceptable: Wi-Fi SSIDs. That’s one way to keep people off your network, I suppose.
- skrebbel 7y agoPostgreSQL TEXT fields
- nneonneo 7y agoUnexpected places where you can use null bytes: gets, fgets and scanf("%s"). All three will read and store null bytes into your string from the input, and keep going: gets and fgets only terminate at a newline character and scanf only terminates at whitespace (which doesn't include the null byte). gets and scanf("%s") are also horrifically unsafe. gets is well-known to be unsafe (to the point where you'll almost certainly get a compiler warning for using it). However, scanf("%s") is unsafe for exactly the same reason (no bound on the buffer length) yet will not produce a compiler warning. Add to the fact that these functions will accept null bytes, and you have a very dangerous buffer overflow waiting to happen.
- fao_ 7y agoThis is why you _always_ write: if (*s && *s != '\n' ...) and never: if (*s != '\n' ...)
- fwsgonzo 7y agoYou can printf a null byte just fine, you just need to provide the length, just like with fwrite: printf("%.*s", (int) nbytes, str);