7 ms·

you missed the point. it's not that this specific api call can't take embedded \0; it's that there is no alternative api that allows embedded null. there's no w

by ferzul 7y ago

you missed the point. it's not that this specific api call can't take embedded \0; it's that there is no alternative api that allows embedded null. there's no way to open a file by using a string containing \0, no matter what api you pick, but you can write data to a file which contains null if you pick the right function. there's no apriori way to know which apis have this duplication and which don't.

kstenerud 7y ago
I don't follow what you're getting at. If there's no length field, it's going to stop scanning on the first null. If there is a length field, it will keep reading until the length is reached. That's the general contract with C APIs. If you actually NEED strings with nulls in them (although I couldn't think of a reason why), you'll need to use/find/create APIs with length fields.
- magicalhippo 7y ago
  I ran across an issue with some Win32 API call which took a char pointer and a length, but which terminated the string at the first null char. I forgot which one it is, but it was related to text boxes (edits) somehow.
  - kstenerud 7y ago
    Passing strings with embedded NULs would probably break the API in other ways anyway. I'm glad they guarded against that.
- rvnx 7y ago
  memcpy/memcmp
- taneq 7y ago
  > If there is a length field, it will keep reading until the length is reached. Well, it might. If it ever uses that string internally with some function that expects a null termination then it'll probably still get truncated.
  - kstenerud 7y ago
    Possibly, depending on the API. But then again, strings should not have embedded NUL characters. There's no good reason for it. You have 32 other control characters to choose from.
- Cogito 7y ago
  > although I couldn't think of a reason why One I've seen in the wild is an encoded data structure, a blob of null terminated strings passed around together.
  - kstenerud 7y ago
    ... in which case you're conceptually dealing with a structure, not a string.
    - Cogito 7y ago
      Sure, once you’ve unpacked it into a structure. It’s used for sending stuff over the wire, but also for storing on disk etc. Conceptually it was dealt with as one string, which happened to contain nulls.
      - kstenerud 7y ago
        You'd be better off encoding it using ETB (end of transmision block). That's what the control code was designed for.
        Cogito 7y ago
        I can’t really speak to the why. In any case. I remembered seeing nul in strings in git, not sure if the same place (maybe?) but a quick search shows up some real world usages https://github.com/git/git/search?q=nul https://github.com/git/git/search?q=nul
- lilyball 7y ago
  That's the whole point. The article even starts with an example of this. printf doesn't let you write strings with embedded NULs, but you can use fwrite instead to write them. The 3 cases given in the article of places where you cannot use NUL bytes are places where not only is there no API that lets you include the NULs, but it's impossible to have one (without changing the kernel interface).
  - kstenerud 7y ago
    That's by design. You wouldn't want a NUL in the middle of a string in any modern system. It's such an esoteric use case that there's no point in even supporting it. There are plenty of binary data interfaces to do what you want in situations where it actually makes sense. PHP supports NUL, and the result is a HUGE amount of extra support code to handle it, spread all through the PHP codebase (introducing TONS of bugs over the years), all for the 0.00001% use case. Not a good trade. Complaining about lack of NUL support would be like complaining that most text editors don't display the DC1 control character. It's just not useful in 99.999% of use cases, so there's no point in supporting it. You'd use a specialized tool instead.
    - Joker_vD 7y ago
      Hey. People wrote actual working SOCKS-proxies in PHP, and given that PHP's low-level functions represents an array of bytes as a string of chars, that'd be impossibly if NUL was forbidden in them.
      - kstenerud 7y ago
        I see... So rather than, say, simply using the binary array data type already present in C, they decided that they needed to subvert strings to support passing non-string data, resulting in tens of thousands of lines of support code to parallel the C runtime library, thousands more lines of bridge code for any outside libraries they might want to make use of, and almost a million lines of PHP implementation code (not including extensions!) carrying this extra cognitive burden in order to support the 1% use case, resulting in countless bugs and security vulnerabilities whenever someone used the C library functions, or stopped on NUL (or not) by mistake. Sounds like a great plan.
- ferzul 7y ago
  Okay, don't think in terms of function pairs, but in terms of use case. You might want to write unknown UTF16 to a file, and you might want to write UTF16 to a filename. But only one of these is possible. Why? because there's an api for writing contents to a handle that takes a length, but not one for opening a file that does. And you can't “create” api's for what the kernel doesn't support. These are limits on what is possible without forking Linux or some other kernel.

by ferzul 7y ago

kstenerud 7y ago

I don't follow what you're getting at. If there's no length field, it's going to stop scanning on the first null. If there is a length field, it will keep reading until the length is reached. That's the general contract with C APIs. If you actually NEED strings with nulls in them (although I couldn't think of a reason why), you'll need to use/find/create APIs with length fields.

magicalhippo 7y ago
I ran across an issue with some Win32 API call which took a char pointer and a length, but which terminated the string at the first null char. I forgot which one it is, but it was related to text boxes (edits) somehow.
- kstenerud 7y ago
  Passing strings with embedded NULs would probably break the API in other ways anyway. I'm glad they guarded against that.
rvnx 7y ago
memcpy/memcmp
taneq 7y ago
> If there is a length field, it will keep reading until the length is reached. Well, it might. If it ever uses that string internally with some function that expects a null termination then it'll probably still get truncated.
- kstenerud 7y ago
  Possibly, depending on the API. But then again, strings should not have embedded NUL characters. There's no good reason for it. You have 32 other control characters to choose from.
Cogito 7y ago
> although I couldn't think of a reason why One I've seen in the wild is an encoded data structure, a blob of null terminated strings passed around together.
- kstenerud 7y ago
  ... in which case you're conceptually dealing with a structure, not a string.
  - Cogito 7y ago
    Sure, once you’ve unpacked it into a structure. It’s used for sending stuff over the wire, but also for storing on disk etc. Conceptually it was dealt with as one string, which happened to contain nulls.
    - kstenerud 7y ago
      You'd be better off encoding it using ETB (end of transmision block). That's what the control code was designed for.
      - Cogito 7y ago
        I can’t really speak to the why. In any case. I remembered seeing nul in strings in git, not sure if the same place (maybe?) but a quick search shows up some real world usages https://github.com/git/git/search?q=nul https://github.com/git/git/search?q=nul
lilyball 7y ago
That's the whole point. The article even starts with an example of this. printf doesn't let you write strings with embedded NULs, but you can use fwrite instead to write them. The 3 cases given in the article of places where you cannot use NUL bytes are places where not only is there no API that lets you include the NULs, but it's impossible to have one (without changing the kernel interface).
- kstenerud 7y ago
  That's by design. You wouldn't want a NUL in the middle of a string in any modern system. It's such an esoteric use case that there's no point in even supporting it. There are plenty of binary data interfaces to do what you want in situations where it actually makes sense. PHP supports NUL, and the result is a HUGE amount of extra support code to handle it, spread all through the PHP codebase (introducing TONS of bugs over the years), all for the 0.00001% use case. Not a good trade. Complaining about lack of NUL support would be like complaining that most text editors don't display the DC1 control character. It's just not useful in 99.999% of use cases, so there's no point in supporting it. You'd use a specialized tool instead.
  - Joker_vD 7y ago
    Hey. People wrote actual working SOCKS-proxies in PHP, and given that PHP's low-level functions represents an array of bytes as a string of chars, that'd be impossibly if NUL was forbidden in them.
    - kstenerud 7y ago
      I see... So rather than, say, simply using the binary array data type already present in C, they decided that they needed to subvert strings to support passing non-string data, resulting in tens of thousands of lines of support code to parallel the C runtime library, thousands more lines of bridge code for any outside libraries they might want to make use of, and almost a million lines of PHP implementation code (not including extensions!) carrying this extra cognitive burden in order to support the 1% use case, resulting in countless bugs and security vulnerabilities whenever someone used the C library functions, or stopped on NUL (or not) by mistake. Sounds like a great plan.
ferzul 7y ago
Okay, don't think in terms of function pairs, but in terms of use case. You might want to write unknown UTF16 to a file, and you might want to write UTF16 to a filename. But only one of these is possible. Why? because there's an api for writing contents to a handle that takes a length, but not one for opening a file that does. And you can't “create” api's for what the kernel doesn't support. These are limits on what is possible without forking Linux or some other kernel.