8 ms·

How do I inform Windows that I'm writing a binary file?

73 points by ingve 5mo ago

saltyoldman 5mo ago
fopen(..., "wb") ?
- qbane 5mo ago
  It's C library taking care of the "b" part for you according to the article.
  - wahern 5mo ago
    It's the other way around. It's the C runtime that treats text ("t") mode differently, because the C standard specifies \n as a line delimiter but the Windows convention is \r\n. In text mode C stdio translates between \n and \r\n. In binary mode it does no translation.
    - qbane 5mo ago
      Note that when neither is supplied, the text mode is the default. This is why I said that it is the C library handling the "b" flag.
lmm 5mo ago
The article seems to be taking the position that the C runtime library is not part of "Windows", which feels like a rather odd view to me. What is the stable API that Windows offers to application developers if not that?
- ChrisSD 5mo ago
  The Win32 API. E.g. using WriteFile to write files (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...) It wasn't until fairly recently that the C runtime was stably shipped with Windows. Previously you had to install the correct version of the C library alongside your application.
  - lmm 5mo ago
    > The Win32 API. E.g. using WriteFile to write files (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/ https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...) Which is called from what, if not C? Does windows really offer no API for writing text (rather than bytes) to files? Or does it rely on the application developer to manage line endings in their own code? Neither of those sounds very developer-friendly.
    - p_l 5mo ago
      The whole issue is specific to C and languages that copied C or use its runtime underneath in implementations (like Python) For reference, Unix has no API other than bytes either.
      - lmm 5mo ago
        > The whole issue is specific to C and languages that copied C or use its runtime underneath in implementations (like Python) So it's "specific to" almost all programming languages in actual use. That's a rather esoteric point. > For reference, Unix has no API other than bytes either. Unix does offer an API for writing C-standard in-memory text strings to Unix-standard on-disk text files, it just happens to be the same one as the API for writing in-memory binary strings to on-disk binary files.
        Joker_vD 5mo ago
        > Unix does offer an API for writing C-standard in-memory text strings Why on bloody Earth should a presumably generic-purpose OS provide a special API for dealing with internal representation of some data structure in a (particular) implementation of a (particular) programming language? Besides, it doesn't offer such an API anyhow; you need to take care to manually pass the result of a strlen() call instead of sizeof()'s as the value for the len parameter of a write() call, otherwise a NUL-terminator will get written into the file as well. And C says nothing about what constitutes a line break, by the way. Nor does it have any concept of a "line", or any utilities for working with lines specifically, it only knows of strings, and that's all. The concept of "text line" is POSIX.
- p_l 5mo ago
  Indeed, C runtime is not part of windows API, and it's normal to have a program include few different copies of C runtime library due to different modules compiled with different compilers/options. C runtime library being part of OS is accidental thing in Unix, 16bit and 32bit Windows API even does not use C-compatible ABI (instead, Pascal-compatible one is present)
  - deleted 5mo ago
    [deleted]
- deleted 5mo ago
  [deleted]
- jcranmer 5mo ago
  There is a very unfortunate situation in Unix systems in that the library named 'libc' is serving several simultaneous different roles. One of those roles--what it is named for--is serving as the C standard library. The more important role is that the library also provides the implementation of a different standard API, the POSIX API, which is the main API used to access system details. There's also yet another role of providing the stable system interface to the kernel in most Unix implementations. On Windows, these roles are provided by different libraries: ucrt (what used to be msvcrt), kernel32, and ntdll, respectively. And for what it's worth, the actual C standard library tends to be fairly rarely used, especially if you consider the malloc/free interface to be part of the system library rather than the C standard library. The C stdio functionality, for example, is extremely underpowered compared to the capabilities of all major operating systems' I/O libraries, and so most applications--even those written entirely in C--will choose to avoid the C standard library and instead use the more direct primitives of the system API layer instead.
  - wjholden 5mo ago
    Not OP, but thank you for your sharing this. If you don't mind a follow-on question, I always hear people talk about the "runtime" in languages like Go and libraries like Tokio. What is that these runtimes are doing that you cannot get from the likes of libc and these Windows DLLs?
    - JdeBP 5mo ago
      MSVCRT, the Microsoft Visual C/C++ Runtime library, is also 'the runtime library'. It was the runtime library for Microsoft's C/C++ compiler. In the days when there were multiple C/C++ implementations for Win32 (which still exist, if one is willing to dig up Watcom C/C++ or some such) there would be different runtime library DLLs for the different C/C++ vendors, even for different versions of their products. Runtime libraries for C/C++ provide two general sets of stuff: the stuff mandated for the Standard C and Standard C++ libraries, and the stuff that is needed by the basic mechanics of the language. The former is everything from abort() to wscanf(). The latter is a bunch of internal functions, calls to which the compiler inserts in order to do stuff. This is basically the split nowadays between UCRT and VCRUNTIME. In the days of programming targetting the 80486SX without an 80487 present, for instance, every piece of floating point arithmetic was not a machine instruction but a call to a runtime library routine that did the floating point operation longhand using non-FPU instructions. Other runtime functionality over the years has included doing 32-bit or 64-bit integer arithmetic on 16-bit and 32-bit architectures where this was not a native word size, functions to do stack checking in the function perilogue, and functions to do C++ run-type type checking and exception processing. This pattern is followed by other (compiled) programming languages. Naturally, the programming languages do not necessarily have any relation to the Standard C or Standard C++ libraries, nor do they generate code that needs the same helper functions for stuff as C/C++ code does. (But the situation is complicated by the POSIX API and the old C language bindings for the MS-DOS system call API, some of which another programming language might also allow program code to use.)
- pjmlp 5mo ago
  Win32. The C standard library is definitely not part of Windows. It is now with the Universal C runtime, introduced in Windows 10, which is ironically written in C++ with extern "C" { ... } On non UNIX clones, including Windows, it has always been the role of commercial C compilers to provide the C standard library on top of the actual C APIs.
Isamu 5mo ago
Going all the way back to the earliest C compilers on DOS. There was a decision made to make “\n” just work on DOS for portability of Unix programs, and to make the examples from the C programming book just work. But in Unix “\n” is a single byte, and in DOS it is 2. So they introduced text and binary modes for files on DOS. Behind the scenes the library will handle the extra byte. This is not necessary in Unix. I used to have to be careful about importing files to DOS. Did the file come from Unix?
- criddell 5mo ago
  Linefeed (\n) is a single byte in DOS as well. I think you are talking about carriage return linefeed pair (CRLF or \r\n), These control codes go back to line printers. Linefeed advances the paper one line and carriage return moves the print head to the left.
  - Isamu 5mo ago
    >Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.
    - foresto 5mo ago
      I'm pretty sure that conversion was done by the C library, just as stated in the article. Not by DOS. ASCII 0x0A '\n' is always one byte*, and C library implementations for DOS would insert an ASCII 0x0D '\r' byte before it at output time if the C FILE stream had been opened in text mode. Note that printf(), which you use in your example, is a C library function that writes writes to a predefined text mode stream. So it follows the same rules. I wasn't able to dig up the source code of a vintage DOS compiler's C library in a few minutes of looking, so I can't prove it right now, but this section of the C standard (7.21.2 - Streams) hints that my recollection is correct: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=316 https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#p... *(On systems where the char type is one byte, of course, which is the case for DOS C compilers.)
wpollock 5mo ago
Interestingly, the IETF has several published RFCs for text protocols, all of which require \r\n line endings. <https://www.rfc-editor.org/old/EOLstory.txt https://www.rfc-editor.org/old/EOLstory.txt> Note this does not apply to file formats (except for RFCs).
nar001 5mo ago
I'd like to know how these "customers" even get into contact with actual people or engineers at MS, because this seems completely impossible now

73 points by ingve 5mo ago

saltyoldman 5mo ago

fopen(..., "wb") ?

qbane 5mo ago
It's C library taking care of the "b" part for you according to the article.
- wahern 5mo ago
  It's the other way around. It's the C runtime that treats text ("t") mode differently, because the C standard specifies \n as a line delimiter but the Windows convention is \r\n. In text mode C stdio translates between \n and \r\n. In binary mode it does no translation.
  - qbane 5mo ago
    Note that when neither is supplied, the text mode is the default. This is why I said that it is the C library handling the "b" flag.

lmm 5mo ago

The article seems to be taking the position that the C runtime library is not part of "Windows", which feels like a rather odd view to me. What is the stable API that Windows offers to application developers if not that?

ChrisSD 5mo ago
The Win32 API. E.g. using WriteFile to write files (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...) It wasn't until fairly recently that the C runtime was stably shipped with Windows. Previously you had to install the correct version of the C library alongside your application.
- lmm 5mo ago
  > The Win32 API. E.g. using WriteFile to write files (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/ https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...) Which is called from what, if not C? Does windows really offer no API for writing text (rather than bytes) to files? Or does it rely on the application developer to manage line endings in their own code? Neither of those sounds very developer-friendly.
  - p_l 5mo ago
    The whole issue is specific to C and languages that copied C or use its runtime underneath in implementations (like Python) For reference, Unix has no API other than bytes either.
    - lmm 5mo ago
      > The whole issue is specific to C and languages that copied C or use its runtime underneath in implementations (like Python) So it's "specific to" almost all programming languages in actual use. That's a rather esoteric point. > For reference, Unix has no API other than bytes either. Unix does offer an API for writing C-standard in-memory text strings to Unix-standard on-disk text files, it just happens to be the same one as the API for writing in-memory binary strings to on-disk binary files.
      - Joker_vD 5mo ago
        > Unix does offer an API for writing C-standard in-memory text strings Why on bloody Earth should a presumably generic-purpose OS provide a special API for dealing with internal representation of some data structure in a (particular) implementation of a (particular) programming language? Besides, it doesn't offer such an API anyhow; you need to take care to manually pass the result of a strlen() call instead of sizeof()'s as the value for the len parameter of a write() call, otherwise a NUL-terminator will get written into the file as well. And C says nothing about what constitutes a line break, by the way. Nor does it have any concept of a "line", or any utilities for working with lines specifically, it only knows of strings, and that's all. The concept of "text line" is POSIX.
p_l 5mo ago
Indeed, C runtime is not part of windows API, and it's normal to have a program include few different copies of C runtime library due to different modules compiled with different compilers/options. C runtime library being part of OS is accidental thing in Unix, 16bit and 32bit Windows API even does not use C-compatible ABI (instead, Pascal-compatible one is present)
- deleted 5mo ago
  [deleted]
deleted 5mo ago
[deleted]
jcranmer 5mo ago
There is a very unfortunate situation in Unix systems in that the library named 'libc' is serving several simultaneous different roles. One of those roles--what it is named for--is serving as the C standard library. The more important role is that the library also provides the implementation of a different standard API, the POSIX API, which is the main API used to access system details. There's also yet another role of providing the stable system interface to the kernel in most Unix implementations. On Windows, these roles are provided by different libraries: ucrt (what used to be msvcrt), kernel32, and ntdll, respectively. And for what it's worth, the actual C standard library tends to be fairly rarely used, especially if you consider the malloc/free interface to be part of the system library rather than the C standard library. The C stdio functionality, for example, is extremely underpowered compared to the capabilities of all major operating systems' I/O libraries, and so most applications--even those written entirely in C--will choose to avoid the C standard library and instead use the more direct primitives of the system API layer instead.
- wjholden 5mo ago
  Not OP, but thank you for your sharing this. If you don't mind a follow-on question, I always hear people talk about the "runtime" in languages like Go and libraries like Tokio. What is that these runtimes are doing that you cannot get from the likes of libc and these Windows DLLs?
  - JdeBP 5mo ago
    MSVCRT, the Microsoft Visual C/C++ Runtime library, is also 'the runtime library'. It was the runtime library for Microsoft's C/C++ compiler. In the days when there were multiple C/C++ implementations for Win32 (which still exist, if one is willing to dig up Watcom C/C++ or some such) there would be different runtime library DLLs for the different C/C++ vendors, even for different versions of their products. Runtime libraries for C/C++ provide two general sets of stuff: the stuff mandated for the Standard C and Standard C++ libraries, and the stuff that is needed by the basic mechanics of the language. The former is everything from abort() to wscanf(). The latter is a bunch of internal functions, calls to which the compiler inserts in order to do stuff. This is basically the split nowadays between UCRT and VCRUNTIME. In the days of programming targetting the 80486SX without an 80487 present, for instance, every piece of floating point arithmetic was not a machine instruction but a call to a runtime library routine that did the floating point operation longhand using non-FPU instructions. Other runtime functionality over the years has included doing 32-bit or 64-bit integer arithmetic on 16-bit and 32-bit architectures where this was not a native word size, functions to do stack checking in the function perilogue, and functions to do C++ run-type type checking and exception processing. This pattern is followed by other (compiled) programming languages. Naturally, the programming languages do not necessarily have any relation to the Standard C or Standard C++ libraries, nor do they generate code that needs the same helper functions for stuff as C/C++ code does. (But the situation is complicated by the POSIX API and the old C language bindings for the MS-DOS system call API, some of which another programming language might also allow program code to use.)
pjmlp 5mo ago
Win32. The C standard library is definitely not part of Windows. It is now with the Universal C runtime, introduced in Windows 10, which is ironically written in C++ with extern "C" { ... } On non UNIX clones, including Windows, it has always been the role of commercial C compilers to provide the C standard library on top of the actual C APIs.

Isamu 5mo ago

Going all the way back to the earliest C compilers on DOS. There was a decision made to make “\n” just work on DOS for portability of Unix programs, and to make the examples from the C programming book just work. But in Unix “\n” is a single byte, and in DOS it is 2. So they introduced text and binary modes for files on DOS. Behind the scenes the library will handle the extra byte. This is not necessary in Unix. I used to have to be careful about importing files to DOS. Did the file come from Unix?

criddell 5mo ago
Linefeed (\n) is a single byte in DOS as well. I think you are talking about carriage return linefeed pair (CRLF or \r\n), These control codes go back to line printers. Linefeed advances the paper one line and carriage return moves the print head to the left.
- Isamu 5mo ago
  >Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.
  - foresto 5mo ago
    I'm pretty sure that conversion was done by the C library, just as stated in the article. Not by DOS. ASCII 0x0A '\n' is always one byte*, and C library implementations for DOS would insert an ASCII 0x0D '\r' byte before it at output time if the C FILE stream had been opened in text mode. Note that printf(), which you use in your example, is a C library function that writes writes to a predefined text mode stream. So it follows the same rules. I wasn't able to dig up the source code of a vintage DOS compiler's C library in a few minutes of looking, so I can't prove it right now, but this section of the C standard (7.21.2 - Streams) hints that my recollection is correct: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=316 https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#p... *(On systems where the char type is one byte, of course, which is the case for DOS C compilers.)

wpollock 5mo ago

Interestingly, the IETF has several published RFCs for text protocols, all of which require \r\n line endings. <https://www.rfc-editor.org/old/EOLstory.txt https://www.rfc-editor.org/old/EOLstory.txt> Note this does not apply to file formats (except for RFCs).

nar001 5mo ago

I'd like to know how these "customers" even get into contact with actual people or engineers at MS, because this seems completely impossible now