9 ms·

Stdio(3) change: FILE is now opaque

167 points by gslin 1y ago

Netch 1y ago
Slightly side note - I donʼt understand why C++ invents new stdio based functions like std::print but rejects to import funopen() or fopencookie() which make stdio flexible similar to iostreams. As for me it should have been done ≈20 years ago.
p0w3n3d 1y ago
However, who should really rely on internals of FILE? Isn't this a bad practice?
- vitaut 1y ago
  In general, it is a bad practice. However, it can be useful for some low-level libraries. For example, https://github.com/fmtlib/fmt https://github.com/fmtlib/fmt provides a type-safe replacement for `printf` that can write directly to the FILE buffer providing comparable or better performance to native stdio.
  - Retr0id 1y ago
    Doesn't fwrite more or less write directly to the FILE buffer, if buffering is enabled? I'm curious to take a closer look at fmtlib/fmt, which APIs treat FILE as non-opaque? Edit: ah, found some of the magic, I think: https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd6b0b3029a0a8c00/include/fmt/format-inl.h#L1513-L1557 https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd... I'm curious how much speedup is gained from this.
    - vitaut 1y ago
      With fwrite that would be another level of buffering in addition to FILE's buffer. If you are interested in what {fmt} is doing, a good starting point is https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd6b0b3029a0a8c00/include/fmt/format-inl.h#L1514 https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd.... It is also possible to bypass stdio completely and get even faster output (https://vitaut.net/posts/2020/optimal-file-buffer-size/ https://vitaut.net/posts/2020/optimal-file-buffer-size/) and while it is great for files, it may introduce interleaving problems with things like stdout.
- cryptonector 1y ago
  In SunOS 4.x `FILE` was not opaque, and `int fileno(FILE *)` was a macro, not a funciton, and the field of the struct that held the fd number was a `char`. Yeah, that sucked for ages, especially since it bled into the Solaris 2.x 32-bit ABI.
  - bodyfour 1y ago
    Indeed, that was the way it originally worked in all UNIXes: https://github.com/dspinellis/unix-history-repo/blob/Research-V7-Snapshot-Development/usr/include/stdio.h#L26 https://github.com/dspinellis/unix-history-repo/blob/Researc... It was a then-important optimization to do the most common operations with macros since calling a function for every getc()/putc() would have slowed I/O down too much. That's why there is also fgetc()/fputc() -- they're the same as getc()/putc() but they're always defined as functions so calling them generated less code at the callsite at the expense of always requiring a function call. A classic speed-vs-space tradeoff. But, yeah, it was a mistake that it originally used a "char" to store the file descriptor. Back then it was typical to limit processes to 20 open files ( https://github.com/dspinellis/unix-history-repo/blob/Research-V7-Snapshot-Development/usr/sys/h/param.h#L13 https://github.com/dspinellis/unix-history-repo/blob/Researc... ) so a "char" I'm sure felt like plenty.
abnercoimbre 1y ago
Can someone elaborate? I always treated FILE as opaque, but never imagined people could poke into it?
- skissane 1y ago
  One issue – ISO C defines setvbuf (configure buffer mode and size of a stdio stream) but not getvbuf (get current buffer mode and size). On many platforms with non-opaque FILE, you can implement your own getvbuf by peering inside its undocumented fields. I guess part of why it is not in the standard is that it is rarely requested functionality, but there are rare use cases where it may have value. And I think it is an unfortunate lack of orthogonality to have a setter but no corresponding getter. stdio_ext.h offers some functionality like a "getvbuf", but not quite – e.g. __fbufsize tells you a stream's buffer size, and __flbf whether it is line-buffered – but it isn't clear how to distinguish fully buffered and unbuffered streams. And stdio_ext.h has never been standardised, it is an extension invented on Solaris and copied by Linux (and a few other platforms too, e.g. IBM z/OS).
- pjmlp 1y ago
  People use reflection for monkey patching and complain when using compiled languages less supportive of such approaches. So it wouldn't surprise me, that a few folks would do some tricks with FILE internals.
- recipe19 1y ago
  The standard doesn't specify any serviceable parts, and I don't think there are any internals of the struct defined in musl libc on Linux (glibc may be a different story). However, on OpenBSD, it did seem to have some user-visible bits: https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51d539ef06f5f3553790#diff-5e1f2789f2dc0618ffb77ce98c2fab185eaedd0897a0326180f0fb4db794e0b3L99-L131 https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51... If you expose it, someone will probably sooner or later use it, but probably not in any sane / portable code. On the face of it, it doesn't seem like a consequential change, but maybe they're mopping up after some vulnerability in that one weird package that did touch this.
- fweimer 1y ago
somat 1y ago
To misquote the street fighter movie: OpenBSD to Linux: "For you the day you changed your ABI was the most important day in your life, but for me? It was Tuesday" I enjoy the dichotomy between how bad the Linux project is at changing their ABI and how good OpenBSD is at the same task. Where for the most part Linux just decides to live with the bad ABI forever. and if they do decide it actually needs to be changed it is a multi year drama with much crying and missteps. I mean sure, linux has additional considerations that make breaking the ABI very scary for them. the big one is the corpus of closed source software, but being a orders of magnitude bigger project and their overall looser integration does not help any.
- viraptor 1y ago
  This has nothing to do with Linux-the-project. An equivalent change would be in glibc / musl / ...
- ioasuncvinvaer 1y ago
  I think the difference is just the amount of people using the technology.
- MrStonedOne 1y ago
  [dead]
- badsectoracula 1y ago
  > bad the Linux project is at changing their ABI and how good OpenBSD is at the same task From my perspective as a user who wants to have his programs keep working whenever the OS updates and as a programmer who does not want to waste their time playing nanny with broken dependency upgrades for previously working code (working in the sense that it did what it was supposed to do), the Linux project is actually doing the thing the right way and OpenBSD the bad way. It is basically the #1 reason i never considered using OpenBSD. Linux' stance on not breaking backwards compatibility is exactly what i want from an OS. Now if only the userspace libraries weren't so happy to break things too...
loeg 1y ago
I think FreeBSD tried to opaque FILE[1], but it was reverted[2] and still non-opaque in main[3]. [1]: https://github.com/freebsd/freebsd-src/commit/c17bf9a9a5a3b59e03108b785f6b15070ff25651 https://github.com/freebsd/freebsd-src/commit/c17bf9a9a5a3b5... [2]: https://github.com/freebsd/freebsd-src/commit/19e03ca8038019fe0fe7f511b91b758b95b78664 https://github.com/freebsd/freebsd-src/commit/19e03ca8038019... [3]: https://github.com/freebsd/freebsd-src/blob/main/include/stdio.h#L124 https://github.com/freebsd/freebsd-src/blob/main/include/std...
- asveikau 1y ago
  OpenBSD tends to commit to breaking changes much more aggressively than others. Something tells me they're not reverting.
  - loeg 1y ago
    I think FreeBSD is also more concerned with performance regression than OpenBSD is.
- cjensen 1y ago
  FreeBSD's implementation of FILE is a nice object-oriented structure which anyone could derive from. Super-easy to make FILE point to a memory buffer or some other user code. I used that a bunch a long time ago. Obviously making FILE opaque completely breaks every program that used this feature, so no surprise it was reverted.
  - seethedeaduu 1y ago
    fopencookie, fmemopen, you don't need transparency.
    - blueflow 1y ago
      fopencookie seems glibc-specific, so unavailable on BSD.
notepad0x90 1y ago
I don't know if I agree, but this is one shining example of what makes *bsd's great, not being afraid of change. Linux should take note. So much of Windows' headaches stem from not wanting to break things, and needing to support old client code.
- justincormack 1y ago
  There isn't really much of "Linux" here - this code is in libc, so glibc, but that was built from portability, it isn't very Linux specific. Linux doesn't have an all encpmpassing community for userspace.
  - notepad0x90 1y ago
    I see. I thought OpenBSD maintained their own downstream fork of glibc or something since the title/link are for their site/lists. It may not be all encompassing,but I was referring to GNU/Linux. you can swap out bits and pieces, but what mainstream distros include by default, that's what I meant.
    - JdeBP 1y ago
      What you are looking at is not a GNU C library at all. It is a BSD C library.
    - wpollock 1y ago
      I think you are looking for the Linux Standard Base. It started out with a great idea, but the LSB grew so large most popular distros publicly stated they would no longer pursue compliance, so the effort kinda fizzled out.
- sedatk 1y ago
  > So much of Windows' headaches stem from not wanting to break things Quite acceptable for not having the headache for things breaking.
  - 1y ago
cperciva 1y ago
In addition to "some code frobs internals", non-opaque FILE also allows for compatibility with code which puts FILE into a structure, since an opaque FILE doesn't have a size.
- GoblinSlayer 1y ago
  It happened to microsoft a while ago, they changed something in FILE and something else broke, so they went for opaque FILE.
- Someone 1y ago
  But code outside the standard library can’t do that, can it? fopen returns a pointer to a FILE, and you can’t know how a struct FILE should be copied. You can’t just memcpy the bits and then mix calls to fread using pointers to the old and the new FILE struct, for example. I think the standard library need not even support calls using a pointer to a FILE struct it didn’t create.
  - cperciva 1y ago
    You can’t just memcpy the bits... You certainly shouldn't, but sadly this is something which people do.
- dmatech 1y ago
  Are there any POSIX or ISO guarantees on "FILE"? I think it's safe to assume that it isn't an incomplete type, but all functions that use it operate on pointers anyway. Storing a copy of a "FILE" object might result in each copy pointing to the same underlying file handle but having different internal state.
quotemstr 1y ago
CHERI would defend against access to internal data structures without having to bounce between address spaces, FWIW.
- mcculley 1y ago
  Please elaborate.
  - quotemstr 1y ago
    fopen would hand out a FILE* without capabilities to do anything with the resulting data structure, but libc itself could work with it. Libraries would get the same kind of memory protections processes do today.
    - purplesyringa 1y ago
      How would libc get a FILE* pointer with capabilities back from a FILE* passed by the user?
      - quotemstr 1y ago
        libc allocates the FILE* from an array of them or a heap of some sort. It has a private capability on the start of the array and so can recover a full-capability pointer by offsetting its private capability by the distance encoded in the user FILE*. No actual memory access required, I'd think. See https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/202306-plarch-library-based-compartmentalisation.pdf https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/202306...
        dwattttt 1y ago
        This sounds about right. Under CHERI when you're returning a pointer from a function, you can choose to limit its valid dereferencable range, I imagine all the way to 0 (i.e. it can't be dereferenced). When the pointer is passed back into libc, libc can combine the pointer with an internal capability that has the actual size/range of the structure. This isn't _too_ different to having libc just hand out arbitrary integers as FILE; libc has to have some way to map the 'FILE' back to the real structure.
JdeBP 1y ago
If you've ever done this to a C library, the first thing that you'll look at when someone else does it is not the FILE type, but how stdin, stdout, and stderr have changed. The big breaking change is usually the historical implementation of the standard streams as addresses of elements of an array rather than as named pointers. (Plauger's example implementation had them as elements 0, 1, and 2 of a _Files[] array, for example.) It's possible to retain binary compatibility with unrecompiled code that uses the old getc/putc/feof/ferror/fclearerr/&c. macros by preserving structure layouts, but changing stdin, stdout, and stderr can make things not link. And indeed that has happened here.
- juped 1y ago
  openbsd has never, ever, ever, even once maintained binary compatibility; this is just a warning about source compatibility breaking
  - JdeBP 1y ago
    You have this backwards. Source compatibility is not broken by such a change, with zero source changes required in order to accommodate it for almost all applications; and far from OpenBSD "never, ever, ever" maintaining binary compatibility Masahiko Yasuoka and Philip Guenther took deliberate steps in this very case to ensure as much binary compatibility as they could, retaining structure layouts as-is and retaining several symbols for library internals that macros used to reference, even those that won't be used by freshly recompiled applications. The warning, and the bumping of several shared library major version numbers, is most definitely about the standard streams breaking binary, not source as you have it, compatibility. Any newly compiled binary that is using the C standard streams won't run on old shared libraries because of the new symbol references for __stdin, __stdout, and __stderr.
xpressvideoz 1y ago
So many words in the commit message and the announcement article, yet not a single mention of the rationale? I have a bad feeling about their practice.
Cadwhisker 1y ago
The best comments always say "why" and that's missing here. Does anyone know why this change was done? Security reasons? Preparing for future changes?
severino 1y ago
Out of curiosity. If we have this inside `bits/types/FILE.h` in Linux/GNU, does it mean the type is opaque? ... struct _IO_FILE; /* The opaque type of streams. This is the definition used elsewhere. */ typedef struct _IO_FILE FILE; ...
- uncircle 1y ago
  Unless struct _IO_FILE is expanded elsewhere, yes, it's opaque and can only be interacted through a pointer, as it would otherwise be an unsized object.
  - severino 1y ago
    The definition of struct _IO_FILE seems to be inside `bits/types/struct_FILE.h`, and goes like this: struct _IO_FILE { int _flags; ... so I suppose this means it's indeed expanded somewhere and, thus, not opaque?
    - clappski 1y ago
      When we're talking about opaque it's really in relation to an individual translation unit - somewhere in the binary or its linked libraries the definition has to exist for the code that uses the opaque type.
      - severino 1y ago
        Forgive my ignorance in this topic, but if "stdio.h" itself includes "bits/types/struct_FILE.h", is anything preventing me from accessing the individual elements of FILE as they are defined in the latter header file?
        yrro 1y ago
        It looks like FILE is not opaque in glibc. Create a translation unit that includes <stdio.h> & declares a FILE variable and it compiles fine. For comparison, create a translation unit that declares your own struct (but does not provide a definition) and declares a variable of the same type, and you'll get a "storage size of 'x' isn't known" error when compiling.
EmanueleAina 1y ago
Hold on. CVS?

167 points by gslin 1y ago

Slightly side note - I donʼt understand why C++ invents new stdio based functions like std::print but rejects to import funopen() or fopencookie() which make stdio flexible similar to iostreams. As for me it should have been done ≈20 years ago.

p0w3n3d 1y ago

However, who should really rely on internals of FILE? Isn't this a bad practice?

vitaut 1y ago
In general, it is a bad practice. However, it can be useful for some low-level libraries. For example, https://github.com/fmtlib/fmt https://github.com/fmtlib/fmt provides a type-safe replacement for `printf` that can write directly to the FILE buffer providing comparable or better performance to native stdio.
- Retr0id 1y ago
  Doesn't fwrite more or less write directly to the FILE buffer, if buffering is enabled? I'm curious to take a closer look at fmtlib/fmt, which APIs treat FILE as non-opaque? Edit: ah, found some of the magic, I think: https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd6b0b3029a0a8c00/include/fmt/format-inl.h#L1513-L1557 https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd... I'm curious how much speedup is gained from this.
  - vitaut 1y ago
    With fwrite that would be another level of buffering in addition to FILE's buffer. If you are interested in what {fmt} is doing, a good starting point is https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd6b0b3029a0a8c00/include/fmt/format-inl.h#L1514 https://github.com/fmtlib/fmt/blob/35dcc58263d6b55419a5932bd.... It is also possible to bypass stdio completely and get even faster output (https://vitaut.net/posts/2020/optimal-file-buffer-size/ https://vitaut.net/posts/2020/optimal-file-buffer-size/) and while it is great for files, it may introduce interleaving problems with things like stdout.
cryptonector 1y ago
In SunOS 4.x `FILE` was not opaque, and `int fileno(FILE *)` was a macro, not a funciton, and the field of the struct that held the fd number was a `char`. Yeah, that sucked for ages, especially since it bled into the Solaris 2.x 32-bit ABI.
- bodyfour 1y ago
  Indeed, that was the way it originally worked in all UNIXes: https://github.com/dspinellis/unix-history-repo/blob/Research-V7-Snapshot-Development/usr/include/stdio.h#L26 https://github.com/dspinellis/unix-history-repo/blob/Researc... It was a then-important optimization to do the most common operations with macros since calling a function for every getc()/putc() would have slowed I/O down too much. That's why there is also fgetc()/fputc() -- they're the same as getc()/putc() but they're always defined as functions so calling them generated less code at the callsite at the expense of always requiring a function call. A classic speed-vs-space tradeoff. But, yeah, it was a mistake that it originally used a "char" to store the file descriptor. Back then it was typical to limit processes to 20 open files ( https://github.com/dspinellis/unix-history-repo/blob/Research-V7-Snapshot-Development/usr/sys/h/param.h#L13 https://github.com/dspinellis/unix-history-repo/blob/Researc... ) so a "char" I'm sure felt like plenty.

abnercoimbre 1y ago

Can someone elaborate? I always treated FILE as opaque, but never imagined people could poke into it?

skissane 1y ago
One issue – ISO C defines setvbuf (configure buffer mode and size of a stdio stream) but not getvbuf (get current buffer mode and size). On many platforms with non-opaque FILE, you can implement your own getvbuf by peering inside its undocumented fields. I guess part of why it is not in the standard is that it is rarely requested functionality, but there are rare use cases where it may have value. And I think it is an unfortunate lack of orthogonality to have a setter but no corresponding getter. stdio_ext.h offers some functionality like a "getvbuf", but not quite – e.g. __fbufsize tells you a stream's buffer size, and __flbf whether it is line-buffered – but it isn't clear how to distinguish fully buffered and unbuffered streams. And stdio_ext.h has never been standardised, it is an extension invented on Solaris and copied by Linux (and a few other platforms too, e.g. IBM z/OS).
pjmlp 1y ago
People use reflection for monkey patching and complain when using compiled languages less supportive of such approaches. So it wouldn't surprise me, that a few folks would do some tricks with FILE internals.
recipe19 1y ago
The standard doesn't specify any serviceable parts, and I don't think there are any internals of the struct defined in musl libc on Linux (glibc may be a different story). However, on OpenBSD, it did seem to have some user-visible bits: https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51d539ef06f5f3553790#diff-5e1f2789f2dc0618ffb77ce98c2fab185eaedd0897a0326180f0fb4db794e0b3L99-L131 https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51... If you expose it, someone will probably sooner or later use it, but probably not in any sane / portable code. On the face of it, it doesn't seem like a consequential change, but maybe they're mopping up after some vulnerability in that one weird package that did touch this.
fweimer 1y ago

somat 1y ago

To misquote the street fighter movie: OpenBSD to Linux: "For you the day you changed your ABI was the most important day in your life, but for me? It was Tuesday" I enjoy the dichotomy between how bad the Linux project is at changing their ABI and how good OpenBSD is at the same task. Where for the most part Linux just decides to live with the bad ABI forever. and if they do decide it actually needs to be changed it is a multi year drama with much crying and missteps. I mean sure, linux has additional considerations that make breaking the ABI very scary for them. the big one is the corpus of closed source software, but being a orders of magnitude bigger project and their overall looser integration does not help any.

viraptor 1y ago
This has nothing to do with Linux-the-project. An equivalent change would be in glibc / musl / ...
ioasuncvinvaer 1y ago
I think the difference is just the amount of people using the technology.
MrStonedOne 1y ago
[dead]
badsectoracula 1y ago
> bad the Linux project is at changing their ABI and how good OpenBSD is at the same task From my perspective as a user who wants to have his programs keep working whenever the OS updates and as a programmer who does not want to waste their time playing nanny with broken dependency upgrades for previously working code (working in the sense that it did what it was supposed to do), the Linux project is actually doing the thing the right way and OpenBSD the bad way. It is basically the #1 reason i never considered using OpenBSD. Linux' stance on not breaking backwards compatibility is exactly what i want from an OS. Now if only the userspace libraries weren't so happy to break things too...

loeg 1y ago

I think FreeBSD tried to opaque FILE[1], but it was reverted[2] and still non-opaque in main[3]. [1]: https://github.com/freebsd/freebsd-src/commit/c17bf9a9a5a3b59e03108b785f6b15070ff25651 https://github.com/freebsd/freebsd-src/commit/c17bf9a9a5a3b5... [2]: https://github.com/freebsd/freebsd-src/commit/19e03ca8038019fe0fe7f511b91b758b95b78664 https://github.com/freebsd/freebsd-src/commit/19e03ca8038019... [3]: https://github.com/freebsd/freebsd-src/blob/main/include/stdio.h#L124 https://github.com/freebsd/freebsd-src/blob/main/include/std...

asveikau 1y ago
OpenBSD tends to commit to breaking changes much more aggressively than others. Something tells me they're not reverting.
- loeg 1y ago
  I think FreeBSD is also more concerned with performance regression than OpenBSD is.
cjensen 1y ago
FreeBSD's implementation of FILE is a nice object-oriented structure which anyone could derive from. Super-easy to make FILE point to a memory buffer or some other user code. I used that a bunch a long time ago. Obviously making FILE opaque completely breaks every program that used this feature, so no surprise it was reverted.
- seethedeaduu 1y ago
  fopencookie, fmemopen, you don't need transparency.
  - blueflow 1y ago
    fopencookie seems glibc-specific, so unavailable on BSD.

notepad0x90 1y ago

I don't know if I agree, but this is one shining example of what makes *bsd's great, not being afraid of change. Linux should take note. So much of Windows' headaches stem from not wanting to break things, and needing to support old client code.

justincormack 1y ago
There isn't really much of "Linux" here - this code is in libc, so glibc, but that was built from portability, it isn't very Linux specific. Linux doesn't have an all encpmpassing community for userspace.
- notepad0x90 1y ago
  I see. I thought OpenBSD maintained their own downstream fork of glibc or something since the title/link are for their site/lists. It may not be all encompassing,but I was referring to GNU/Linux. you can swap out bits and pieces, but what mainstream distros include by default, that's what I meant.
  - JdeBP 1y ago
    What you are looking at is not a GNU C library at all. It is a BSD C library.
  - wpollock 1y ago
    I think you are looking for the Linux Standard Base. It started out with a great idea, but the LSB grew so large most popular distros publicly stated they would no longer pursue compliance, so the effort kinda fizzled out.
sedatk 1y ago
> So much of Windows' headaches stem from not wanting to break things Quite acceptable for not having the headache for things breaking.
- 1y ago

cperciva 1y ago

In addition to "some code frobs internals", non-opaque FILE also allows for compatibility with code which puts FILE into a structure, since an opaque FILE doesn't have a size.

GoblinSlayer 1y ago
It happened to microsoft a while ago, they changed something in FILE and something else broke, so they went for opaque FILE.
Someone 1y ago
But code outside the standard library can’t do that, can it? fopen returns a pointer to a FILE, and you can’t know how a struct FILE should be copied. You can’t just memcpy the bits and then mix calls to fread using pointers to the old and the new FILE struct, for example. I think the standard library need not even support calls using a pointer to a FILE struct it didn’t create.
- cperciva 1y ago
  You can’t just memcpy the bits... You certainly shouldn't, but sadly this is something which people do.
dmatech 1y ago
Are there any POSIX or ISO guarantees on "FILE"? I think it's safe to assume that it isn't an incomplete type, but all functions that use it operate on pointers anyway. Storing a copy of a "FILE" object might result in each copy pointing to the same underlying file handle but having different internal state.

quotemstr 1y ago

CHERI would defend against access to internal data structures without having to bounce between address spaces, FWIW.

mcculley 1y ago
Please elaborate.
- quotemstr 1y ago
  fopen would hand out a FILE* without capabilities to do anything with the resulting data structure, but libc itself could work with it. Libraries would get the same kind of memory protections processes do today.
  - purplesyringa 1y ago
    How would libc get a FILE* pointer with capabilities back from a FILE* passed by the user?
    - quotemstr 1y ago
      libc allocates the FILE* from an array of them or a heap of some sort. It has a private capability on the start of the array and so can recover a full-capability pointer by offsetting its private capability by the distance encoded in the user FILE*. No actual memory access required, I'd think. See https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/202306-plarch-library-based-compartmentalisation.pdf https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/202306...
      - dwattttt 1y ago
        This sounds about right. Under CHERI when you're returning a pointer from a function, you can choose to limit its valid dereferencable range, I imagine all the way to 0 (i.e. it can't be dereferenced). When the pointer is passed back into libc, libc can combine the pointer with an internal capability that has the actual size/range of the structure. This isn't _too_ different to having libc just hand out arbitrary integers as FILE; libc has to have some way to map the 'FILE' back to the real structure.

JdeBP 1y ago

If you've ever done this to a C library, the first thing that you'll look at when someone else does it is not the FILE type, but how stdin, stdout, and stderr have changed. The big breaking change is usually the historical implementation of the standard streams as addresses of elements of an array rather than as named pointers. (Plauger's example implementation had them as elements 0, 1, and 2 of a _Files[] array, for example.) It's possible to retain binary compatibility with unrecompiled code that uses the old getc/putc/feof/ferror/fclearerr/&c. macros by preserving structure layouts, but changing stdin, stdout, and stderr can make things not link. And indeed that has happened here.

juped 1y ago
openbsd has never, ever, ever, even once maintained binary compatibility; this is just a warning about source compatibility breaking
- JdeBP 1y ago
  You have this backwards. Source compatibility is not broken by such a change, with zero source changes required in order to accommodate it for almost all applications; and far from OpenBSD "never, ever, ever" maintaining binary compatibility Masahiko Yasuoka and Philip Guenther took deliberate steps in this very case to ensure as much binary compatibility as they could, retaining structure layouts as-is and retaining several symbols for library internals that macros used to reference, even those that won't be used by freshly recompiled applications. The warning, and the bumping of several shared library major version numbers, is most definitely about the standard streams breaking binary, not source as you have it, compatibility. Any newly compiled binary that is using the C standard streams won't run on old shared libraries because of the new symbol references for __stdin, __stdout, and __stderr.

xpressvideoz 1y ago

So many words in the commit message and the announcement article, yet not a single mention of the rationale? I have a bad feeling about their practice.

Cadwhisker 1y ago

The best comments always say "why" and that's missing here. Does anyone know why this change was done? Security reasons? Preparing for future changes?

severino 1y ago

Out of curiosity. If we have this inside `bits/types/FILE.h` in Linux/GNU, does it mean the type is opaque? ... struct _IO_FILE; /* The opaque type of streams. This is the definition used elsewhere. */ typedef struct _IO_FILE FILE; ...

uncircle 1y ago
Unless struct _IO_FILE is expanded elsewhere, yes, it's opaque and can only be interacted through a pointer, as it would otherwise be an unsized object.
- severino 1y ago
  The definition of struct _IO_FILE seems to be inside `bits/types/struct_FILE.h`, and goes like this: struct _IO_FILE { int _flags; ... so I suppose this means it's indeed expanded somewhere and, thus, not opaque?
  - clappski 1y ago
    When we're talking about opaque it's really in relation to an individual translation unit - somewhere in the binary or its linked libraries the definition has to exist for the code that uses the opaque type.
    - severino 1y ago
      Forgive my ignorance in this topic, but if "stdio.h" itself includes "bits/types/struct_FILE.h", is anything preventing me from accessing the individual elements of FILE as they are defined in the latter header file?
      - yrro 1y ago
        It looks like FILE is not opaque in glibc. Create a translation unit that includes <stdio.h> & declares a FILE variable and it compiles fine. For comparison, create a translation unit that declares your own struct (but does not provide a definition) and declares a variable of the same type, and you'll get a "storage size of 'x' isn't known" error when compiling.

EmanueleAina 1y ago

Hold on. CVS?