5 ms·

Can someone elaborate? I always treated FILE as opaque, but never imagined people could poke into it?

skissane 1y ago
One issue – ISO C defines setvbuf (configure buffer mode and size of a stdio stream) but not getvbuf (get current buffer mode and size). On many platforms with non-opaque FILE, you can implement your own getvbuf by peering inside its undocumented fields. I guess part of why it is not in the standard is that it is rarely requested functionality, but there are rare use cases where it may have value. And I think it is an unfortunate lack of orthogonality to have a setter but no corresponding getter. stdio_ext.h offers some functionality like a "getvbuf", but not quite – e.g. __fbufsize tells you a stream's buffer size, and __flbf whether it is line-buffered – but it isn't clear how to distinguish fully buffered and unbuffered streams. And stdio_ext.h has never been standardised, it is an extension invented on Solaris and copied by Linux (and a few other platforms too, e.g. IBM z/OS).
pjmlp 1y ago
People use reflection for monkey patching and complain when using compiled languages less supportive of such approaches. So it wouldn't surprise me, that a few folks would do some tricks with FILE internals.
recipe19 1y ago
The standard doesn't specify any serviceable parts, and I don't think there are any internals of the struct defined in musl libc on Linux (glibc may be a different story). However, on OpenBSD, it did seem to have some user-visible bits: https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51d539ef06f5f3553790#diff-5e1f2789f2dc0618ffb77ce98c2fab185eaedd0897a0326180f0fb4db794e0b3L99-L131 https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51... If you expose it, someone will probably sooner or later use it, but probably not in any sane / portable code. On the face of it, it doesn't seem like a consequential change, but maybe they're mopping up after some vulnerability in that one weird package that did touch this.
fweimer 1y ago
In gnulib, there is code that patches FILE internals for various platforms to modify behavior of <stdio.h> functions, or implement new functionality. https://cgit.git.savannah.gnu.org/cgit/gnulib.git/tree/lib/stdio-impl.h https://cgit.git.savannah.gnu.org/cgit/gnulib.git/tree/lib/s... Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported).
- collinfunk 1y ago
  Yes, that part of Gnulib has caused some problems previously. It is mostly used to implement <stdio_ext.h> functions on non-glibc systems. However, it is also needed for some buggy implementations of ftello, fseeko, and fflush. P.S. Hi Florian :)
  - collinfunk 1y ago
    And now updated for this change. https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=69a02b5307df043ca22c4b75eaec12be21493910 https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=69a0...
- quotemstr 1y ago
  > Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported). Hyrum's law strikes again. People cast dl_info and poke at internal bits all the time too. glibc and others should be using kernel-style compiler-driven struct layout randomization to fight it.
  - jancsika 1y ago
    > Hyrum's law strikes again. Is there a name for APIs that are drawn directly from some subset of observed behaviors? Like Crockford going, "Hey, there's a nice little data format buried in these JS objects. Schloink"
ksherlock 1y ago
*BSD stdio.h used to include macro versions of some stdio functions (feof, ferror, clearerr, fileno, getc, putc) so they would be inlined. /* * This has been tuned to generate reasonable code on the vax using pcc. */*
pm215 1y ago
The MH and nmh mail clients used to directly look into FILE internals. If you look for LINUX_STDIO in this old version of the relevant file you can see the kind of ugliness that resulted: https://cgit.git.savannah.gnu.org/cgit/nmh.git/tree/sbr/m_getfld.c?id=ffbc91b2b571f18eb8c1ed3a90e76e5d0ad65bd1 https://cgit.git.savannah.gnu.org/cgit/nmh.git/tree/sbr/m_ge... It's basically searching an email file to find the contents of either a given header or the mail body. These days there is no need to go under the hood of libc for this (and this code got ripped out over a decade ago), but back when the mail client was running on elderly VAXen this ate up significant time. Sneaking in and reading directly from the internal stdio buffer lets you avoid copying all the data the way an fread would. The same function also used to have a bit of inline vax assembly for string searching... The only reason this "works" is that traditionally the FILE struct is declared in a public header so libc can have some of its own functions implemented as macros for speed, and that there was not (when this hack was originally put in in the 1980s) yet much divergence in libc implementations.
loeg 1y ago
Historically some FILE designs exposed the structure somewhere so that some of the f* methods could be implemented as macros or inline functions (e.g., `fileno()`).
- abnercoimbre 1y ago
  I guess these things usually boil down to performance concerns right?
bitwize 1y ago
Hyrum's Law applies: the API of any software component is the entire exposed surface, not just what you've documented. Hence, if you have FILE well-defined somewhere in a programmer-accessible header, somebody somewhere can and will poke at the internal bits in order to achieve some hack or optimization.
- krylon 1y ago
  OTOH, yes. OTOH, when coding, I consider FILE to be effectively opaque in the sense that it probably is not portable, and that the implementers might change it at any time. I am reminded of this fine article by Raymond Chen, which covers a similar situation on Windows way back when: https://devblogs.microsoft.com/oldnewthing/20031015-00/?p=42163 https://devblogs.microsoft.com/oldnewthing/20031015-00/?p=42...
  - brokencode 1y ago
    Yes, it would not be sane to depend on implementation details of something like this. But the sad reality is that many developers (myself included earlier in my career) will do insane things to fix a critical bug or performance problem when faced with a tight deadline.
    - krylon 1y ago
      Such is life, yes.
- ars 1y ago
  > to achieve some hack or optimization. Or functionality. Happens to me all the time I have some Java class that's marked Final, so instead of just extending the class and moving on, I have to copy/paste the entire class wholesale to accomplish my goal. Personally I hate "nanny" languages that block you from accessing things. It's my computer, and my code, and my compiler. Please don't do things "for my own good", I can decide that for myself. (And yes, I am aware of the argument that this lets the original programmer change the internals, in practice it's not such a big problem. Or the cure is worse than the problem - for example my copy/paste example.) Another example is a private constant. Instead of allowing me to reference it, I have to copy it. How is that any better? If the programmer has to change how the constant works then they can do so, and at that point my code will break and I'll .... copy the constant. But until then I can just use the constant.
zahlman 1y ago
I always assumed that people could poke into it, but shuddered at the thought.
asveikau 1y ago
I've seen old code do this over the years. When you consider for example that snprintf() didn't used to be standardized until the late 1990s. People would mock up a fake FILE* and use fprintf.

One issue – ISO C defines setvbuf (configure buffer mode and size of a stdio stream) but not getvbuf (get current buffer mode and size). On many platforms with non-opaque FILE, you can implement your own getvbuf by peering inside its undocumented fields. I guess part of why it is not in the standard is that it is rarely requested functionality, but there are rare use cases where it may have value. And I think it is an unfortunate lack of orthogonality to have a setter but no corresponding getter. stdio_ext.h offers some functionality like a "getvbuf", but not quite – e.g. __fbufsize tells you a stream's buffer size, and __flbf whether it is line-buffered – but it isn't clear how to distinguish fully buffered and unbuffered streams. And stdio_ext.h has never been standardised, it is an extension invented on Solaris and copied by Linux (and a few other platforms too, e.g. IBM z/OS).

pjmlp 1y ago

People use reflection for monkey patching and complain when using compiled languages less supportive of such approaches. So it wouldn't surprise me, that a few folks would do some tricks with FILE internals.

recipe19 1y ago

The standard doesn't specify any serviceable parts, and I don't think there are any internals of the struct defined in musl libc on Linux (glibc may be a different story). However, on OpenBSD, it did seem to have some user-visible bits: https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51d539ef06f5f3553790#diff-5e1f2789f2dc0618ffb77ce98c2fab185eaedd0897a0326180f0fb4db794e0b3L99-L131 https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51... If you expose it, someone will probably sooner or later use it, but probably not in any sane / portable code. On the face of it, it doesn't seem like a consequential change, but maybe they're mopping up after some vulnerability in that one weird package that did touch this.

fweimer 1y ago

In gnulib, there is code that patches FILE internals for various platforms to modify behavior of <stdio.h> functions, or implement new functionality. https://cgit.git.savannah.gnu.org/cgit/gnulib.git/tree/lib/stdio-impl.h https://cgit.git.savannah.gnu.org/cgit/gnulib.git/tree/lib/s... Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported).

collinfunk 1y ago
Yes, that part of Gnulib has caused some problems previously. It is mostly used to implement <stdio_ext.h> functions on non-glibc systems. However, it is also needed for some buggy implementations of ftello, fseeko, and fflush. P.S. Hi Florian :)
- collinfunk 1y ago
  And now updated for this change. https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=69a02b5307df043ca22c4b75eaec12be21493910 https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=69a0...
quotemstr 1y ago
> Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported). Hyrum's law strikes again. People cast dl_info and poke at internal bits all the time too. glibc and others should be using kernel-style compiler-driven struct layout randomization to fight it.
- jancsika 1y ago
  > Hyrum's law strikes again. Is there a name for APIs that are drawn directly from some subset of observed behaviors? Like Crockford going, "Hey, there's a nice little data format buried in these JS objects. Schloink"

ksherlock 1y ago

*BSD stdio.h used to include macro versions of some stdio functions (feof, ferror, clearerr, fileno, getc, putc) so they would be inlined. /* * This has been tuned to generate reasonable code on the vax using pcc. */*

pm215 1y ago

The MH and nmh mail clients used to directly look into FILE internals. If you look for LINUX_STDIO in this old version of the relevant file you can see the kind of ugliness that resulted: https://cgit.git.savannah.gnu.org/cgit/nmh.git/tree/sbr/m_getfld.c?id=ffbc91b2b571f18eb8c1ed3a90e76e5d0ad65bd1 https://cgit.git.savannah.gnu.org/cgit/nmh.git/tree/sbr/m_ge... It's basically searching an email file to find the contents of either a given header or the mail body. These days there is no need to go under the hood of libc for this (and this code got ripped out over a decade ago), but back when the mail client was running on elderly VAXen this ate up significant time. Sneaking in and reading directly from the internal stdio buffer lets you avoid copying all the data the way an fread would. The same function also used to have a bit of inline vax assembly for string searching... The only reason this "works" is that traditionally the FILE struct is declared in a public header so libc can have some of its own functions implemented as macros for speed, and that there was not (when this hack was originally put in in the 1980s) yet much divergence in libc implementations.

loeg 1y ago

Historically some FILE designs exposed the structure somewhere so that some of the f* methods could be implemented as macros or inline functions (e.g., `fileno()`).

abnercoimbre 1y ago
I guess these things usually boil down to performance concerns right?

bitwize 1y ago

Hyrum's Law applies: the API of any software component is the entire exposed surface, not just what you've documented. Hence, if you have FILE well-defined somewhere in a programmer-accessible header, somebody somewhere can and will poke at the internal bits in order to achieve some hack or optimization.

krylon 1y ago
OTOH, yes. OTOH, when coding, I consider FILE to be effectively opaque in the sense that it probably is not portable, and that the implementers might change it at any time. I am reminded of this fine article by Raymond Chen, which covers a similar situation on Windows way back when: https://devblogs.microsoft.com/oldnewthing/20031015-00/?p=42163 https://devblogs.microsoft.com/oldnewthing/20031015-00/?p=42...
- brokencode 1y ago
  Yes, it would not be sane to depend on implementation details of something like this. But the sad reality is that many developers (myself included earlier in my career) will do insane things to fix a critical bug or performance problem when faced with a tight deadline.
  - krylon 1y ago
    Such is life, yes.
ars 1y ago
> to achieve some hack or optimization. Or functionality. Happens to me all the time I have some Java class that's marked Final, so instead of just extending the class and moving on, I have to copy/paste the entire class wholesale to accomplish my goal. Personally I hate "nanny" languages that block you from accessing things. It's my computer, and my code, and my compiler. Please don't do things "for my own good", I can decide that for myself. (And yes, I am aware of the argument that this lets the original programmer change the internals, in practice it's not such a big problem. Or the cure is worse than the problem - for example my copy/paste example.) Another example is a private constant. Instead of allowing me to reference it, I have to copy it. How is that any better? If the programmer has to change how the constant works then they can do so, and at that point my code will break and I'll .... copy the constant. But until then I can just use the constant.

zahlman 1y ago

I always assumed that people could poke into it, but shuddered at the thought.

asveikau 1y ago

I've seen old code do this over the years. When you consider for example that snprintf() didn't used to be standardized until the late 1990s. People would mock up a fake FILE* and use fprintf.