10 ms·

The Unix C library API can only be reliably used from C

91 points by jandeboevrie 7y ago

Eikon 7y ago
> There is no useful errno variable to load in your own language's runtime after a call to, say, the open() function FWIW, in rust there is std::io::last_os_error(). libc::perror works too.
- x3ro 7y ago
  The point of the article is that there's glue code needed for this, I think. It seems like Rust also needs this, either through linker magic or extern C: https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=errno&type= https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=er...
miohtama 7y ago
The whole C interface is clunky because C does not support multiple return values. More advanced languages should not be forced to deal with it.
- smitty1e 7y ago
  "Should" is as wobbly as errno, no?
  - masklinn 7y ago
    > 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
    - smitty1e 7y ago
      That seems a defensible definition, but also at odds with your usage: > More advanced languages should not be forced to deal with it (I'm more in agreement with the definition you offer than your usage.)
      - dbtx 7y ago
        Just a tick-- the quoted usage and the definition appear to be from two different people.
- guerrilla 7y ago
  Are you sure its not the other way around? Might C not have return values because of the systems it ran on? How would that work today in Linux, for example? Map a page to have some memory shared between the kernel and userspace?
- masklinn 7y ago
  > The whole C interface is clunky because C does not support multiple return values. C has pointers which can easily enough be used as "out parameteters". In fact the C standard library regularly makes use of these. It's not amazing, but it works fine, and it's usually fairly easy to go back and forth e.g. expressing a stat(2) call is reasonably straightforward.
  - jononor 7y ago
    However there is no way to know that programatically, and no standard way to annotate it either. So in FFI / language bindings context this is really tedious to work with.
  - merb 7y ago
    well most of the time the return value is just an error integer, while the real return value is a pointer parameter.
- moomin 7y ago
  My hot take here is that multiple return values are a conceptually complex way of handling tuple return values. The C API could do this, it just doesn’t.
- rwmj 7y ago
  You can return structs from C functions (note, I don't mean pointers, I mean whole structs). You can pass in structs too. It's not commonly done and is a bit clunky compared to returning a tuple, but it is possible.
  - masklinn 7y ago
    > It's not commonly done It’s really rather common eg several time or socket functions take a pointer as « output parameter », fill it and return an error code. Not the best but not the worst either.
  - chewxy 7y ago
    What's the difference? My understanding is as follows: A struct is a tuple. It's just some structured memory. Where a tuple would be indexed numerically, a struct is indexed with names of the fields (equiv in Python to NamedTuple)
    - rwmj 7y ago
      What happens under the hood would be the same, but in C you have to define a new struct for each return type which is clunky. In ML or Python you can return ad hoc tuples, eg: let res, _, _ = fun_which_returns_3_tuple () vs struct my_result { int res; int other; int third; }; ... struct my_result r = fun_which_returns_struct (); printf ("%d\n", r.res);
      - deleted 7y ago
        [deleted]
    - jacquesm 7y ago
      Tuples can pull their contents from different memory locations whereas structs are laid out continuously in memory (modulo padding if enabled). Of course you could store pointers in the structs you return but that would defeat the whole 'single return' concept, then you may as well return a pointer to a struct which is ugly because it will never be thread safe and is a particularly nice pitfall regarding ownership of that struct. So either return the whole struct as values or, alternatively - and this is the way most library functions do it - call the function with a pointer to an empty struct which the function then fills. That leaves the return value for error status.
- justincormack 7y ago
  The Unix API even returned a pair of file descriptors for the pipe syscall. Most ISAs have a way of returning multiple values eg for structs by value. However they fell out of fashion, I think only NetBSD still has a multiple return pipe on some architectures.
- mannykannot 7y ago
  The issue here is the interface to the operating system, which should be as universal as is feasible. Having multiple return values would work against that, just as using macros does.
zozbot234 7y ago
Back in the day, other languages had their own bindings to the POSIX/UNIX API's. There are standardized bindings for FORTRAN and Ada; not sure if others exist.
Too 7y ago
This is a fairly common problem when trying to interface other languages with any C API, not just errno. The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically, so you have to be re-implement them all in the new language. Funny because usually the reason for having a C API is to provide universal compatibility to be called from any language. If you are doing this, please avoid macros.
- arnsholt 7y ago
  I ran into that at work a few years back. I was tasked with integrating OpenSSL into the commercial Smalltalk used for the project (last updated in 1999, for more or less good reasons). A priori, not an unreasonable task as the Smalltalk had a pretty good FFI mechanism. Except of course the neverending cavalcade of functions documented in the OpenSSL docs that were actually preprocessor macros. In the end I wrote a Python script that parsed the C headers and generated FFI entry points for normal functions and Smalltalk code for preprocessor macros that weren't simple literals. The result was a small mountain of code, but it made all of OpenSSL available to our application, which was great.
  - wruza 7y ago
    You’re still lucky it wasn’t like DirectDraw or something WinAPI in general. I had to reinvent entire preprocessors and investigate implicit compiler defaults to be able to wrap that, and still there was LOWORD, MAKE_XYZ, etc, which were official arbitrary code-in-a-header interfaces. Preprocessor is a scourge of an API.
    - pjmlp 7y ago
      Hence why since Vista all major new Windows APIs are COM based (now UWP).
      - jstimpfle 7y ago
        COM is a (distributed) object model. It's mostly about structured object lifetimes. I don't think COM offers any solution for replacing bad use of preprocessor in APIs where we couldn't simply use plain C functions instead.
GlitchMr 7y ago
A function internally called by `errno` macro is part of an ABI, and changing its name will break all already compiled applications for no real reason. Most operating systems provide stable libc ABI (exception: OpenBSD). So it's a rather safe thing to depend on.
- ainar-g 7y ago
  > (exceptions: FreeBSD, OpenBSD) These are some pretty significant exceptions though, especially in the networking-related fields. I don't want the programmes on my router to suddenly not boot after a system upgrade. Especially when I don't have the source code or if the compilers haven't caught up yet.
deleted 7y ago
[deleted]
rkangel 7y ago
The issue here is that you shouldn't need to interface with C, unless you need to talk to some C Code. You absolutely shouldn't have to interface with C to talk to the kernel. I come at this from a Linux perspective, but to me the user/kernel interface (sys calls etc.) should be the well defined abstraction boundary. The C library is a useful api making that easy to use, but to dictate that people can only make sys calls through C is silly and ends up with these sort of problems.
- matheusmoreira 7y ago
  This happens due to POSIX. Nothing stops people from using direct system calls but they would rather use a C library for portability reasons. I'd even say the Linux system call interface is better than the POSIX one. Well documented, no errno. I wrote a liblinux in C and Rust and it's been useful for accessing newer system calls with no libc stubs. The more subtle problem: standard C libraries are incompatible with direct system calls. They maintain a lot of global state which will be invalidated if the program calls the kernel directly. For example, clone() cannot be used with glibc. The ideal solution is to start from scratch and build a new user space without using C at all.
- amluto 7y ago
  There is quite a bit of variation here. Linux and, AIUI, quite a few microkernels, have a defined ABI at the user/kernel boundary. BIOS, a bunch of hypervisors, and some RISC-V M-mode interfaces are similar, albeit with “user” and “kernel” replaced with other execution modes. OpenBSD and, very notably, Windows instead have a defined library interface. Windows has somewhat stable “system calls”, but only through NTDLL.dll. The real stable interface is kernel32, etc. There are arguments in favor of both approaches. One significant, if rarely appreciated benefit of the user/kernel approach is that it enables user programs that don’t use the C ABI at all. Go on Linux is like this, and Go and the vDSO have interoperability issues because the vDSO is a bit FFI-ish in Go, and Go’s FFI is not very fast.
  - the_why_of_y 7y ago
    A stable syscall ABI is a Linux peculiarity and was never a part of the UNIX tradition. Here's a talk where Bryan Cantrill describes the tightly coupled Solaris kernel/libc combination as a "welded unit", and mocks the Linux "libc ecosystem" and the complexity of ENOSYS handling in glibc. https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10 https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10
xvilka 7y ago
I think it is time to create the new POSIX standard, with better portability and safety/security. Decades of experience can help to design the better API.
- ailideex 7y ago
  New POSIX standards are released quite frequently and participation is quite open, so ... have at it :)
- pjmlp 7y ago
  It will never happen, because POSIX is literally the UNIX part of C libraries that did not land as part of the ISO C standard library.
touisteur 7y ago
One of the worst offenders is statvfs with its use of __USE_FILE_OFFSET64...
skissane 7y ago
The complaint is that the underlying function which returns the thread-local errno pointer isn’t a public API, isn’t standardised, and its name varies from OS to OS (__errno or __error or __errno_location) This specific case would be resolved if some standard (ISO C, POSIX, etc) defined a public API to do it, e.g. errno_r (Of course, that doesn’t fix any other cases where some public functions are really macros to private functions, but this appears to be the most significant such case.)

91 points by jandeboevrie 7y ago

> There is no useful errno variable to load in your own language's runtime after a call to, say, the open() function FWIW, in rust there is std::io::last_os_error(). libc::perror works too.

x3ro 7y ago
The point of the article is that there's glue code needed for this, I think. It seems like Rust also needs this, either through linker magic or extern C: https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=errno&type= https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=er...

miohtama 7y ago

The whole C interface is clunky because C does not support multiple return values. More advanced languages should not be forced to deal with it.

smitty1e 7y ago
"Should" is as wobbly as errno, no?
- masklinn 7y ago
  > 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
  - smitty1e 7y ago
    That seems a defensible definition, but also at odds with your usage: > More advanced languages should not be forced to deal with it (I'm more in agreement with the definition you offer than your usage.)
    - dbtx 7y ago
      Just a tick-- the quoted usage and the definition appear to be from two different people.
guerrilla 7y ago
Are you sure its not the other way around? Might C not have return values because of the systems it ran on? How would that work today in Linux, for example? Map a page to have some memory shared between the kernel and userspace?
masklinn 7y ago
> The whole C interface is clunky because C does not support multiple return values. C has pointers which can easily enough be used as "out parameteters". In fact the C standard library regularly makes use of these. It's not amazing, but it works fine, and it's usually fairly easy to go back and forth e.g. expressing a stat(2) call is reasonably straightforward.
- jononor 7y ago
  However there is no way to know that programatically, and no standard way to annotate it either. So in FFI / language bindings context this is really tedious to work with.
- merb 7y ago
  well most of the time the return value is just an error integer, while the real return value is a pointer parameter.
moomin 7y ago
My hot take here is that multiple return values are a conceptually complex way of handling tuple return values. The C API could do this, it just doesn’t.
rwmj 7y ago
You can return structs from C functions (note, I don't mean pointers, I mean whole structs). You can pass in structs too. It's not commonly done and is a bit clunky compared to returning a tuple, but it is possible.
- masklinn 7y ago
  > It's not commonly done It’s really rather common eg several time or socket functions take a pointer as « output parameter », fill it and return an error code. Not the best but not the worst either.
- chewxy 7y ago
  What's the difference? My understanding is as follows: A struct is a tuple. It's just some structured memory. Where a tuple would be indexed numerically, a struct is indexed with names of the fields (equiv in Python to NamedTuple)
  - rwmj 7y ago
    What happens under the hood would be the same, but in C you have to define a new struct for each return type which is clunky. In ML or Python you can return ad hoc tuples, eg: let res, _, _ = fun_which_returns_3_tuple () vs struct my_result { int res; int other; int third; }; ... struct my_result r = fun_which_returns_struct (); printf ("%d\n", r.res);
    - deleted 7y ago
      [deleted]
  - jacquesm 7y ago
    Tuples can pull their contents from different memory locations whereas structs are laid out continuously in memory (modulo padding if enabled). Of course you could store pointers in the structs you return but that would defeat the whole 'single return' concept, then you may as well return a pointer to a struct which is ugly because it will never be thread safe and is a particularly nice pitfall regarding ownership of that struct. So either return the whole struct as values or, alternatively - and this is the way most library functions do it - call the function with a pointer to an empty struct which the function then fills. That leaves the return value for error status.
justincormack 7y ago
The Unix API even returned a pair of file descriptors for the pipe syscall. Most ISAs have a way of returning multiple values eg for structs by value. However they fell out of fashion, I think only NetBSD still has a multiple return pipe on some architectures.
mannykannot 7y ago
The issue here is the interface to the operating system, which should be as universal as is feasible. Having multiple return values would work against that, just as using macros does.

zozbot234 7y ago

Back in the day, other languages had their own bindings to the POSIX/UNIX API's. There are standardized bindings for FORTRAN and Ada; not sure if others exist.

Too 7y ago

This is a fairly common problem when trying to interface other languages with any C API, not just errno. The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically, so you have to be re-implement them all in the new language. Funny because usually the reason for having a C API is to provide universal compatibility to be called from any language. If you are doing this, please avoid macros.

arnsholt 7y ago
I ran into that at work a few years back. I was tasked with integrating OpenSSL into the commercial Smalltalk used for the project (last updated in 1999, for more or less good reasons). A priori, not an unreasonable task as the Smalltalk had a pretty good FFI mechanism. Except of course the neverending cavalcade of functions documented in the OpenSSL docs that were actually preprocessor macros. In the end I wrote a Python script that parsed the C headers and generated FFI entry points for normal functions and Smalltalk code for preprocessor macros that weren't simple literals. The result was a small mountain of code, but it made all of OpenSSL available to our application, which was great.
- wruza 7y ago
  You’re still lucky it wasn’t like DirectDraw or something WinAPI in general. I had to reinvent entire preprocessors and investigate implicit compiler defaults to be able to wrap that, and still there was LOWORD, MAKE_XYZ, etc, which were official arbitrary code-in-a-header interfaces. Preprocessor is a scourge of an API.
  - pjmlp 7y ago
    Hence why since Vista all major new Windows APIs are COM based (now UWP).
    - jstimpfle 7y ago
      COM is a (distributed) object model. It's mostly about structured object lifetimes. I don't think COM offers any solution for replacing bad use of preprocessor in APIs where we couldn't simply use plain C functions instead.

GlitchMr 7y ago

A function internally called by `errno` macro is part of an ABI, and changing its name will break all already compiled applications for no real reason. Most operating systems provide stable libc ABI (exception: OpenBSD). So it's a rather safe thing to depend on.

ainar-g 7y ago
> (exceptions: FreeBSD, OpenBSD) These are some pretty significant exceptions though, especially in the networking-related fields. I don't want the programmes on my router to suddenly not boot after a system upgrade. Especially when I don't have the source code or if the compilers haven't caught up yet.

deleted 7y ago

[deleted]

rkangel 7y ago

The issue here is that you shouldn't need to interface with C, unless you need to talk to some C Code. You absolutely shouldn't have to interface with C to talk to the kernel. I come at this from a Linux perspective, but to me the user/kernel interface (sys calls etc.) should be the well defined abstraction boundary. The C library is a useful api making that easy to use, but to dictate that people can only make sys calls through C is silly and ends up with these sort of problems.

matheusmoreira 7y ago
This happens due to POSIX. Nothing stops people from using direct system calls but they would rather use a C library for portability reasons. I'd even say the Linux system call interface is better than the POSIX one. Well documented, no errno. I wrote a liblinux in C and Rust and it's been useful for accessing newer system calls with no libc stubs. The more subtle problem: standard C libraries are incompatible with direct system calls. They maintain a lot of global state which will be invalidated if the program calls the kernel directly. For example, clone() cannot be used with glibc. The ideal solution is to start from scratch and build a new user space without using C at all.
amluto 7y ago
There is quite a bit of variation here. Linux and, AIUI, quite a few microkernels, have a defined ABI at the user/kernel boundary. BIOS, a bunch of hypervisors, and some RISC-V M-mode interfaces are similar, albeit with “user” and “kernel” replaced with other execution modes. OpenBSD and, very notably, Windows instead have a defined library interface. Windows has somewhat stable “system calls”, but only through NTDLL.dll. The real stable interface is kernel32, etc. There are arguments in favor of both approaches. One significant, if rarely appreciated benefit of the user/kernel approach is that it enables user programs that don’t use the C ABI at all. Go on Linux is like this, and Go and the vDSO have interoperability issues because the vDSO is a bit FFI-ish in Go, and Go’s FFI is not very fast.
- the_why_of_y 7y ago
  A stable syscall ABI is a Linux peculiarity and was never a part of the UNIX tradition. Here's a talk where Bryan Cantrill describes the tightly coupled Solaris kernel/libc combination as a "welded unit", and mocks the Linux "libc ecosystem" and the complexity of ENOSYS handling in glibc. https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10 https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10

xvilka 7y ago

I think it is time to create the new POSIX standard, with better portability and safety/security. Decades of experience can help to design the better API.

ailideex 7y ago
New POSIX standards are released quite frequently and participation is quite open, so ... have at it :)
pjmlp 7y ago
It will never happen, because POSIX is literally the UNIX part of C libraries that did not land as part of the ISO C standard library.

touisteur 7y ago

One of the worst offenders is statvfs with its use of __USE_FILE_OFFSET64...

skissane 7y ago

The complaint is that the underlying function which returns the thread-local errno pointer isn’t a public API, isn’t standardised, and its name varies from OS to OS (__errno or __error or __errno_location) This specific case would be resolved if some standard (ISO C, POSIX, etc) defined a public API to do it, e.g. errno_r (Of course, that doesn’t fix any other cases where some public functions are really macros to private functions, but this appears to be the most significant such case.)