6 ms·

Consistently Making Wrong Decisions Whilst Writing Recreational C

215 points by luu 2y ago

ashvardanian 2y ago
Very cool! I've started a similar project around `LD_PRELOAD` a few months ago to profile the time different programs spend on LibC calls. Provoking failures was the next step :) Logging nicely was also an issue. I decided to avoid linking to any other symbols and implemented it with inline Assembly for x86/64 and aarch64: https://github.com/ashvardanian/LibSee/blob/fdae92e71c449c9196a7d3b7d547bdbd6417e481/libsee.c#L425-L542 https://github.com/ashvardanian/LibSee/blob/fdae92e71c449c91...
dmlorenzetti 2y ago
It seems like it would be a lot easier to just have students call, e.g., `ta_fork()` rather than `fork()`, and then provide an implementing file to be linked with their program. Then `ta_fork()` allows the TA to trigger errors, either probabilistically or deterministically (say, by setting environment variables). This approach would also give students insight into testing strategies like mocking, plus it would work on more operating systems. [Edit: Not to disparage this project. It seems like it would have lots of uses, and it was probably a lot of fun to develop.]
- l33t7332273 2y ago
  My intro to systems course did something similar with shared memory calls if I recall correctly
- foobarbecue 2y ago
  Seems pretty clear from the article that this person had no interest in doing this in any of the easy ways!
jrpelkonen 2y ago
Great article. Although I’ve been a Linux user since the time when stack of Slackware floppies was the prevailing installation media, I just recently learned that libc.so is also an executable.
- jagged-chisel 2y ago
  In what sense do you mean “executable?” Does it include a main() and can you launch it? Or do you mean because its code is invoked before your own executable’s code?
  - olalonde 2y ago
    It's in the article, it includes a main(): icterid$ /usr/lib/x86_64-linux-gnu/libc.so.6 GNU C Library (Debian GLIBC 2.36-9+deb12u7) stable release version 2.36. Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 12.2.0. libc ABIs: UNIQUE IFUNC ABSOLUTE Minimum supported kernel: 3.2.0 For bug reporting instructions, please see: <http://www.debian.org/Bugs/>.
    - deleted 2y ago
      [deleted]
    - rzzzt 2y ago
      Both the Makefile as well as the source file were linked from the article, ld's "-e" flag is used to set a custom entry point to "__libc_main".
- kstrauser 2y ago
  Fun trivia: so is /lib/ld-*.so*. I busted that out in an interview at a FAANG when the question was "how could you recover from accidentally running 'chmod a-x /bin/*'"? My answer: '/lib/ld.so /bin/chmod a+x /bin/*'. The interviewer paused to get out his laptop and confirm it because he had never heard of it. After a fun detour of geeking out over something new an interesting, the followup question was modified to "How else would you do it?" It's spelled "/lib/ld-linux-aarch64.so.1" on my nearest Linux box but is still executable today.
  - mandarax8 2y ago
fuhsnn 2y ago
Perl, at build time, get errno numbers of the system in a similar way[0]: preprocess errno.h with `$CC -E` and recursively scan all files in # markers for macro defines. The configure script even checks the existence of several system headers this way, so if your C compiler don't support # markers in -E output, you get missing includes everywhere. [0] https://github.com/Perl/perl5/blob/blead/ext/Errno/Errno_pm.PL https://github.com/Perl/perl5/blob/blead/ext/Errno/Errno_pm....
Joker_vD 2y ago
> (cannot dynamically load position-independent executable) ...why though? I mean, it's position-independent, just load and relocate it wherever? Or does "PIE" mean something different in Linux from what it does in Windows?
- pkal 2y ago
  I am not sure what the rationale was (probably security), but this was a concious upstream change: https://bugzilla.redhat.com/show_bug.cgi?id=1764223 https://bugzilla.redhat.com/show_bug.cgi?id=1764223
  - d110af5ccf 2y ago
    The upstream bug: https://sourceware.org/bugzilla/show_bug.cgi?id=24323 https://sourceware.org/bugzilla/show_bug.cgi?id=24323 An example of a problem provided in a different bug: https://sourceware.org/bugzilla/show_bug.cgi?id=11754#c15 https://sourceware.org/bugzilla/show_bug.cgi?id=11754#c15 But I don't really understand what's going on in that example. Aside, it only took 9 years for that patch to get conclusively rejected.
woodruffw 2y ago
Nice writeup! One limitation to this approach is that the fault injections happen at the dynamic linkage to libc layer, meaning that an enterprising student who either statically links their binary or invokes syscalls directly will circumvent the interposed functions. But in a teaching setting I could imagine this isn’t a practical concern :-) (I built a similar tool[1] a few years ago, but at the syscall layer to ensure that statically linked binaries could also have faults injected into them reliably. My colleagues used it to find a handful of bugs on prominent Go codebases.) [1]: https://blog.trailofbits.com/2019/01/17/how-to-write-a-rootkit-without-really-trying/ https://blog.trailofbits.com/2019/01/17/how-to-write-a-rootk...
jcalvinowens 2y ago
You can use pthread_once() to simplify the initialization part: https://man.archlinux.org/man/pthread_once.3.en https://man.archlinux.org/man/pthread_once.3.en I don't understand the desire not to link to pthread, it's about as ubiquitous as a library can be. I doubt it's really a problem in this application... but naive userspace spinlocks are absolutely horrendous, see NOTES here: https://man.archlinux.org/man/pthread_spin_init.3.en https://man.archlinux.org/man/pthread_spin_init.3.en User-space spin locks [...] are, by definition, prone to priority inversion and unbounded spin times. A programmer using spin locks must be exceptionally careful not only in the code, but also in terms of system configuration, thread placement, and priority assignment.
nneonneo 2y ago
This is cute! It’s worth pointing out that strace ships a similar feature (-e fault) which works for any syscall, even if the binary is statically linked. It works using ptrace, which is lower level than LD_PRELOAD. Although -e fault doesn’t support probabilistic failure, it does provide a flexible way to target specific invocations of a syscall. For example, to fail every second fork() call: -e fault=fork,errno=ENOMEM,when=1+2.
- nneonneo 2y ago
  Of course, typing this on my phone, I didn't get the syntax exactly right. Sorry! The actual strace argument should be -e fault=clone:error=ENOMEM:when=1+2 First, modern glibc doesn't use the `fork` syscall directly; it uses clone() instead, so you have to fault clone to see anything happen on a glibc-based system. Second, I botched the syntax, oops! For practical use, you probably want something like this: strace -e fault=clone:error=ENOMEM:when=2+2 -e fault=write:error=EFAULT:when=3+3 -c -o /dev/null bash This faults both clone and write at different rates (try it, the results are sort of funny), sets strace to only count events (reducing its overhead), and then suppresses the count printout at the end. Might be fun to use as a prank login shell if you want someone to think their machine is broken :)
  - AP1166 2y ago
    [flagged]
petters 2y ago
> and parsing it out of the man pages is not something I’d like to imagine doing reliably. So I must satisfy myself by manually writing these facts down. And this turns out to be the bottleneck of the entire operation. You can probably use an LLM for this.
- lionkor 2y ago
  If you do, you must unit-test the LLM stage. How do you do that without wasting a lot of time and resources? If the unit tests run through a few thousand times, would you bet your life on it never failing? I would if it was any other code.
  - samatman 2y ago
    Not necessarily. I find LLMs very helpful when the task is annoyingly underdefined / understructured, but the result I want is easy to eyeball-audit. This seems like one of those. Boiling down manpages to a consistent structure which a program can consume is going to involve a lot of special-casing a script, because they aren't written to be scraped like that. But opening the result in one window, then loading the manpages one at a time in the other, and sanity-checking the contents, is less effort than manually copy-pasting everything and getting it into a consistent data format by hand. Feeding the result of an LLM-grep sight-unseen into another program is an insane thing to do, of course. But using it like the above could save a lot of time.
rapidlua 2y ago
It was fun to read, but it would’ve probably been easier to rely on seccomp filters instead.
supriyo-biswas 2y ago
From what I understand, it'd had been far easier to use a seccomp filter instead. Would have worked with statically linked binaries too.

215 points by luu 2y ago

Very cool! I've started a similar project around `LD_PRELOAD` a few months ago to profile the time different programs spend on LibC calls. Provoking failures was the next step :) Logging nicely was also an issue. I decided to avoid linking to any other symbols and implemented it with inline Assembly for x86/64 and aarch64: https://github.com/ashvardanian/LibSee/blob/fdae92e71c449c9196a7d3b7d547bdbd6417e481/libsee.c#L425-L542 https://github.com/ashvardanian/LibSee/blob/fdae92e71c449c91...

dmlorenzetti 2y ago

It seems like it would be a lot easier to just have students call, e.g., `ta_fork()` rather than `fork()`, and then provide an implementing file to be linked with their program. Then `ta_fork()` allows the TA to trigger errors, either probabilistically or deterministically (say, by setting environment variables). This approach would also give students insight into testing strategies like mocking, plus it would work on more operating systems. [Edit: Not to disparage this project. It seems like it would have lots of uses, and it was probably a lot of fun to develop.]

l33t7332273 2y ago
My intro to systems course did something similar with shared memory calls if I recall correctly
foobarbecue 2y ago
Seems pretty clear from the article that this person had no interest in doing this in any of the easy ways!

jrpelkonen 2y ago

Great article. Although I’ve been a Linux user since the time when stack of Slackware floppies was the prevailing installation media, I just recently learned that libc.so is also an executable.

jagged-chisel 2y ago
In what sense do you mean “executable?” Does it include a main() and can you launch it? Or do you mean because its code is invoked before your own executable’s code?
- olalonde 2y ago
  It's in the article, it includes a main(): icterid$ /usr/lib/x86_64-linux-gnu/libc.so.6 GNU C Library (Debian GLIBC 2.36-9+deb12u7) stable release version 2.36. Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 12.2.0. libc ABIs: UNIQUE IFUNC ABSOLUTE Minimum supported kernel: 3.2.0 For bug reporting instructions, please see: <http://www.debian.org/Bugs/>.
  - deleted 2y ago
    [deleted]
  - rzzzt 2y ago
    Both the Makefile as well as the source file were linked from the article, ld's "-e" flag is used to set a custom entry point to "__libc_main".
kstrauser 2y ago
Fun trivia: so is /lib/ld-*.so*. I busted that out in an interview at a FAANG when the question was "how could you recover from accidentally running 'chmod a-x /bin/*'"? My answer: '/lib/ld.so /bin/chmod a+x /bin/*'. The interviewer paused to get out his laptop and confirm it because he had never heard of it. After a fun detour of geeking out over something new an interesting, the followup question was modified to "How else would you do it?" It's spelled "/lib/ld-linux-aarch64.so.1" on my nearest Linux box but is still executable today.
- mandarax8 2y ago

fuhsnn 2y ago

Perl, at build time, get errno numbers of the system in a similar way[0]: preprocess errno.h with `$CC -E` and recursively scan all files in # markers for macro defines. The configure script even checks the existence of several system headers this way, so if your C compiler don't support # markers in -E output, you get missing includes everywhere. [0] https://github.com/Perl/perl5/blob/blead/ext/Errno/Errno_pm.PL https://github.com/Perl/perl5/blob/blead/ext/Errno/Errno_pm....

Joker_vD 2y ago

> (cannot dynamically load position-independent executable) ...why though? I mean, it's position-independent, just load and relocate it wherever? Or does "PIE" mean something different in Linux from what it does in Windows?

pkal 2y ago
I am not sure what the rationale was (probably security), but this was a concious upstream change: https://bugzilla.redhat.com/show_bug.cgi?id=1764223 https://bugzilla.redhat.com/show_bug.cgi?id=1764223
- d110af5ccf 2y ago
  The upstream bug: https://sourceware.org/bugzilla/show_bug.cgi?id=24323 https://sourceware.org/bugzilla/show_bug.cgi?id=24323 An example of a problem provided in a different bug: https://sourceware.org/bugzilla/show_bug.cgi?id=11754#c15 https://sourceware.org/bugzilla/show_bug.cgi?id=11754#c15 But I don't really understand what's going on in that example. Aside, it only took 9 years for that patch to get conclusively rejected.

woodruffw 2y ago

Nice writeup! One limitation to this approach is that the fault injections happen at the dynamic linkage to libc layer, meaning that an enterprising student who either statically links their binary or invokes syscalls directly will circumvent the interposed functions. But in a teaching setting I could imagine this isn’t a practical concern :-) (I built a similar tool[1] a few years ago, but at the syscall layer to ensure that statically linked binaries could also have faults injected into them reliably. My colleagues used it to find a handful of bugs on prominent Go codebases.) [1]: https://blog.trailofbits.com/2019/01/17/how-to-write-a-rootkit-without-really-trying/ https://blog.trailofbits.com/2019/01/17/how-to-write-a-rootk...

jcalvinowens 2y ago

You can use pthread_once() to simplify the initialization part: https://man.archlinux.org/man/pthread_once.3.en https://man.archlinux.org/man/pthread_once.3.en I don't understand the desire not to link to pthread, it's about as ubiquitous as a library can be. I doubt it's really a problem in this application... but naive userspace spinlocks are absolutely horrendous, see NOTES here: https://man.archlinux.org/man/pthread_spin_init.3.en https://man.archlinux.org/man/pthread_spin_init.3.en User-space spin locks [...] are, by definition, prone to priority inversion and unbounded spin times. A programmer using spin locks must be exceptionally careful not only in the code, but also in terms of system configuration, thread placement, and priority assignment.

nneonneo 2y ago

This is cute! It’s worth pointing out that strace ships a similar feature (-e fault) which works for any syscall, even if the binary is statically linked. It works using ptrace, which is lower level than LD_PRELOAD. Although -e fault doesn’t support probabilistic failure, it does provide a flexible way to target specific invocations of a syscall. For example, to fail every second fork() call: -e fault=fork,errno=ENOMEM,when=1+2.

nneonneo 2y ago
Of course, typing this on my phone, I didn't get the syntax exactly right. Sorry! The actual strace argument should be -e fault=clone:error=ENOMEM:when=1+2 First, modern glibc doesn't use the `fork` syscall directly; it uses clone() instead, so you have to fault clone to see anything happen on a glibc-based system. Second, I botched the syntax, oops! For practical use, you probably want something like this: strace -e fault=clone:error=ENOMEM:when=2+2 -e fault=write:error=EFAULT:when=3+3 -c -o /dev/null bash This faults both clone and write at different rates (try it, the results are sort of funny), sets strace to only count events (reducing its overhead), and then suppresses the count printout at the end. Might be fun to use as a prank login shell if you want someone to think their machine is broken :)
- AP1166 2y ago
  [flagged]

petters 2y ago

> and parsing it out of the man pages is not something I’d like to imagine doing reliably. So I must satisfy myself by manually writing these facts down. And this turns out to be the bottleneck of the entire operation. You can probably use an LLM for this.

lionkor 2y ago
If you do, you must unit-test the LLM stage. How do you do that without wasting a lot of time and resources? If the unit tests run through a few thousand times, would you bet your life on it never failing? I would if it was any other code.
- samatman 2y ago
  Not necessarily. I find LLMs very helpful when the task is annoyingly underdefined / understructured, but the result I want is easy to eyeball-audit. This seems like one of those. Boiling down manpages to a consistent structure which a program can consume is going to involve a lot of special-casing a script, because they aren't written to be scraped like that. But opening the result in one window, then loading the manpages one at a time in the other, and sanity-checking the contents, is less effort than manually copy-pasting everything and getting it into a consistent data format by hand. Feeding the result of an LLM-grep sight-unseen into another program is an insane thing to do, of course. But using it like the above could save a lot of time.

rapidlua 2y ago

It was fun to read, but it would’ve probably been easier to rely on seccomp filters instead.

supriyo-biswas 2y ago

From what I understand, it'd had been far easier to use a seccomp filter instead. Would have worked with statically linked binaries too.