13 ms·

Dietlibc – a Libc optimized for small size

88 points by kick 7y ago

tyingq 7y ago
A comparison of dietlibc, uclibc, musl, and glibc, by the author of musl: http://www.etalabs.net/compare_libcs.html http://www.etalabs.net/compare_libcs.html
- jchw 7y ago
  Came here for this- musl is a very nice libc and I don’t find the comparison too biased personally.
- alexhutcheson 7y ago
  Another factor to consider is release cadence: - musl: 4 releases so far in 2019, 2 in 2018, 3 in 2017[1] - uClibc: No releases since 2012[2] - uClibc-ng: 2 releases in 2019, 3 in 2018, 6 in 2017[3] - dietlibc: only 1 release since 2013[4] [1] https://www.musl-libc.org/releases/ https://www.musl-libc.org/releases/ [2] https://uclibc.org/ https://uclibc.org/ [3] https://downloads.uclibc-ng.org/releases/ https://downloads.uclibc-ng.org/releases/ [4] https://www.fefe.de/dietlibc/ https://www.fefe.de/dietlibc/
  - unlinked_dll 7y ago
    is libc something that generally requires multiple releases every year?
    - projektfu 7y ago
      It needs to keep up with underlying changes in the operating system from time to time.
    - Twirrim 7y ago
      Given the chip architectural changes in the last 6 years, I'd hope to see at least more than one release in that time frame.
      - tlamponi 7y ago
        Often (not allways, naturally) it's enough to just rebuild with a newer compiler to leverage the new arch features. Out of interest, what are some new things (those bringing real performance gain) from the last 6 years which a standard library needs to adapt to actively? Edit: also note that OP made this pointed on purpose, the last release is from 2018, one year ago, not 2013...
        throwaway2048 7y ago
        The standard library benefits a great deal from ASM optimized implementations of functions. New instructions get added that can speed things up significantly. the compiler will almost never automatically generate this code.
        saagarjha 7y ago
        This is all true, but I’m not sure the authors of these libraries are that interested in performance.
  - KindOne 7y ago
    musl has 4 for 2019. musl-1.1.24.tar.gz (sig) - October 13, 2019 musl-1.1.23.tar.gz (sig) - July 16, 2019 musl-1.1.22.tar.gz (sig) - April 9, 2019 musl-1.1.21.tar.gz (sig) - January 21, 2019 http://www.musl-libc.org/oldversions.html http://www.musl-libc.org/oldversions.html
    - alexhutcheson 7y ago
      Good note - I missed those because the format of the text was different on the page I linked to. Fixed!
  - jackewiehose 7y ago
    > - dietlibc: only 1 release since 2013[4] you missed the first entry (20180924). > Another factor to consider is release cadence It's questionable that "more releases" == "better". Sometimes software is (ideally) just done (except for security issues). I use a lot of software that has no significant changes since decades. Regarding a libc-replacement I couldn't name a single important thing for the last years. But of course, no release could also be a bad sign, like beeing abandoned or that is has just no users (testers).
    - jcranmer 7y ago
      > Regarding a libc-replacement I couldn't name a single important thing for the last years. * Supporting new syscalls, such as getrandom(), or extra options on existing syscalls (e.g., socket options). * Support for libc-components of new hardware versions, such as the register state information that you get on signals. * Supporting new versions of C, such as C11. * Improving performance of threading and memory allocation, among other things. * Updating Unicode tables for locale information.
    - tedunangst 7y ago
      I don't think openbsd libc has ever kept the same version for two consecutive releases (six months). Which doesn't even count bug fixes as a version change. There's always something to add or remove.
- abainbridge 7y ago
  Is there a reason none of these replacement libcs are available for Windows?
  - hackcasual 7y ago
    Because part of what libc does is bridge the OS and user space. This is very much OS dependent. You can get a rough idea by looking at where it makes "syscalls."
    - BubRoss 7y ago
      That's not true for the vast majority of libc. It's usually file IO, memory mapping and command line IO that use system calls and everything else builds on those or just modifies memory.
      - hackcasual 7y ago
        Threading as well. It means porting requires significant work.
        BubRoss 7y ago
        How so? I've dug into the source of musl and allocators like jemalloc. Windows has VirtualAlloc as its analog to mmap and a simple function to print to the command line. Starting threads is similar. Most of libc is pure C. Even most of musl is optimizations for different platforms and compilers. The simple straightforward versions of functions are not anywhere near the complexity of what gets used most of the time.
  - earenndil 7y ago
    Most libcs aren't cross-platform; there is mingw, which provides an alternate libc on windows. I'm not actually sure why the libcs aren't cross-platform; I'm currently working on one which plans to support freebsd, linux, and windows.
  - anta40 7y ago
- earenndil 7y ago
  Should note that glibc now supports c11 threads. AFAIK there was some problem with their initial specification that caused them to want not to implement them, and c18 amended that so they're now comfortable supporting them. Also note that openrisc has been mostly superceded by risc-v, which afaik glibc and musl both support; don't know about the other libcs.
- ainar-g 7y ago
  These are rather old though: musl 1.1.5 uClibc 0.9.33.2 (Buildroot 2015.02) dietlibc 0.32 glibc 2.19 musl is at 1.1.24 at the time of writing.
vnglst 7y ago
I like the proverb on the top of the page: Person who say it cannot be done should not interrupt person doing it. --Chinese Proverb
- rdescartes 7y ago
  However, I think it maybe a fake proverb. https://www.npr.org/2018/06/12/619140377/after-ivanka-trump-quotes-chinese-proverb-a-hunt-for-the-proverbial-author https://www.npr.org/2018/06/12/619140377/after-ivanka-trump-...
trileansoftware 7y ago
Interesting project. What is the footprint of this compiled for a MCU, something like a STM32?
- kbumsik 7y ago
  This libc is for Linux, so I guess you can't compile it for a bare-metal system like STM32.
  - alexhutcheson 7y ago
    For microcontrollers the most popular open source option is newlib[1]. [1] http://sourceware.org/newlib/ http://sourceware.org/newlib/
    - mlyle 7y ago
      Keith Packard maintains an excellent "picolibc" that I have used recently (and contributed to, a little). It was formerly called newlib-nano, and consists of newlib with the string functions replaced with cheaper (slower) versions from avrlibc and various other slimming changes (e.g. reduction of struct reent stuff, and some locale stuff, by default). https://github.com/keith-packard/picolibc https://github.com/keith-packard/picolibc
mhd 7y ago
If I remember correctly, dietlibc was part of a whole bunch of DJB-fanboy releases by fefe way back when dinosaurs and qmail ruled the earth. Not that much happening since then, I think he's mostly CCC-ish-blogging these days.
- jlg23 7y ago
  If I remember correctly it grew out of frustration with glibc size which turned into research into how to make a libc as small as possible. No DJB involved. > Not that much happening since then, I think he's mostly CCC-ish-blogging these days. Latest release was last year and the changelog[1] suggests it was a group effort. Do I smell a personal dislike for the author? [1] http://www.fefe.de/dietlibc/changes-0.34.txt http://www.fefe.de/dietlibc/changes-0.34.txt
  - mhd 7y ago
    Neither for the author nor for DJB. I was on the mailing list when dietlibc and libowfat (the alternative, non-traditional, DJB-inspired core library) were created. Was an interesting time and I liked the philosophy way more than e.g. the whole 'suckless' stuff (which is more in the Plan 9 corner). I don't regard "fanboy" as a derogatory term, by the way. There was quite a lot of development and thoughts back then, but in recent years I hadn't heard a lot out of that corner, and in German circles I mostly saw von Leitner's name mentioned when it came to some digital rights thing (our various German and EU "privacy" laws).
    - jackewiehose 7y ago
      > I don't regard "fanboy" as a derogatory term, by the way. I do. Whenever I read "fanboy", I assume the author is at least no fan of the subject (in this case I assume you don't like DJB). Otherwise it would just be a "fan"... ?
avian 7y ago
I recently encountered dietlibc while trying to debug a segfault in integrit [1]. As packaged in Debian, integrit is a static binary using dietlibc. For some unknown reason I found it completely impossible to get any usable stacktrace out of it with gdb and the usual tools and after a few days of frustration I gave up on finding the cause. [1] https://github.com/integrit/integrit https://github.com/integrit/integrit
- breadbox 7y ago
  Yes, part of the usual process of making a tiny binary is jettisoning much of the overhead that debuggers depend on in order to find their way through an external program. Not just the debugging symbols themselves, but things like section header tables and maintenance of linked stack frames in the running code. Dumping all of that saves space, at the expense of making the resulting binary more and more opaque.
- saagarjha 7y ago
  It’s probably because the code optimized out the frame pointer; if you really wanted a backtrace you’d need to look back at the stack to see where the return address was.
rwmj 7y ago
I used dietlibc as a replacement for glibc in supermin, when making an initramfs. As background the size of the initramfs is key for boot performance since qemu takes quite some time (where "some time" means milliseconds) to load a large initramfs; and the binaries in the initramfs are usually statically linked. The final size of the init binary was about 1/40th as big as when using glibc: https://github.com/libguestfs/supermin/blob/c97b3917068597a0e68e88d9a905da766ade40da/README#L147 https://github.com/libguestfs/supermin/blob/c97b3917068597a0...
- pmoriarty 7y ago
  I don't do anything special to my initramfs, and the largest of them that I've generated over the years is 16 MB in size. For my desktop with a 3 TB drive, having the initramfs take up half a meg instead won't make any appreciable difference. I could see it being very useful on resource constrained microcontrollers or something, though.
  - rwmj 7y ago
    It makes a huge difference when trying to boot an appliance virtual machine in milliseconds. Paper: http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2016-eng-talk http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2016...
  - CameronNemo 7y ago
    I believe the point is boot speed. Larger initramfs take longer to unpack.

88 points by kick 7y ago

A comparison of dietlibc, uclibc, musl, and glibc, by the author of musl: http://www.etalabs.net/compare_libcs.html http://www.etalabs.net/compare_libcs.html

jchw 7y ago
Came here for this- musl is a very nice libc and I don’t find the comparison too biased personally.
alexhutcheson 7y ago
Another factor to consider is release cadence: - musl: 4 releases so far in 2019, 2 in 2018, 3 in 2017[1] - uClibc: No releases since 2012[2] - uClibc-ng: 2 releases in 2019, 3 in 2018, 6 in 2017[3] - dietlibc: only 1 release since 2013[4] [1] https://www.musl-libc.org/releases/ https://www.musl-libc.org/releases/ [2] https://uclibc.org/ https://uclibc.org/ [3] https://downloads.uclibc-ng.org/releases/ https://downloads.uclibc-ng.org/releases/ [4] https://www.fefe.de/dietlibc/ https://www.fefe.de/dietlibc/
- unlinked_dll 7y ago
  is libc something that generally requires multiple releases every year?
  - projektfu 7y ago
    It needs to keep up with underlying changes in the operating system from time to time.
  - Twirrim 7y ago
    Given the chip architectural changes in the last 6 years, I'd hope to see at least more than one release in that time frame.
    - tlamponi 7y ago
      Often (not allways, naturally) it's enough to just rebuild with a newer compiler to leverage the new arch features. Out of interest, what are some new things (those bringing real performance gain) from the last 6 years which a standard library needs to adapt to actively? Edit: also note that OP made this pointed on purpose, the last release is from 2018, one year ago, not 2013...
      - throwaway2048 7y ago
        The standard library benefits a great deal from ASM optimized implementations of functions. New instructions get added that can speed things up significantly. the compiler will almost never automatically generate this code.
        saagarjha 7y ago
        This is all true, but I’m not sure the authors of these libraries are that interested in performance.
- KindOne 7y ago
  musl has 4 for 2019. musl-1.1.24.tar.gz (sig) - October 13, 2019 musl-1.1.23.tar.gz (sig) - July 16, 2019 musl-1.1.22.tar.gz (sig) - April 9, 2019 musl-1.1.21.tar.gz (sig) - January 21, 2019 http://www.musl-libc.org/oldversions.html http://www.musl-libc.org/oldversions.html
  - alexhutcheson 7y ago
    Good note - I missed those because the format of the text was different on the page I linked to. Fixed!
- jackewiehose 7y ago
  > - dietlibc: only 1 release since 2013[4] you missed the first entry (20180924). > Another factor to consider is release cadence It's questionable that "more releases" == "better". Sometimes software is (ideally) just done (except for security issues). I use a lot of software that has no significant changes since decades. Regarding a libc-replacement I couldn't name a single important thing for the last years. But of course, no release could also be a bad sign, like beeing abandoned or that is has just no users (testers).
  - jcranmer 7y ago
    > Regarding a libc-replacement I couldn't name a single important thing for the last years. * Supporting new syscalls, such as getrandom(), or extra options on existing syscalls (e.g., socket options). * Support for libc-components of new hardware versions, such as the register state information that you get on signals. * Supporting new versions of C, such as C11. * Improving performance of threading and memory allocation, among other things. * Updating Unicode tables for locale information.
  - tedunangst 7y ago
    I don't think openbsd libc has ever kept the same version for two consecutive releases (six months). Which doesn't even count bug fixes as a version change. There's always something to add or remove.
abainbridge 7y ago
Is there a reason none of these replacement libcs are available for Windows?
- hackcasual 7y ago
  Because part of what libc does is bridge the OS and user space. This is very much OS dependent. You can get a rough idea by looking at where it makes "syscalls."
  - BubRoss 7y ago
    That's not true for the vast majority of libc. It's usually file IO, memory mapping and command line IO that use system calls and everything else builds on those or just modifies memory.
    - hackcasual 7y ago
      Threading as well. It means porting requires significant work.
      - BubRoss 7y ago
        How so? I've dug into the source of musl and allocators like jemalloc. Windows has VirtualAlloc as its analog to mmap and a simple function to print to the command line. Starting threads is similar. Most of libc is pure C. Even most of musl is optimizations for different platforms and compilers. The simple straightforward versions of functions are not anywhere near the complexity of what gets used most of the time.
- earenndil 7y ago
  Most libcs aren't cross-platform; there is mingw, which provides an alternate libc on windows. I'm not actually sure why the libcs aren't cross-platform; I'm currently working on one which plans to support freebsd, linux, and windows.
- anta40 7y ago
earenndil 7y ago
Should note that glibc now supports c11 threads. AFAIK there was some problem with their initial specification that caused them to want not to implement them, and c18 amended that so they're now comfortable supporting them. Also note that openrisc has been mostly superceded by risc-v, which afaik glibc and musl both support; don't know about the other libcs.
ainar-g 7y ago
These are rather old though: musl 1.1.5 uClibc 0.9.33.2 (Buildroot 2015.02) dietlibc 0.32 glibc 2.19 musl is at 1.1.24 at the time of writing.

vnglst 7y ago

I like the proverb on the top of the page: Person who say it cannot be done should not interrupt person doing it. --Chinese Proverb

rdescartes 7y ago
However, I think it maybe a fake proverb. https://www.npr.org/2018/06/12/619140377/after-ivanka-trump-quotes-chinese-proverb-a-hunt-for-the-proverbial-author https://www.npr.org/2018/06/12/619140377/after-ivanka-trump-...

trileansoftware 7y ago

Interesting project. What is the footprint of this compiled for a MCU, something like a STM32?

kbumsik 7y ago
This libc is for Linux, so I guess you can't compile it for a bare-metal system like STM32.
- alexhutcheson 7y ago
  For microcontrollers the most popular open source option is newlib[1]. [1] http://sourceware.org/newlib/ http://sourceware.org/newlib/
  - mlyle 7y ago
    Keith Packard maintains an excellent "picolibc" that I have used recently (and contributed to, a little). It was formerly called newlib-nano, and consists of newlib with the string functions replaced with cheaper (slower) versions from avrlibc and various other slimming changes (e.g. reduction of struct reent stuff, and some locale stuff, by default). https://github.com/keith-packard/picolibc https://github.com/keith-packard/picolibc

mhd 7y ago

If I remember correctly, dietlibc was part of a whole bunch of DJB-fanboy releases by fefe way back when dinosaurs and qmail ruled the earth. Not that much happening since then, I think he's mostly CCC-ish-blogging these days.

jlg23 7y ago
If I remember correctly it grew out of frustration with glibc size which turned into research into how to make a libc as small as possible. No DJB involved. > Not that much happening since then, I think he's mostly CCC-ish-blogging these days. Latest release was last year and the changelog[1] suggests it was a group effort. Do I smell a personal dislike for the author? [1] http://www.fefe.de/dietlibc/changes-0.34.txt http://www.fefe.de/dietlibc/changes-0.34.txt
- mhd 7y ago
  Neither for the author nor for DJB. I was on the mailing list when dietlibc and libowfat (the alternative, non-traditional, DJB-inspired core library) were created. Was an interesting time and I liked the philosophy way more than e.g. the whole 'suckless' stuff (which is more in the Plan 9 corner). I don't regard "fanboy" as a derogatory term, by the way. There was quite a lot of development and thoughts back then, but in recent years I hadn't heard a lot out of that corner, and in German circles I mostly saw von Leitner's name mentioned when it came to some digital rights thing (our various German and EU "privacy" laws).
  - jackewiehose 7y ago
    > I don't regard "fanboy" as a derogatory term, by the way. I do. Whenever I read "fanboy", I assume the author is at least no fan of the subject (in this case I assume you don't like DJB). Otherwise it would just be a "fan"... ?

avian 7y ago

I recently encountered dietlibc while trying to debug a segfault in integrit [1]. As packaged in Debian, integrit is a static binary using dietlibc. For some unknown reason I found it completely impossible to get any usable stacktrace out of it with gdb and the usual tools and after a few days of frustration I gave up on finding the cause. [1] https://github.com/integrit/integrit https://github.com/integrit/integrit

breadbox 7y ago
Yes, part of the usual process of making a tiny binary is jettisoning much of the overhead that debuggers depend on in order to find their way through an external program. Not just the debugging symbols themselves, but things like section header tables and maintenance of linked stack frames in the running code. Dumping all of that saves space, at the expense of making the resulting binary more and more opaque.
saagarjha 7y ago
It’s probably because the code optimized out the frame pointer; if you really wanted a backtrace you’d need to look back at the stack to see where the return address was.

rwmj 7y ago

I used dietlibc as a replacement for glibc in supermin, when making an initramfs. As background the size of the initramfs is key for boot performance since qemu takes quite some time (where "some time" means milliseconds) to load a large initramfs; and the binaries in the initramfs are usually statically linked. The final size of the init binary was about 1/40th as big as when using glibc: https://github.com/libguestfs/supermin/blob/c97b3917068597a0e68e88d9a905da766ade40da/README#L147 https://github.com/libguestfs/supermin/blob/c97b3917068597a0...

pmoriarty 7y ago
I don't do anything special to my initramfs, and the largest of them that I've generated over the years is 16 MB in size. For my desktop with a 3 TB drive, having the initramfs take up half a meg instead won't make any appreciable difference. I could see it being very useful on resource constrained microcontrollers or something, though.
- rwmj 7y ago
  It makes a huge difference when trying to boot an appliance virtual machine in milliseconds. Paper: http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2016-eng-talk http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2016...
- CameronNemo 7y ago
  I believe the point is boot speed. Larger initramfs take longer to unpack.