9 ms·

Relibc: C Library in Rust for Redox and Linux

248 points by dralley 9y ago

zakk 9y ago
How many security-critical bugs have been found in libc in the last ten years? I know that in principle Rust is safer, but I am wondering it makes sense to re-implement code so heavily audited as libc. Wouldn’t it be better to redirect the effort to some other less audited code?
- rwmj 9y ago
  Lots: https://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-767/GNU-Glibc.html https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
  - sebcat 9y ago
    musl: https://www.cvedetails.com/vulnerability-list/vendor_id-16859/Musl-libc.html https://www.cvedetails.com/vulnerability-list/vendor_id-1685... Can't find separate listings for libc in FreeBSD, OpenBSD but the list is smaller than glibc if you check e.g., VuXML or https://www.cvedetails.com/vulnerability-list/vendor_id-97/product_id-163/Openbsd-Openbsd.html https://www.cvedetails.com/vulnerability-list/vendor_id-97/p... Looks like glibc is the worst sinner here.
    - rwmj 9y ago
      More people study and use glibc, so I guess they find more bugs. On the other hand glibc has a larger code base. Anyway it's difficult to compare bug counts across projects.
  - zuzun 9y ago
    Half of them are logic errors and around 15 bugs are integer overflows of which the project above is riddled with.
  - archi42 9y ago
    No no, these are not bugs, you are just misinterpreting the {standard, API, intention, stars} ;-) /sarcasm
    - archi42 9y ago
      Hm, at those who downvoted me: Read some older glibc bug reports. For example the one about sscanf("foo", "%10s", strptr) returning 1. Hint: According to the standard, it should return 0. Musl does it right, which recently broke some of my test cases. But the bug is known and not fixed since years, maybe a decade...
- dralley 9y ago
  While better security is a goal of relibc it is not actually one of the primary goals. Most existing libc implementations are not particularly portable across different kernels, and especially not to microkernels such as Redox OS. The goal is to have a standard library that is easier for the Redox OS devs to develop in (obviously, Rust makes sense here), and not so tied down to one kernel like musl is for example.
  - shmerl 9y ago
    Will Rust be able to switch to it completely as to cut out relying on glibc on Linux for example?
    - jackpot51 9y ago
      Yes, that would be possible.
    - thristian 9y ago
      There are important reasons why Rust should continue to use glibc by default, but the Rust compiler already supports musl libc on Linux, so glibc is not required.
      - shmerl 9y ago
        > here are important reasons why Rust should continue to use glibc by default To be more compatible with system code?
        comex 9y ago
        That’s one reason. Another is NSS: http://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html http://www.gnu.org/software/libc/manual/html_node/Name-Servi... It’s a mechanism in glibc for loading dynamic libraries at runtime (based on a config file) to handle DNS and some other things. Anything statically linked can’t support this, which can cause issues with some setups.
- CJefferson 9y ago
  A friend of mine found a bug in memmove over Christmas. Rust would have caught it (signed/unsigned issue), but the bug was in sse2 assembler so would still exist if they allowed optimised ask in the rust version.
  - dikaiosune 9y ago
    The path to stabilizing explicit simd intrinsics for rust should make many of those kinds of errors easier to catch with rusts casting requirements.
- Thaxll 9y ago
  Like Redox this lib will be probably be never be seriously used. It's a pet project.
  - nh2 9y ago
    How do you know that? Look how much Redox has achieved in so little time and with so few people. It has everything from a kernel to a fully working desktop GUI. All because they use a good language that allows easy code reuse and proper library packaging. Also remember that Linux at some point was somebody's pet project.
  - pjmlp 9y ago
    "Hello everybody out there using minix – I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things). I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them :)" -- Linus
    - jhasse 9y ago
      survivorship bias
      - carlmr 9y ago
        It is, but everything starts at humble beginnings.
wyldfire 9y ago
Also of interest is japaric's steed [1] -- a rust standard library (not a "C lib"). [1] https://github.com/japaric/steed https://github.com/japaric/steed
- surajrmal 9y ago
  I'm wondering if it was abandoned. It looks like it hasn't had commits in months.
  - dEnigma 9y ago
    Well, the "auto" branch saw a commit 20 days ago, and before that in January. But they mostly seem to be small changes to "unbreak stuff".
andrewmcwatters 9y ago
Gonna go on a tangent here. I'm always curious about stuff like this. Everyone and their mother wants to replace C, but what's people's answer to it being a part of POSIX? I mean it's literally a part of the standards family, and no one seems to address this. I can never take aspiring C-replacements seriously because no one comes out and openly says, "This is how we replace C." They're all really just claims that C is bad, and we shouldn't use it. Okay, well how do you replace it? Or was that never the goal to begin with and you just want me to use this language?
- floatboth 9y ago
  Who cares about POSIX? And yeah, Redox is how "we replace C". A full operating system where C is not the main language.
  - tzahola 9y ago
    Why does an OS have a “main” language?
    - vbezhenar 9y ago
      If you know this language, you can change any part of OS. I think that's a valuable asset. For example if you're using OpenBSD and know C, you can hack anything from kernel to smtp daemon. If you're OS is written with 10 languages, it's much harder to learn them all.
      - adrianmonk 9y ago
        That's a good argument for keeping the set of languages limited. But it doesn't necessarily have to be just one.
        ibotty 9y ago
        It never is. Even ancient unix had shell and (maybe more to your point) yacc. Yacc still generates C files, but that's more of an implementation detail. The C _interop_ is key.
unfamiliar 9y ago
I wonder how they are going to handle testing. The tests directory seems pretty bare, surely there are some libc tests (written in C I guess) that can be directly imported?
- dralley 9y ago
  There has been discussion about leveraging parts of musl's test suite.
pornel 9y ago
If I understand correctly, that will eventually allow 100% pure Rust executables, and these executables will be easily redistributable and work on any distro (only the latter is currently possible with the MUSL target).
- ris 9y ago
  > these executables will be easily redistributable and work on any distro While this sounds amazing and I'm sure is very useful in various situations, I think encouraging a culture of distributing software in such a fashion is an incredibly bad idea. This is a big issue I have with the Go culture. Downloading a statically linked binary from some site and just sticking it somewhere is extremely bad practise to encourage. The user will never get prompted to update it, vulnerabilities and bugs in the vendored libraries will never be addressed... the list of issues is endless. Distributions are not really about distribution, they're about maintenance.
  - heavenlyblue 9y ago
    It reminds me of why the protobuf compiler had never received any functionality regarding the generation of python code with the redefined library roots [1]: Google has a monorepo, therefore they never needed this. It is especially amusing when it is later being explained [2]: > Overall, your directory structure appears to be more complicated > than what we generally do at Google (which is the environment where > all this behavior was designed/evolved). At Google we generally have > a single directory structure for all source files, including protos. Well, yes. You do... At Google. [1] https://github.com/google/protobuf/issues/881 https://github.com/google/protobuf/issues/881 [2] https://github.com/google/protobuf/issues/1491 https://github.com/google/protobuf/issues/1491
    - haberman 9y ago
      Hi there, those are my words you quoted above. I just wanted to emphasize that we would help open-source users out on this if we could. This isn't a case of "we didn't need it, so we never implemented it." The problem is that no one has proposed a solution that is compatible with our need to do parallel compilation inside Google. The only proposal I saw involved removing the 1:1 file correspondence between .proto input and _pb2.py output. I can understand why people would want this. But inside Google, we compile all our .proto files in parallel, and our whole build system depends on being able to parallelize things in this way. If you have a solution to this problem that doesn't break any of our existing use cases, we're all ears.
snarfy 9y ago
Once you've built this easy way out I fear it will turn into a crutch that stops new, pure rust apps from being written. Do you really want a C library in Redox?
- jackpot51 9y ago
  I highly doubt that
- ris 9y ago
  Do you want Redox unable to run 90% of the world's existing systems software?
- tdbgamer 9y ago
  There is tons of C software out there that depends on libc that is never going to be rewritten in Rust. Also think about interpreted languages like Python, Ruby, etc. I imagine they all depend on libc. The only real way for Redox to gain adoption is through cross platform compatibility.
  - imtringued 9y ago
    Does the rust compiler depend on libc?
    - Sean1708 9y ago
      Presumably. Rust programs certainly do.
      - steveklabnik 9y ago
        The compiler does. Rust programs don't inherently depend on libc; it's libstd that does. Build without the standard library, no libc required.
HumanDrivenDev 9y ago
Is there a good guide on how to make libraries with C bindings in rust? I've been learning it for the past few days but without much of a use case. Fast libraries with a C API in a higher level language would be very useful though.
- vvanders 9y ago
  It's mostly straightforward, you just define with #[no_mangle] + extern fn ... The Rust docs cover it in small detail[1]. FWIW you can do the same with JNI and have direct Java->Rust bindings. If you want to get really clever you can even parse the Rust AST(using syntex) to auto-generate java/headers(there may be libs to do this now, I haven't looked in a while). What's nice about the C FFI is you can pretty much embed Rust anywhere, I've used it in C/C++, Java, C# and Python pretty easily. [1] https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe-rust.html#calling-rust-functions-from-other-languages https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe...
  - Manishearth 9y ago
    There's a tool called cbindgen (https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen) that generates header files from rust code . It's a bit experimental. The reverse tool, bindgen (https://github.com/rust-lang-nursery/rust-bindgen/ https://github.com/rust-lang-nursery/rust-bindgen/) is more mature -- generates rust bindings from header files.
    - vvanders 9y ago
      Oh yeah, I should have mentioned bindgen, it's fantastic!
- samjs 9y ago
  I wrote on a similar topic [1], about writing system libraries in Rust with a C API and bindings to other languages. Which covers a bit of this, plus some of the sharp edges of FFI. Most of the trickier parts of it are covered in the FFI omnibus [2]. Also worth checking out is the cbindgen tool [3]. [1]: https://libpasta.github.io/blog/bindings/ https://libpasta.github.io/blog/bindings/ [2]: http://jakegoulding.com/rust-ffi-omnibus/ http://jakegoulding.com/rust-ffi-omnibus/ [3]: https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen
JackNichelson 9y ago
That's just stupid.
dikaiosune 9y ago
Awesome! It looks like they have some tests in the directory -- I'd be curious whether they're able to re-use musl's test suite some. I had an experiment a while ago to incrementally port musl to rust while keeping the test suite passing: https://github.com/anp/rusl https://github.com/anp/rusl. Haven't put any time into it in a while, but I think matching musl's program-facing behavior is still a good idea.
brianon99 9y ago
There are many projects recently titled like "xxx in Rust" or "xxx in Go", rewriting an existing software, seems to be built not because there is a problem in the existing software, but without a clear reason. For learning a new language, it can be a good practice. But if you hope the project to grow successfully I think there is no hope.
- sanxiyn 9y ago
  relibc is built to solve a problem in the existing software, namely newlib. It has a clear reason. Redox is an opearting system. An operating system needs a C library. Redox used to use newlib, but developers found it unsatisfying.

248 points by dralley 9y ago

zakk 9y ago

How many security-critical bugs have been found in libc in the last ten years? I know that in principle Rust is safer, but I am wondering it makes sense to re-implement code so heavily audited as libc. Wouldn’t it be better to redirect the effort to some other less audited code?

rwmj 9y ago
Lots: https://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-767/GNU-Glibc.html https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
- sebcat 9y ago
  musl: https://www.cvedetails.com/vulnerability-list/vendor_id-16859/Musl-libc.html https://www.cvedetails.com/vulnerability-list/vendor_id-1685... Can't find separate listings for libc in FreeBSD, OpenBSD but the list is smaller than glibc if you check e.g., VuXML or https://www.cvedetails.com/vulnerability-list/vendor_id-97/product_id-163/Openbsd-Openbsd.html https://www.cvedetails.com/vulnerability-list/vendor_id-97/p... Looks like glibc is the worst sinner here.
  - rwmj 9y ago
    More people study and use glibc, so I guess they find more bugs. On the other hand glibc has a larger code base. Anyway it's difficult to compare bug counts across projects.
- zuzun 9y ago
  Half of them are logic errors and around 15 bugs are integer overflows of which the project above is riddled with.
- archi42 9y ago
  No no, these are not bugs, you are just misinterpreting the {standard, API, intention, stars} ;-) /sarcasm
  - archi42 9y ago
    Hm, at those who downvoted me: Read some older glibc bug reports. For example the one about sscanf("foo", "%10s", strptr) returning 1. Hint: According to the standard, it should return 0. Musl does it right, which recently broke some of my test cases. But the bug is known and not fixed since years, maybe a decade...
dralley 9y ago
While better security is a goal of relibc it is not actually one of the primary goals. Most existing libc implementations are not particularly portable across different kernels, and especially not to microkernels such as Redox OS. The goal is to have a standard library that is easier for the Redox OS devs to develop in (obviously, Rust makes sense here), and not so tied down to one kernel like musl is for example.
- shmerl 9y ago
  Will Rust be able to switch to it completely as to cut out relying on glibc on Linux for example?
  - jackpot51 9y ago
    Yes, that would be possible.
  - thristian 9y ago
    There are important reasons why Rust should continue to use glibc by default, but the Rust compiler already supports musl libc on Linux, so glibc is not required.
    - shmerl 9y ago
      > here are important reasons why Rust should continue to use glibc by default To be more compatible with system code?
      - comex 9y ago
        That’s one reason. Another is NSS: http://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html http://www.gnu.org/software/libc/manual/html_node/Name-Servi... It’s a mechanism in glibc for loading dynamic libraries at runtime (based on a config file) to handle DNS and some other things. Anything statically linked can’t support this, which can cause issues with some setups.
CJefferson 9y ago
A friend of mine found a bug in memmove over Christmas. Rust would have caught it (signed/unsigned issue), but the bug was in sse2 assembler so would still exist if they allowed optimised ask in the rust version.
- dikaiosune 9y ago
  The path to stabilizing explicit simd intrinsics for rust should make many of those kinds of errors easier to catch with rusts casting requirements.
Thaxll 9y ago
Like Redox this lib will be probably be never be seriously used. It's a pet project.
- nh2 9y ago
  How do you know that? Look how much Redox has achieved in so little time and with so few people. It has everything from a kernel to a fully working desktop GUI. All because they use a good language that allows easy code reuse and proper library packaging. Also remember that Linux at some point was somebody's pet project.
- pjmlp 9y ago
  "Hello everybody out there using minix – I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things). I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them :)" -- Linus
  - jhasse 9y ago
    survivorship bias
    - carlmr 9y ago
      It is, but everything starts at humble beginnings.

wyldfire 9y ago

Also of interest is japaric's steed [1] -- a rust standard library (not a "C lib"). [1] https://github.com/japaric/steed https://github.com/japaric/steed

surajrmal 9y ago
I'm wondering if it was abandoned. It looks like it hasn't had commits in months.
- dEnigma 9y ago
  Well, the "auto" branch saw a commit 20 days ago, and before that in January. But they mostly seem to be small changes to "unbreak stuff".

andrewmcwatters 9y ago

Gonna go on a tangent here. I'm always curious about stuff like this. Everyone and their mother wants to replace C, but what's people's answer to it being a part of POSIX? I mean it's literally a part of the standards family, and no one seems to address this. I can never take aspiring C-replacements seriously because no one comes out and openly says, "This is how we replace C." They're all really just claims that C is bad, and we shouldn't use it. Okay, well how do you replace it? Or was that never the goal to begin with and you just want me to use this language?

floatboth 9y ago
Who cares about POSIX? And yeah, Redox is how "we replace C". A full operating system where C is not the main language.
- tzahola 9y ago
  Why does an OS have a “main” language?
  - vbezhenar 9y ago
    If you know this language, you can change any part of OS. I think that's a valuable asset. For example if you're using OpenBSD and know C, you can hack anything from kernel to smtp daemon. If you're OS is written with 10 languages, it's much harder to learn them all.
    - adrianmonk 9y ago
      That's a good argument for keeping the set of languages limited. But it doesn't necessarily have to be just one.
      - ibotty 9y ago
        It never is. Even ancient unix had shell and (maybe more to your point) yacc. Yacc still generates C files, but that's more of an implementation detail. The C _interop_ is key.

unfamiliar 9y ago

I wonder how they are going to handle testing. The tests directory seems pretty bare, surely there are some libc tests (written in C I guess) that can be directly imported?

dralley 9y ago
There has been discussion about leveraging parts of musl's test suite.

pornel 9y ago

If I understand correctly, that will eventually allow 100% pure Rust executables, and these executables will be easily redistributable and work on any distro (only the latter is currently possible with the MUSL target).

ris 9y ago
> these executables will be easily redistributable and work on any distro While this sounds amazing and I'm sure is very useful in various situations, I think encouraging a culture of distributing software in such a fashion is an incredibly bad idea. This is a big issue I have with the Go culture. Downloading a statically linked binary from some site and just sticking it somewhere is extremely bad practise to encourage. The user will never get prompted to update it, vulnerabilities and bugs in the vendored libraries will never be addressed... the list of issues is endless. Distributions are not really about distribution, they're about maintenance.
- heavenlyblue 9y ago
  It reminds me of why the protobuf compiler had never received any functionality regarding the generation of python code with the redefined library roots [1]: Google has a monorepo, therefore they never needed this. It is especially amusing when it is later being explained [2]: > Overall, your directory structure appears to be more complicated > than what we generally do at Google (which is the environment where > all this behavior was designed/evolved). At Google we generally have > a single directory structure for all source files, including protos. Well, yes. You do... At Google. [1] https://github.com/google/protobuf/issues/881 https://github.com/google/protobuf/issues/881 [2] https://github.com/google/protobuf/issues/1491 https://github.com/google/protobuf/issues/1491
  - haberman 9y ago
    Hi there, those are my words you quoted above. I just wanted to emphasize that we would help open-source users out on this if we could. This isn't a case of "we didn't need it, so we never implemented it." The problem is that no one has proposed a solution that is compatible with our need to do parallel compilation inside Google. The only proposal I saw involved removing the 1:1 file correspondence between .proto input and _pb2.py output. I can understand why people would want this. But inside Google, we compile all our .proto files in parallel, and our whole build system depends on being able to parallelize things in this way. If you have a solution to this problem that doesn't break any of our existing use cases, we're all ears.

snarfy 9y ago

Once you've built this easy way out I fear it will turn into a crutch that stops new, pure rust apps from being written. Do you really want a C library in Redox?

jackpot51 9y ago
I highly doubt that
ris 9y ago
Do you want Redox unable to run 90% of the world's existing systems software?
tdbgamer 9y ago
There is tons of C software out there that depends on libc that is never going to be rewritten in Rust. Also think about interpreted languages like Python, Ruby, etc. I imagine they all depend on libc. The only real way for Redox to gain adoption is through cross platform compatibility.
- imtringued 9y ago
  Does the rust compiler depend on libc?
  - Sean1708 9y ago
    Presumably. Rust programs certainly do.
    - steveklabnik 9y ago
      The compiler does. Rust programs don't inherently depend on libc; it's libstd that does. Build without the standard library, no libc required.

HumanDrivenDev 9y ago

Is there a good guide on how to make libraries with C bindings in rust? I've been learning it for the past few days but without much of a use case. Fast libraries with a C API in a higher level language would be very useful though.

vvanders 9y ago
It's mostly straightforward, you just define with #[no_mangle] + extern fn ... The Rust docs cover it in small detail[1]. FWIW you can do the same with JNI and have direct Java->Rust bindings. If you want to get really clever you can even parse the Rust AST(using syntex) to auto-generate java/headers(there may be libs to do this now, I haven't looked in a while). What's nice about the C FFI is you can pretty much embed Rust anywhere, I've used it in C/C++, Java, C# and Python pretty easily. [1] https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe-rust.html#calling-rust-functions-from-other-languages https://doc.rust-lang.org/book/second-edition/ch19-01-unsafe...
- Manishearth 9y ago
  There's a tool called cbindgen (https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen) that generates header files from rust code . It's a bit experimental. The reverse tool, bindgen (https://github.com/rust-lang-nursery/rust-bindgen/ https://github.com/rust-lang-nursery/rust-bindgen/) is more mature -- generates rust bindings from header files.
  - vvanders 9y ago
    Oh yeah, I should have mentioned bindgen, it's fantastic!
samjs 9y ago
I wrote on a similar topic [1], about writing system libraries in Rust with a C API and bindings to other languages. Which covers a bit of this, plus some of the sharp edges of FFI. Most of the trickier parts of it are covered in the FFI omnibus [2]. Also worth checking out is the cbindgen tool [3]. [1]: https://libpasta.github.io/blog/bindings/ https://libpasta.github.io/blog/bindings/ [2]: http://jakegoulding.com/rust-ffi-omnibus/ http://jakegoulding.com/rust-ffi-omnibus/ [3]: https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen

JackNichelson 9y ago

That's just stupid.

dikaiosune 9y ago

Awesome! It looks like they have some tests in the directory -- I'd be curious whether they're able to re-use musl's test suite some. I had an experiment a while ago to incrementally port musl to rust while keeping the test suite passing: https://github.com/anp/rusl https://github.com/anp/rusl. Haven't put any time into it in a while, but I think matching musl's program-facing behavior is still a good idea.

brianon99 9y ago

There are many projects recently titled like "xxx in Rust" or "xxx in Go", rewriting an existing software, seems to be built not because there is a problem in the existing software, but without a clear reason. For learning a new language, it can be a good practice. But if you hope the project to grow successfully I think there is no hope.

sanxiyn 9y ago
relibc is built to solve a problem in the existing software, namely newlib. It has a clear reason. Redox is an opearting system. An operating system needs a C library. Redox used to use newlib, but developers found it unsatisfying.