20 ms·

Baby Steps: Slowly Porting Musl to Rust

• quotemstr 10y ago
  I'm disappointed by the idea that something is semantic nonsense, but it's okay to do it anyway. The right line of code isn't "return usize::MAX", but "abort()". (I'd also accept "__builtin_unreachable()".) In general, if a program enters an impossible state, you need to stop running the program, not do something you know is wrong just so the program keeps working a while longer. That's why strcpy_s is much better than strlcpy. It's also really not fair to compare the legibility of Rust to the legibility of musl: whatever its other merits, musl is written in that old-school C style that makes one imagine that programs run faster when variable names are shorter and braces are omitted.
  • CUViper 10y ago
    Rust does have unreachable!() which panics.
    • dikaiosune 10y ago
      Panics are possible with #![no_std], but it does require that you supply a working implementation of panic_fmt, which I haven't gotten to yet (mostly because I was waiting on panic-as-abort to land which it recently did).
      • steveklabnik 10y ago
        NB: you still need panic_fmt with panic == abort; the default panic handler still prints a message, even though it doesn't go through with the unwinding.
      • CUViper 10y ago
        OK, another option is to make it an explicit infinite "loop{...}" which only exits by returning. The "0.." is effectively infinite anyway, because it will overflow freely in release mode. (But surely you'll hit an invalid memory page before that happens.)
        • dikaiosune 10y ago
          You're right, I think this version is a bit clearer: https://is.gd/lck8c7 https://is.gd/lck8c7
      • loeg 10y ago
        panic_fmt can be as simple as (C pseudocode): printf("..."); kill(getpid(), SIGABRT); while(true) /* In case kill(ABRT) returns, halt. */;
  • dikaiosune 10y ago
    This is a hobby project for my own edification, and maybe maybe to eventually show Rust's suitability for this kind of work. While I appreciate that the semantics of returning usize::MAX aren't correct, I'm also not an expert on C standard library implementations. Just a hobbyist writing a little bit about something I'm trying out. Potentially relevant is this section of the repo's README: https://github.com/dikaiosune/rusl#goals-and-non-goals https://github.com/dikaiosune/rusl#goals-and-non-goals EDIT: After a suggestion in a sibling here, here's a version which neither explicitly aborts (musl doesn't do abort or unreachable), nor uses an explicit return of usize::MAX: https://is.gd/lck8c7 https://is.gd/lck8c7 It doesn't autovectorize, but I suspect that with some work I could maybe convince LLVM to do so.
    • briansmith 10y ago
      IMO, it would be more interesting to see it done top-down, instead of bottom-up. That is, instead of starting with rewriting `strlen`, why not start with rewriting `getaddrinfo`? That would be a very interesting project because you could write `getaddrinfo` with a safe Rust API (i.e. one that doesn't require using `unsafe` to call it), and then wrap it in an `unsafe` function that exports the unsafe C `getaddrinfo` function. This could arguably then be an improvement on the musl code.
      • dikaiosune 10y ago
        I think it very well could be. However, I would imagine (not having looked at the problem in depth) that one would want to be able to use things like Vec<T> or CString from the Rust standard library when implementing a DNS resolver. Since the Rust standard library depends on libc, I'm not sure how one would properly allow the Rust symbols to replace the linked-in libc symbols while still depending on the rest of libc. Maybe it's not an issue? Maybe there's some magic which could be done? Nonetheless, I mostly started doing this to see if it could be done, not to prove any point (although I'd love it if a point is proven along the way), so a bottom-up approach with #![no_std] was the easiest way to prevent any sort of cyclic dependencies and/or linker issues.
        • 10y ago
  • nwmcsween 10y ago
    No you don't abort, aborting or trapping is of last effort. A shared library aborting will kill whatever it's linked to which allows zero chance of recovering. There are two sane ways to go about handling 'impossible states': The API is designed in a way that it returns error info somehow, a type system that tracks side effects in order to allow rolling back.
    • astrange 10y ago
      You can't roll back real side effects like network I/O; that's what it means to be a side effect. Plus crashing (on some OSes) dumps a lot of useful process state and the current stack, but if a function just returns some error code 50 there's not even a way to find which bits of its internals returned it. If you separate things by process, you can have your crash-only recovery and keep the rest of the system going.
      • dleslie 10y ago
        This is why "interrupted" and "incomplete" are often valid error states.
      • nwmcsween 10y ago
        You can roll back most real side effects like network I/O, console output it just depends on details such as protocol and framebuffer type. You also handle errors at the point of issue, not exceptions. The issue with separating by process is you have to flush the TLB on context switches.
    • deleted 10y ago
      [deleted]
  • revelation 10y ago
    It's really not fair to compare the legibility because that code is not at all doing the same. He's basically comparing his naive "one byte at a time" strlen to one at least processing at register size. The naive version looks just as good in C, frankly better, because what is this "0.." offset nonsense.
    • lmm 10y ago
      > The naive version looks just as good in C, frankly better, because what is this "0.." offset nonsense. I'm pretty sure "for i in 0.." is clearer (to a reader unfamiliar with either) than whatever it is you'd do in C. In C it would be something like "for(int i=0;;i++)", right?
      • pbsd 10y ago
        I'm pretty sure a simple while loop is better than either: size_t strlen(const char * s) { size_t index = 0; while(s[index] != 0) { index += 1; } return index; }
        • lmm 10y ago
          That's substantially harder to follow than the code in the article - you have to jump around to see where index changes and how that interacts with the loop condition.
          • matthewaveryusa 10y ago
            Just out of curiosity how would you rank strlen{1,2,3} in terms of difficulty to read? size_t strlen1(const char * s) { const char* it =s ; while(*it != '\0') { ++it; } return it - s; } size_t ptr_distance(const void *begin, const void* end) { return end - begin; } size_t strlen2(const char * s) { const char* it =s ; while(*it != '\0') { ++it; } return ptr_distance(s,it); } int is_null(const char* p) { return *p == '\0'; } const char* find_first(const char* s, int (*predicate)(const char*)) { while(1) { if(predicate(s)) { return s; } ++s; } } size_t strlen3(const char * s) { return ptr_distance(s,find_first(s,is_null)); }
  • deleted 10y ago
    [deleted]
  • _pmf_ 10y ago
    > The right line of code isn't "return usize::MAX", but "abort()". No, it's not. Assertions and aborts at the library level make a library completely unusable for embedded systems. The caller checks the result for plausibility and then decides to restart or simply not to perform the guarded action.
    • rkangel 10y ago
      There's a difference between handling all possible error cases (however unlikely) and handling impossible cases. The moment you hit the latter case you can no longer make any assumptions about the state of the device and an abort/assert is an entirely appropriate thing to do.
    • tremon 10y ago
      If the caller didn't verify the plausibility of the parameters before calling the function, why is it safe to assume the caller will verify the plausibility of the function result? Especially for embedded systems, I want the system to blow up on the developer's desk, not in the field, so I'd prefer to fail early.
      • nwmcsween 10y ago
        Because the caller shouldn't know specifics on what is valid for the callee unless the caller has a specific and different value of valid. You don't leak information from separate functions this is how you get brittle garbage code.
        • programmer_dude 10y ago
          > caller shouldn't know specifics on what is valid for the callee Why? How do you even use an API if you don't know what you can pass into it?
          • nwmcsween 10y ago
            For example strtol, you may pass whatever (implicitly nul terminated, which is bad) and it either returns a value or an error. I wasnt advocating _you_ don't know what to pass but that the caller doesn't include specifics of the callee.
• bitmadness 10y ago
  Wonderful work! Keep it up!
  • dikaiosune 10y ago
    Thanks!
• cpeterso 10y ago
  Has anyone experimented with a C-to-Rust translator? Google wrote a C-to-Go translator to aid their conversion of the Go compiler from C to Go. Carol Nichols gave a talk about her function-by-function, by-hand rewrite of the Zopfli compression library from C to Rust. I wonder how much of that could be automated, even if most of the resulting Rust code is in unsafe blocks. Carol's slides: https://github.com/carols10cents/rust-out-your-c-talk https://github.com/carols10cents/rust-out-your-c-talk
  • legulere 10y ago
    I guess the output would be very ugly as the C semantics are probably ugly to reproduce in rust. For instance arithmetic with unsigned types in C is defined to be wrapping, so you would need e.g. calls to wrapping_add() instead of + all the time.
    • pornel 10y ago
      I'd be OK with a shallow syntactic translator that only does the first laborious step of the conversion, without trying to preserve too much semantics. Rust with exact C semantics isn't going to be better than C, so I'd prefer readable code that is easier to read & refactor. I've converted a few utilities from C to Rust. Rewriting `void foo(int a)` to `fn foo(a:int)` is boring menial job. Only the next step of replacing C's pile'o'pointers with Rust idioms is fun.
• pron 10y ago
  While Rust can indeed provide great benefits over C in many respects (mostly security and safety, but also maintainability), rewriting "everything" in Rust, as the author says, is so monumental an effort that there is no chance of it happening, certainly not before a "better" language than Rust comes along (say, in another ten or fifteen years), and then what? Rewrite everything in that language? A far more feasible approach (though still very expensive) -- and probably a more "correct" one, even theoretically -- is to verify existing C code using some of the many excellent verification tools (from static analysis to white-box fuzzing and concolic testing) that exist for C[1]. It is more "correct" because, while perhaps not providing some of the other benefits, those tools can prove more properties than the Rust type system can, and such tools don't (yet) exist for Rust. Even if you want to write a green-field project today that absolutely must be correct, it's better to do so in C (or in some of the verified languages that compile to C, like SCADE) and use its powerful verification tools, than to write it in Rust. Verification tools are usually much more powerful than any language feature (even though Rust's borrowing system seems like it can prevent a lot of expensive real-world bugs). Of course, it would be optimal if we could do both, but the Rust ecosystem simply isn't there just yet. This is part of a more general problem, namely that rewriting code in new languages -- any new language -- is so costly (and often risky, too), that the benefits are rarely if ever commensurate with the effort. This is why good interop with legacy libraries is so important (something, I understand, Rust does well), often much more important than most language features. Rewriting code is often the worst possible way to increase confidence in a library/program in terms of cost/benefit, both because verification tools are often more powerful than any language feature, as well as because new bugs are introduced -- either in the code or in the compiler (bugs that won't be uncovered by the usually incomplete legacy test suite, but would break some of the millions of clients). Rewriting is usually worth the effort if the codebase is about to undergo a significant change anyway, but almost never to increase confidence in the existing functionality. EDIT: Clarification: if you're not going to use verification tools in your new project, anyway, than obviously using Rust would give you much better guarantees than C. [1]: C and Java are the languages with the best collection of powerful verification tools (maybe Ada as well, but there's far less Ada code out there than C or Java).
  • fpoling 10y ago
• nikital 10y ago
  I wonder what is the benefit of rewriting the libc in Rust, as most of the functions are actually unsafe. Implementing memcpy in Rust won't make memcpy any safer. As far as I understand, the main selling point of Rust is safety, so if we don't get any safety, why bother? Am I missing something? (Other than having some fun with Rust, which is obviously welcome :) )
  • mwcampbell 10y ago
    For simple functions like memcpy, I think you're right. But it may be worthwhile to implement higher-level libc features like the DNS resolver in Rust.
  • pcwalton 10y ago
    I came to this thread hoping to answer this exact question. :) There isn't a huge benefit to writing things like strlen in Rust. But a lot of libc is big, complicated things like the DNS resolution logic in gethostbyname. These are also where we find a huge number of vulnerabilities in popular libcs. The internal logic of functions like gethostbyname should be able to profitably take advantage of Rust's safety features.
    • dikaiosune 10y ago
      For this project, the advantage to rewriting strlen in Rust is for my learning and to test my Frankenstein-ian link-time munging, not because Rust is better at unsafely dereferencing random pointers (C's got that pretty well covered). I do hope to soon move on to the more meaty bits and show ways in which Rust can improve safety, but I wanted to get my initial thoughts and results written down so I could reference back to the post later. Frankly did not expect to see it here :).
    • chubot 10y ago
      Also, stuff like glob(), fnmatch(), regcomp()/regexec(), should be better in Rust. They involve a bit of parsing.
      • 10y ago
• dman 10y ago
  Any plans for rust to switch to an allocator written in Rust?
  • steveklabnik 10y ago
    On nightly, we have the ability to swap out allocators. But I'm not aware of an allocator written in Rust that's as high-quality as jemalloc yet; it'd have to be pretty good before we'd move over. I'm not aware of any serious effort to write one, yet.
• matt_wulfeck 10y ago
  The first example uses a "unsafe" method. I'm not very familiar with rust, but isn't this an inherently "bad thing"?
  • stirner 10y ago
    All C functions are considered unsafe, because they don't have the Rust safety guarantees. Every function in the normal libc crate is marked unsafe.
  • dikaiosune 10y ago
    Not an inherently bad thing, really. The unsafe keyword has two meanings: 1. An unsafe block tells the compiler "trust me, I know this is unsafe and I know how to do it safely" 2. An unsafe function tells the compiler "this is unsafe! it needs to be put inside another unsafe function or an unsafe block" So an unsafe block essentially terminates an unsafe call chain. There are a bunch of things which are inherently marked as unsafe in Rust, usually because the compiler can't reason about them, like dereferencing a raw pointer as opposed to a borrow-checked Rust reference. That's what calls for an unsafe function in the case of strlen -- to only iterate over the string once we just have to assume that the pointer points to valid memory and is null-terminated, so like a lot of things in C, we just cross our fingers and go for it. The fact that Rust knows this could fail miserably is why it needs to have an unsafe block.
    • xienze 10y ago
      > 1. An unsafe block tells the compiler "trust me, I know this is unsafe and I know how to do it safely" Right, so aren't you giving up the biggest advantage of Rust, that your code will be safe by virtue of it getting through the Rust compiler's safety checking?
      • dikaiosune 10y ago
        Yes, although there are other advantages to using Rust (about halfway down there's a "What does Rust offer here?" section that discusses in more detail). Frankly, the POSIX C standard library does not seem to have been designed with safety in mind (I don't think this requires an expert eye to see). So writing it in Rust is going to require a number of compromises on Rust's safety. Also, if I were taking this more seriously, the wildly unsafe Rust code would just be a beach head for a bottom-up conversion of a project. You have a project which uses strlen? Great, now use this Rust version. Now incrementally rewrite the strlen-using code in Rust. Now change strlen to use a Rust CString...and so on. Each time you identify a new module boundary, rewrite its dependencies in Rust, and then consume the module while preserving it's outward facing interfaces.
• fpgaminer 10y ago
  As a corollary to writing the C standard library in Rust, are there any plans by the Rust devs to remove the C standard library dependency itself? That, to me, is just as interesting of a proposition.
  • steveklabnik 10y ago
    So, with libcore, you don't have a libc dependency. With libstd, you do. There has been some interest in making it easier to port libstd away from it, but it would likely be a cross target, like musl itself, rather than an official move. We'll see what the future holds!
  • pcwalton 10y ago
    It's not a high priority task, no. (Though it would be quite interesting to see it done.) On Windows and Mac OS X you shouldn't really avoid C libraries (kernel32.dll and libSystem.dylib respectively) because the syscall interface is considered a private API. (You can avoid it technically, and apps sometimes do, but Microsoft and Apple would be within their rights to break us at any time if we did that.)
    • ben0x539 10y ago
      On Windows, does Rust currently require/use (nontrivial features of) the C standard library in addition to the C "win32"/"winapi"/w/e libraries?
      • heinrich5991 10y ago
        The math library AFAIK, in order to provide the same wrong results that the MSVC implementation gives.
        • adrianratnapala 10y ago
          Heh? Can you point to a description of the wrongness of MSVC maths. I noticed that they were a bit slow to support infinities and NaN, but that is not the same thing a being wrong.
• kamli 10y ago
  Thanks! Sorry for stealing your name! Should have done a GitHub search beforehand. Yeah, I've poked around the various things for the weak linkage, but I haven't yet learned enough here to properly replicate the effects of the weak_alias macro in musl.
  • dikaiosune 10y ago
    So... I wrote this comment on the the rust subreddit in reply to someone else who also had a repo named "rusl" where they were planning to do something similar. Not sure how it's relevant here, though. Also, if you're going to quote me it'd be nice if you mark it as a quote and source it :).
    • dang 10y ago
      Spammers have been copying comments from other places onto HN lately, presumably to build up karma. We ban such accounts. If anyone notices evidence of this please flag the comments and/or let us know at hn@ycombinator.com.
      • dikaiosune 10y ago
        Is there some market value to HN karma? It's even less useful than gold in an MMO!
        • pygy_ 10y ago
          Things like downvote and flag privileges would allow botnets to censor HN and spin comment threads their way.
• bombless 10y ago
  Actually I'd prefer to give a `unreachable!()` hint to reader and compiler, for the `strlen` example.
• scotty79 10y ago
  For me coolest part is the algorithm for determining if word contains zero byte. #define ALIGN (sizeof(size_t)) #define ONES ((size_t)-1/UCHAR_MAX) #define HIGHS (ONES * (UCHAR_MAX/2+1)) #define HASZERO(x) ((x)-ONES & ~(x) & HIGHS) Assuming sizeof(size_t) is 4 and UCHAR_MAX is FF... It takes FF FF FF FF (by casting -1 to unsigned size_t), divides it by FF, getting 01 01 01 01 (ONES). Then gets 80 80 80 80 (HIGHS) by multiplying 01 01 01 01 by 80 (which is gets from (FF+1)/2). Once it has these two values it tests which bytes are lower than 80 by doing ~(x) & HIGHS and keeps result in highest bit of each byte and zeroes lower ones. Other part is subtracting ONES from (x) which results in highest bit set in each byte either because (x) had more than 80 in that byte or (x) having zero in that byte and borrowing from higher byte (underflow in case of highest byte). ANDing those two cases rules out highest bit set due to (x) byte being more than 80. What's left is all zeroes if and only if all bytes of (x) were non-zero. Funny thing is that if borrowing from higher byte happens (because some byte was zero) it might mess up test for that higher byte but it doesn't matter because you will already detect zero in lower byte so you don't care about test on higher bytes then. If what I wrote above is unclear then try here: http://stackoverflow.com/a/34643025/166921 http://stackoverflow.com/a/34643025/166921 This is algorithm that checks if word has a byte that is less than given value (1 in that case, but surely valid up to 7F ... maybe more by some magic? probably not...). Also works for different lengths of "words" and "bytes" provided that whole number of "bytes" fits in a "word".
  • scotty79 10y ago
    Thus I give you my first Rust program ;) // needs in main.rs or lib.rs #![feature(const_fn)] use std::os::raw::c_schar; use std::mem; const ALLSET:usize = (-1_isize) as usize; const MAXCHR:usize = u8::max_value() as usize; const ONES:usize = ALLSET / MAXCHR; const HIGHS:usize = ONES * ((MAXCHR+1)/2); const ALIGN:usize = 8; // mem::size_of::<usize>(); // mem::size_of<T>() is not const fn (yet?) pub unsafe extern "C" fn strlen(s: *const c_schar) -> usize { let mut i:usize = 0; loop { if *s.offset(i as isize) == 0 { return i; } i += 1; if (i % ALIGN) == 0 { break; } } loop { let v = *mem::transmute::<*const c_schar, *const usize>(s.offset(i as isize)); if v.wrapping_sub(ONES) & !v & HIGHS != 0 { break; } i += ALIGN; } loop { if *s.offset(i as isize) == 0 { return i; } i += 1; } }