21 ms·

How I Wrote a Modern C++ Library in Rust

406 points by hsivonen 8y ago

pedrocr 8y ago
Is there some kind of target or plan for converting C/C++ to Rust in Firefox? Or is it just happening as there are people interested in working on parts of the stack? These progress numbers are very interesting: https://twitter.com/eroc/status/1061049330574884864 https://twitter.com/eroc/status/1061049330574884864 and I was wondering how directed the effort was.
- robin_reala 8y ago
  It’s the Oxidation project: https://wiki.mozilla.org/Oxidation https://wiki.mozilla.org/Oxidation
  - simfoo 8y ago
    So the uncrustification is done by adding rust :P
    - Tade0 8y ago
      It's a protective layer.
      - mitchty 8y ago
        So like aluminum then.
- hsivonen 8y ago
  Not a specific target of having to replace C++ for the sake of replacing C++. The way Rust code gets added includes: * A new feature needs an identifiable library, so the new library can be written in Rust to begin with. (Example: U2F token USB integration.) * Old code needs a rewrite anyway, so the rewrite can be in Rust. (Example: Character encoding converters.) * Servo has proven a component, so it makes sense to bring it over. (Examples: Stylo and WebRender) * History of vulnerabilities in code that was replaced. (Example: MP4 metadata parser)
  - Already__Taken 8y ago
    Is anyone aware of secondary effects this has had? e.g. removed C++ code that has later found to have bugs, or newly re-written crates now more useful to the wider community than the same code locked up in C++.
    - hsivonen 8y ago
      > e.g. removed C++ code that has later found to have bugs The article links to a longer article (https://hsivonen.fi/encoding_rs/ https://hsivonen.fi/encoding_rs/) about encoding_rs. The longer article mentions a bug that got fixed in Firefox ESR after the code had been replaced with encoding_rs in non-ESR Firefox. (I wrote the bug, too, though.) > or newly re-written crates now more useful to the wider community than the same code locked up in C++. encoding_rs is an example of a crate developed for Firefox but also developed as a crates.io crate from the start. ripgrep is probably the best-known Rust-only app that uses encoding_rs. Since Visual Studio Code bundles ripgrep, I believe Microsoft shipped encoding_rs before Mozilla did!
      - vtesucks 8y ago
        In terms of deploying rust directly to make money- Microsoft is probably the leader right now with actix used in azure iot
        polskibus 8y ago
        Did you mean this actix? https://github.com/actix/actix https://github.com/actix/actix Is it mature? Can you shed more light on how is it used in production?
        steveklabnik 8y ago
        They're referring to https://news.ycombinator.com/item?id=17433142 https://news.ycombinator.com/item?id=17433142 I don't believe that it uses Actix, though. Actix was created by and is maintained by a Microsoft employee.
- kannanvijayan 8y ago
  As far as I know there is no overarching plan, but there are many people who do hold that motivation - some are more aggressive about their intent and others are more lax, and yet others are skeptical - as with any community of developers. The broad sentiment seems to me that it's a good thing to move over, as pragmatism allows. Personally I'm a converted skeptic - I had my doubts about the language and my initial stabs at it left me somewhat frustrated - but as I've grown to internalize its semantics and behaviour - the benefits in terms of safety and clarity of intent and optimization potential are clear (e.g. aliasing semantics are just so much better). A mix of factors enter into whether a component moves over or not, including the views of the developer in question, the complexity of the API boundary between the main codebase and the subcomponent, and the complexity of the component logic itself.
SloopJon 8y ago
If I'm reading this correctly, the "modern C++" part of the library is in fact supplied by a C++ wrapper around a C-style API.
- hsivonen 8y ago
  Correct. The Rust API is recreated in C++ using the corresponding C++ facilities with a C API in between.
  - lightedman 8y ago
    "with a C API in between." I smell vulnerabilities miles away with that sort of implementation. Hope the Rust programmers remember basic garbage collection in C, since C itself doesn't have it automated.
    - kibwen 8y ago
      Just because it's using a C-style API doesn't mean there needs to be any C code involved at all (I don't know if there is in this case, but in general it's not necessary). You have one side say "hey, pretend this code I've got here is C", and the other side says "hey, let me call this function that I think is C". Every language has a way of calling C functions, so you don't even necessarily need a C compiler.
      - hsivonen 8y ago
        > I don't know if there is in this case In this case, there indeed is no .c compilation unit between the C++ and Rust code that see each other via C linkage.
    - msbarnett 8y ago
      You should really consider reading the article before commenting on it. The mechanisms for deleting heap memory obtained from Rust are discussed in depth.
bluGill 8y ago
I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format. It doesn't need to be complete (which is to say I'm fine with having to use , just enough that 90% of my program can be written/rewritten in whatever language makes sense and used in the other without having to drop to C. For starters it needs to have classes (probably using PIMPL like things internally by default). It needs to have some sort of error handling. It needs to support some basic data like std::vector (but they can start from scratch). Edit: my fingers typed API not ABI first...
- tolle 8y ago
  Something like https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/ilec/ilecmain.htm https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/... ?
- ChrisSD 8y ago
  Reminds me of COM.
  - bluGill 8y ago
    Maybe, but COM's reputation is it is too complex. From what I can tell the reputation is well deserved.
    - pjc50 8y ago
      Much of COM's horror comes from being an OO interop system designed to work in non-OO C. All sorts of manual reference counting. If you use COM objects in C# it's much less horrifying, although there's a certain amount of "body horror" involved in firing up one whole windows application inside another one. You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.
      - maxxxxx 8y ago
        Working with VB made COM really nice. Both for writing COM servers and consuming them. C# is actually a step back because it doesn't do the reference counting right and instead relies on GC to free objects.
the_mitsuhiko 8y ago
I have started it more than once but gave up on the sheer complexity involved as many times but I really would love to have attributes to auto generate a C ABI from a Rust API alongside headers and then high level python and C++ bindings to it. With the recent improvements to wasm-bindgen I looked into if stuff can be repurposed there but it seems very custom crafted sadly. If someone is interested in that as well, maybe there are some ways to join forces.
- anp 8y ago
  I’ve longed for something similar in writing JavascriptCore bindings. Something lighter weight than swig but which still has an IR for describing the full API surface for codegen plugins would be immensely useful.
- Sean1708 8y ago
  Are you aware of cbindgen[0]? It's not exactly what you want, but it's certainly far closer than wasm-bindgen is. [0]: https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen
  - detaro 8y ago
    He's published some tooling using it: https://blog.sentry.io/2017/11/14/evolving-our-rust-with-milksnake https://blog.sentry.io/2017/11/14/evolving-our-rust-with-mil...
  - the_mitsuhiko 8y ago
    We are using that atm but it requires manually writing a cabi.
  - rhodysurf 8y ago
    It still requires you expose C api, instead of it being generated automatically with proc macros or something.
kyberias 8y ago
Sounds like a huge effort just to get Rust in. I predict huge problems for the Firefox program if they continue this mixing. It's a huge added complexity.
- simias 8y ago
  I mean developing Rust in the first place is probably a huge added complexity on its own, clearly they're not doing this on a whim. There's also probably a huge upfront cost that's going to pay off as more and more Rust code make it into Firefox.
- kibwen 8y ago
  It's hardly any more intrinsic complexity than adding a C library to a C++ project.
- kbd 8y ago
  > ... just to get Rust in You make it sound like they're just cargo culting a shiny new language when in fact they invented the language in the first place to solve their complexity problems with C++.
cryptonector 8y ago
C ABI to the rescue! That may be the one thing left of C in 50 years' time.
- MaxBarraclough 8y ago
  A dead lingua franca, Like Latin. Could happen, but I suspect embedded programmers will stick with C forever.
  - kccqzy 8y ago
    This is one reason I don't really like embedded programming. There seems to be generally a fear of newer systems languages like Rust and a fear of newer features in C++. An embedded programmer told me he doesn't use the C++ STL because he doesn't want functions allocating stuff on his back. Doesn't seem very convincing as you can always redefine the global operator new.
    - kstenerud 8y ago
      It's not a problem of how things are allocated, but rather when. In an embedded environment, you need to be very aware of what is being used at any given time and how, and complex templates like in STL have a tendency to allocate and deallocate in ways that can be difficult to follow, or might have a worst case memory footprint that blows through your RAM.
      - foundry27 8y ago
        Not disagreeing with your point about STL classes having a tendency to make egregious allocations (heck, a lot of them are fundamentally designed around using allocators for pretty much everything), but the fact that they’re often complicated templates is totally orthogonal to that, and I think it’s disingenuous to say that there’s a causal relationship between the two. If anything, templating encourages statically-verifiable, compiler-enforced code invariants and outputs, which can be pretty darn nice for embedded.
    - 8y ago
rokob 8y ago
The epilog alone was worth the price of admission. Having one compiler tell the other what to do via code generation is a great way around the lack of ABI compatibility.

406 points by hsivonen 8y ago

Is there some kind of target or plan for converting C/C++ to Rust in Firefox? Or is it just happening as there are people interested in working on parts of the stack? These progress numbers are very interesting: https://twitter.com/eroc/status/1061049330574884864 https://twitter.com/eroc/status/1061049330574884864 and I was wondering how directed the effort was.

robin_reala 8y ago
It’s the Oxidation project: https://wiki.mozilla.org/Oxidation https://wiki.mozilla.org/Oxidation
- simfoo 8y ago
  So the uncrustification is done by adding rust :P
  - Tade0 8y ago
    It's a protective layer.
    - mitchty 8y ago
      So like aluminum then.
hsivonen 8y ago
Not a specific target of having to replace C++ for the sake of replacing C++. The way Rust code gets added includes: * A new feature needs an identifiable library, so the new library can be written in Rust to begin with. (Example: U2F token USB integration.) * Old code needs a rewrite anyway, so the rewrite can be in Rust. (Example: Character encoding converters.) * Servo has proven a component, so it makes sense to bring it over. (Examples: Stylo and WebRender) * History of vulnerabilities in code that was replaced. (Example: MP4 metadata parser)
- Already__Taken 8y ago
  Is anyone aware of secondary effects this has had? e.g. removed C++ code that has later found to have bugs, or newly re-written crates now more useful to the wider community than the same code locked up in C++.
  - hsivonen 8y ago
    > e.g. removed C++ code that has later found to have bugs The article links to a longer article (https://hsivonen.fi/encoding_rs/ https://hsivonen.fi/encoding_rs/) about encoding_rs. The longer article mentions a bug that got fixed in Firefox ESR after the code had been replaced with encoding_rs in non-ESR Firefox. (I wrote the bug, too, though.) > or newly re-written crates now more useful to the wider community than the same code locked up in C++. encoding_rs is an example of a crate developed for Firefox but also developed as a crates.io crate from the start. ripgrep is probably the best-known Rust-only app that uses encoding_rs. Since Visual Studio Code bundles ripgrep, I believe Microsoft shipped encoding_rs before Mozilla did!
    - vtesucks 8y ago
      In terms of deploying rust directly to make money- Microsoft is probably the leader right now with actix used in azure iot
      - polskibus 8y ago
        Did you mean this actix? https://github.com/actix/actix https://github.com/actix/actix Is it mature? Can you shed more light on how is it used in production?
        steveklabnik 8y ago
        They're referring to https://news.ycombinator.com/item?id=17433142 https://news.ycombinator.com/item?id=17433142 I don't believe that it uses Actix, though. Actix was created by and is maintained by a Microsoft employee.
kannanvijayan 8y ago
As far as I know there is no overarching plan, but there are many people who do hold that motivation - some are more aggressive about their intent and others are more lax, and yet others are skeptical - as with any community of developers. The broad sentiment seems to me that it's a good thing to move over, as pragmatism allows. Personally I'm a converted skeptic - I had my doubts about the language and my initial stabs at it left me somewhat frustrated - but as I've grown to internalize its semantics and behaviour - the benefits in terms of safety and clarity of intent and optimization potential are clear (e.g. aliasing semantics are just so much better). A mix of factors enter into whether a component moves over or not, including the views of the developer in question, the complexity of the API boundary between the main codebase and the subcomponent, and the complexity of the component logic itself.

SloopJon 8y ago

If I'm reading this correctly, the "modern C++" part of the library is in fact supplied by a C++ wrapper around a C-style API.

hsivonen 8y ago
Correct. The Rust API is recreated in C++ using the corresponding C++ facilities with a C API in between.
- lightedman 8y ago
  "with a C API in between." I smell vulnerabilities miles away with that sort of implementation. Hope the Rust programmers remember basic garbage collection in C, since C itself doesn't have it automated.
  - kibwen 8y ago
    Just because it's using a C-style API doesn't mean there needs to be any C code involved at all (I don't know if there is in this case, but in general it's not necessary). You have one side say "hey, pretend this code I've got here is C", and the other side says "hey, let me call this function that I think is C". Every language has a way of calling C functions, so you don't even necessarily need a C compiler.
    - hsivonen 8y ago
      > I don't know if there is in this case In this case, there indeed is no .c compilation unit between the C++ and Rust code that see each other via C linkage.
  - msbarnett 8y ago
    You should really consider reading the article before commenting on it. The mechanisms for deleting heap memory obtained from Rust are discussed in depth.

bluGill 8y ago

I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format. It doesn't need to be complete (which is to say I'm fine with having to use , just enough that 90% of my program can be written/rewritten in whatever language makes sense and used in the other without having to drop to C. For starters it needs to have classes (probably using PIMPL like things internally by default). It needs to have some sort of error handling. It needs to support some basic data like std::vector (but they can start from scratch). Edit: my fingers typed API not ABI first...

tolle 8y ago
Something like https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/ilec/ilecmain.htm https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/... ?
ChrisSD 8y ago
Reminds me of COM.
- bluGill 8y ago
  Maybe, but COM's reputation is it is too complex. From what I can tell the reputation is well deserved.
  - pjc50 8y ago
    Much of COM's horror comes from being an OO interop system designed to work in non-OO C. All sorts of manual reference counting. If you use COM objects in C# it's much less horrifying, although there's a certain amount of "body horror" involved in firing up one whole windows application inside another one. You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.
    - maxxxxx 8y ago
      Working with VB made COM really nice. Both for writing COM servers and consuming them. C# is actually a step back because it doesn't do the reference counting right and instead relies on GC to free objects.

the_mitsuhiko 8y ago

I have started it more than once but gave up on the sheer complexity involved as many times but I really would love to have attributes to auto generate a C ABI from a Rust API alongside headers and then high level python and C++ bindings to it. With the recent improvements to wasm-bindgen I looked into if stuff can be repurposed there but it seems very custom crafted sadly. If someone is interested in that as well, maybe there are some ways to join forces.

anp 8y ago
I’ve longed for something similar in writing JavascriptCore bindings. Something lighter weight than swig but which still has an IR for describing the full API surface for codegen plugins would be immensely useful.
Sean1708 8y ago
Are you aware of cbindgen[0]? It's not exactly what you want, but it's certainly far closer than wasm-bindgen is. [0]: https://github.com/eqrion/cbindgen https://github.com/eqrion/cbindgen
- detaro 8y ago
  He's published some tooling using it: https://blog.sentry.io/2017/11/14/evolving-our-rust-with-milksnake https://blog.sentry.io/2017/11/14/evolving-our-rust-with-mil...
- the_mitsuhiko 8y ago
  We are using that atm but it requires manually writing a cabi.
- rhodysurf 8y ago
  It still requires you expose C api, instead of it being generated automatically with proc macros or something.

kyberias 8y ago

Sounds like a huge effort just to get Rust in. I predict huge problems for the Firefox program if they continue this mixing. It's a huge added complexity.

simias 8y ago
I mean developing Rust in the first place is probably a huge added complexity on its own, clearly they're not doing this on a whim. There's also probably a huge upfront cost that's going to pay off as more and more Rust code make it into Firefox.
kibwen 8y ago
It's hardly any more intrinsic complexity than adding a C library to a C++ project.
kbd 8y ago
> ... just to get Rust in You make it sound like they're just cargo culting a shiny new language when in fact they invented the language in the first place to solve their complexity problems with C++.

cryptonector 8y ago

C ABI to the rescue! That may be the one thing left of C in 50 years' time.

MaxBarraclough 8y ago
A dead lingua franca, Like Latin. Could happen, but I suspect embedded programmers will stick with C forever.
- kccqzy 8y ago
  This is one reason I don't really like embedded programming. There seems to be generally a fear of newer systems languages like Rust and a fear of newer features in C++. An embedded programmer told me he doesn't use the C++ STL because he doesn't want functions allocating stuff on his back. Doesn't seem very convincing as you can always redefine the global operator new.
  - kstenerud 8y ago
    It's not a problem of how things are allocated, but rather when. In an embedded environment, you need to be very aware of what is being used at any given time and how, and complex templates like in STL have a tendency to allocate and deallocate in ways that can be difficult to follow, or might have a worst case memory footprint that blows through your RAM.
    - foundry27 8y ago
      Not disagreeing with your point about STL classes having a tendency to make egregious allocations (heck, a lot of them are fundamentally designed around using allocators for pretty much everything), but the fact that they’re often complicated templates is totally orthogonal to that, and I think it’s disingenuous to say that there’s a causal relationship between the two. If anything, templating encourages statically-verifiable, compiler-enforced code invariants and outputs, which can be pretty darn nice for embedded.
  - 8y ago

rokob 8y ago

The epilog alone was worth the price of admission. Having one compiler tell the other what to do via code generation is a great way around the lack of ABI compatibility.