9 ms·

Bzip2 crate switches from C to 100% Rust

336 points by Bogdanp 1y ago

dralley 1y ago
How realistic is it for the Trifecta Tech implementation to start displacing the "official" implementation used by linux distros, which hasn't seen an upstream release since 2019? Fedora recently swapped the original Adler zlib implementation with zlib-ng, so that sort of thing isn't impossible. You just need to provide a C ABI compatible with the original one.
- masfuerte 1y ago
  They do provide a compatible C ABI. Someone "just" needs to do the work to make it happen.
- rlpb 1y ago
  > You just need to provide a C ABI compatible with the original one. How does this interact with dynamic linking? Doesn't the current Rust toolchain mandate static linking?
  - sedatk 1y ago
    No. https://doc.rust-lang.org/reference/linkage.html#r-link.dylib https://doc.rust-lang.org/reference/linkage.html#r-link.dyli...
  - arcticbull 1y ago
    Rust lets you generate dynamic C-linkage libraries. Use crate-type=["cdylib"]
  - nicoburns 1y ago
    Dynamic linking works fine if you target the C ABI.
  - timeon 1y ago
    You can use dynamic linking in Rust with C ABI. Which means going through `unsafe` keyword - also known as 'trust me bro'. Static linking directly to Rust source means it is checked by compiler so there is no need for unsafe.
  - bluGill 1y ago
    Rust cannot dynamic link to rust. It can dynamic link to C and be dynamicly linked by C - if you combine the two you can cheat but it is still C that you are dealing with not rust even if rust is on both sides.
    - deleted 1y ago
      [deleted]
    - mjevans 1y ago
      It might help to think of it as two IPC 'servers' written in rust that happen to have the C ABI interfaces as their communication protocol.
    - filmor 1y ago
      Rust can absolutely link to Rust libraries dynamically. There is no stable ABI, so it has to be the same compiler version, but it will still be dynamically linked.
  - conradev 1y ago
    Rust importing Rust must be statically linked, yes. You can statically link Rust into a dynamic library that other libraries link to, though!
  - alxhill 1y ago
    The commenters below are confusing two things - Rust binaries can be dynamically linked, but because Rust doesn’t have a stable ABI you can’t do this across compiler versions the way you would with C. So in practice, everything is statically linked.
    - eru 1y ago
      Static linking also produces smaller binaries and lets you do link-time-optimisation.
      - emidln 1y ago
        Static linking doesn't produce smaller binaries. You are literally adding the symbols from a library into your executable rather than simply mentioning them and letting the dynamic linker figure out how to map those symbols at runtime. The sum size of a dynamic binary plus the dynamic libraries may be larger than one static linked binary, but whether that holds for more static binaries (2, 3, or 100s) depends on the surface area your application uses of those libraries. It's relatively common to see certain large libraries only dynamically linked, with the build going to great lengths to build certain libraries as shared objects with the executables linking them using a location-relative RPATH (using the $ORIGIN feature) to avoid the extra binary size bloat over large sets of binaries.
        IshKebab 1y ago
        Static linking does produce smaller binaries when you bundle dependencies. You're conflating two things - static vs dynamic linking, and bundled vs shared dependencies. They are often conflated because you can't have shared dependencies with static linking, and bundling dynamically linked libraries is uncommon in FOSS Linux software. It's very common on Windows or with commercial software on Linux though.
        guappa 1y ago
        You know how the page cache works? Static linking makes it not work. So 3000 processes won't share the same pages for the libc but will have to load it 3000 times.
- wmf 1y ago
  Ubuntu is using Rust sudo so it's definitely possible.
  - egorfine 1y ago
    It's not. At least not yet. It's planned for 25.10, but thankfully sudo will be packaged and available for a few versions after that as promised [1]. [1] https://discourse.ubuntu.com/t/adopting-sudo-rs-by-default-in-ubuntu-25-10/60583 https://discourse.ubuntu.com/t/adopting-sudo-rs-by-default-i...
- tiffanyh 1y ago
  I think that is the goal of uutils. https://uutils.github.io/ https://uutils.github.io/
  - cocoa19 1y ago
    I hope some are improved too. The performance boost in tools like ripgrep and tokei is insane compared to the tools they replace (grep and cloc respectively).
    - egorfine 1y ago
      I absolutely hate it when people call their tools a "replacement" for something that is part of core standards, something, that did just fine for decades. ripgrep is an excellent tool. But it's not a grep replacement. And should not ever be.
      - mprovost 1y ago
        The GNU utils were a replacement for the BSD utils which were a replacement for the original AT&T utils. Every replacement added new functionality and improvements, and every time someone complained that they didn't stick closer to the thing they replaced. Looking specifically at grep, there used to be new versions like egrep and fgrep that added functionalities beyond standard grep's, but those were eventually pulled into "standard" grep (GNU or BSD). If we stuck with standards we'd all still be using the Bourne shell. The GNU utilities have been around long enough that they feel like the standard now, but I'm glad that we're coming into a new phase of innovation in command-line utilities. And this didn't start with Rust - the new generation of search utilities started with ack (Perl) and then ag (C).
        egorfine 1y ago
        > which were a replacement for the You have a point here. I have to agree. "X but rewritten in Z" is a terrible marketing, though. Makes me instantly want to hate the tool and its authors. (Love rust. Hate the vibe).
- deknos 1y ago
  i wait until they come to the hard stuff like awk, sed and grep.
  - GuB-42 1y ago
    ripgrep is one of the best grep replacement you can find, maybe even the best, and also one of the most famous Rust projects. I don't know of a sed equivalent, but I guess that would be easy to implement as Rust has good regex support (see ripgrep), and 90%+ of sed usage is search-and-replace. The other commands don't look hard to implement and because they are not used as much, optimizing these is less of a priority. I don't know about awk, it is a full programming language, but I guess it is far from an impossible task to implement. Now the real hard part is making a true, bug-for-bug compatible replacement of the GNU version of these tools, but while good to have, it is not strictly necessary. For example, Busybox is very popular, maybe even more so than GNU in terms of number of devices, and it has its own (likely simplified) version of grep, sed and awk.
    - scns 1y ago
      There is sd, not a drop in replacement though. https://github.com/chmln/sd https://github.com/chmln/sd
  - egorfine 1y ago
    What would be the point?
- kpcyrd 1y ago
  I briefly looked a this and there's already cargo-c configuration, which is good, but it's currently namespaced differently, so it won't get automatically detected by C programs as `libbz2`: https://github.com/trifectatechfoundation/libbzip2-rs/blob/8214be22d32e91d31a122409eba5f3c67dc80659/libbz2-rs-sys-cdylib/Cargo.toml#L30-L39 https://github.com/trifectatechfoundation/libbzip2-rs/blob/8... I'm not familiar enough with the symbols of bzip2 to say anything about ABI compatibility. I have a toy project to explore things like that, but it's difficult to set aside the amount of time needed to maintain an implementation of the GNU operating system. I would welcome pull requests though: https://github.com/kpcyrd/platypos https://github.com/kpcyrd/platypos
- Pesthuf 1y ago
  If it hasn't seen an upstream release since 2019, doesn't that mean the implementation is just... finished? Maybe there's no more bugs to fix and features to add. And in that case, I don't see what's wrong with it.
  - LinusU 1y ago
    Isn't 10-15% faster compression, and 5-10% faster decompression, a very nice "feature"? > [...] doesn't that mean the implementation is just... finished? I don't think that it _necessarily_ means that, e.g. all projects that haven't had a release since 2019 aren't finished? Probably most of them are simply abandoned? On the other hand, a finished implementation is certainly a _possible_ explanation for why there have been no releases. In this specific case, there are a handful of open bugs on their issue tracker. So that would indicate that the project isn't finished. ref: https://sourceware.org/bugzilla/buglist.cgi?product=bzip2 https://sourceware.org/bugzilla/buglist.cgi?product=bzip2
    - restalis 1y ago
      "Isn't 10-15% faster compression, and 5-10% faster decompression, a very nice «feature»?" Although much code can be optimized to get it to run 10-15% faster, if that comes at the expense of legibility then such "feature" get rejected nowadays. Translating an existing codebase into a language that makes things more difficult¹ and (because of that) has (and most likely will have) fewer engineers willing to working in it looks very much akin to applying legibility-affecting optimizations to me. ¹ ...and not only. I've already described in other comments some of the issues I see with it as a software development tool, here https://news.ycombinator.com/item?id=31565024 https://news.ycombinator.com/item?id=31565024 here https://news.ycombinator.com/item?id=33390634 https://news.ycombinator.com/item?id=33390634 and here https://news.ycombinator.com/item?id=42241516 https://news.ycombinator.com/item?id=42241516
      - LinusU 1y ago
        > Although much code can be optimized to get it to run 10-15% faster, if that comes at the expense of legibility then such "feature" get rejected nowadays. Makes sense, and I'd probably make the same call if I was a maintainer and someone submitted a patch which increased performance at the cost of maintainability... > Translating an existing codebase into a language that makes things more difficult¹ and (because of that) has (and most likely will have) fewer engineers willing to working in it looks very much akin to applying legibility-affecting optimizations to me. Here I have to personally disagree. I think that Rust is easier than both C and C++. Especially when coming into an already existing project. The chance of me contributing to a Rust project is higher than to a C project, because I feel more comfortable in knowing that my code is "correct". It's also easier to match the "style" since most Rust projects follow the same structures and patterns, whereas in C it can vary wildly. E.g. I contributed my first feature to Valkey (Redis fork, C codebase) recently, and figuring out how the tests worked took me quite some time. In fact, in the end I couldn't figure out how to run a specific test so I just ran all tests and grepped the output for my test name. But the tests take tens of minutes to run so this was sub-optimal. On the other hand, 99% of all Rust projects use `cargo test`, and to run a single test I can just click the play button in my editor (Zed) that shows up next to the test (or `cargo test "testname"`). (with this said, I think that Valkey is a really well structured code base! Great work on their part) Anyhow, this is just to illustrate my experience. I'm sure that for someone more used to C or C++ they would be more productive in that. And I could go on for ages on all the features that make me miss Rust every day at work when I have to work in other languages, especially algebraic data types!
_zagj 1y ago
[flagged]
- jeffbee 1y ago
  You should of course verify these results in your scenario. However, I somewhat doubt that the person exists who cares greatly about performance, and is still willing to consider bzip2. There isn't a point anywhere in the design space where bzip2 beats zstd. You can get smaller outputs from zstd in 1/20th the time for many common inputs, or you can spend the same amount of time and get a significantly smaller output, and zstd decompression is again 20-50x faster depending. So the speed of your bzip2 implementation hardly seems worth arguing over.
  - MBCook 1y ago
    Sure there is: someone provided you bzip2 files. Or required you give them files in that format. Then you don’t have a choice. And if you have to use it, 14% is a really nice speed up.
- vlovich123 1y ago
  > After the uutils debacle Which debacle?
  - _zagj 1y ago
    [flagged]
    - 0cf8612b2e1e 1y ago
      So what was I supposed to get from that 4chan wannabe site? That the project is not currently at fast as GNU? Where is the lying?
      - _zagj 1y ago
        [flagged]
dale_huevo 1y ago
A lot of this "rewrite X in Rust" stuff feels like burning your own house down so you can rebuild and paint it a different color. Counting CPU cycles as if it's an accomplishment seems irrelevant in a world where 50% of modern CPU resources are allocated toward UI eye candy.
- 0cf8612b2e1e 1y ago
  Every cycle saved is longer battery life. Someone paid the one time cost of porting it, and now we can enjoy better performance forever.
  - dale_huevo 1y ago
    They kicked off the article saying that no one uses bzip2 anymore. A million cycles saved for something no one uses (according to them) is still 0% battery life saved. If modern CPUs are so power efficient and have so many spare cycles to allocate to e.g. eye candy no one asked for, then no one is counting and the comparison is irrelevant.
    - jimktrains2 1y ago
      Isn't bzip used quite a bit, especially for tar files?
      - jeffbee 1y ago
        If so, only by misguided users. Why would anyone choose bz2 in 2025?
        0x457 1y ago
        To unpack an archive made from the time when bz2 was used?
        ben-schaaf 1y ago
        Of course no one uses systems, tools and files created before 2025!
firesteelrain 1y ago
Anyone know if this will by default resolve the 11 outstanding CVEs? Ironically there is one CVE reported in the bzip2 crate [1] https://app.opencve.io/cve/?product=bzip2&vendor=bzip2_project https://app.opencve.io/cve/?product=bzip2&vendor=bzip2_proje...
- Philpax 1y ago
  > The bzip2 crate before 0.4.4 They're releasing 0.6.0 today :>
- tialaramex 1y ago
  There's certainly a contrast between the "Oops a huge file causes a runtime failure" reported for that crate and a bunch of "Oops we have bounds misses" in C. I wonder how hard anybody worked on trying to exploit the bounds misses to get code execution. It may or may not be impossible to achieve that escalation.
- HackerThemAll 1y ago
  [flagged]
  - jorams 1y ago
    But it does apply to the bzip2 crate, which is the topic of discussion. Its new pure-rust implementation is libbz2-rs-sys, not bzip2-rs. The last sentence is irrelevant.
  - debugnik 1y ago
    This article is about the bzip2 crate, not the bzip2-rs crate, despite the repo for the former having the name of the latter.
    - conorjh 1y ago
      [dead]
rwaksmunski 1y ago
I use this crate to process 100s of TB of Common Crawl data, I appreciate the speedups.
- malux85 1y ago
  Yeah came here to say a 14% speed up in compression is pretty good!
  - koakuma-chan 1y ago
    It's blazingly fast
  - aidenn0 1y ago
    bzip2 (particularly parallel implementations thereof) are already relatively competitive for compression. The decompression time is where it lags behind because lz77 based algorithms can be incredibly fast at decompression.
- viraptor 1y ago
  What's the reason for using bz2 here? Wouldn't it be faster to do a one off conversion to zstd? It beats bzip2 in every metric at higher compression levels as far as I know.
  - declan_roberts 1y ago
    That assumes you're processing the data more than once.
    - deleted 1y ago
      [deleted]
  - rwaksmunski 1y ago
    Common Crawl delivers the data as bz2. Indeed I store intermediate data in zstd with ZFS.
- 1y ago
solarized 1y ago
Do they use any llm to transpile the C to Rust ?
- solarized 1y ago
  So, have the "toxic" people from StackExchange migrated here too? I just asked out of curiosity and got downvoted. :Sigh
- nightfly 1y ago
  Task that requires precision and potentially hard to audit? Exactly where I'd use an LLM /s
  - CGamesPlay 1y ago
    Without commenting on whether an LLM is the right approach, I don't think this task is particularly hard to audit. There is almost assuredly a huge test suite for bzip2 archives; fuzzing file formats is very easy; and you can restrict / audit the use of unsafe by the translator.
    - MBCook 1y ago
      You’re right, there is a large existing test suite. It’s mentioned in an article linked from this one. https://trifectatech.org/blog/translating-bzip2-with-c2rust/ https://trifectatech.org/blog/translating-bzip2-with-c2rust/ I suspect attempting to debug it would be a nightmare though. Given the LLM could hallucinate anything anywhere you’d likely waste a ton of time. I suspect it would be faster to just try and write a new implementation based on the spec and debug that against the test suite. You’d likely be closer. In fact, since they used c2rust, they had a perfectly working version from the start. From there they just had to clean up the Rust code and make sure it didn’t break anything. Clearly the best of the three options.
    - j16sdiz 1y ago
      > and you can restrict / audit the use of unsafe by the translator. No. You need to audit for correctness in additional to safety.
a-dub 1y ago
i'd be curious if they're using the same llvm codegen (with the same optimization) backend for the c and rust versions. if so, where the speedups are coming from? (ie, is it some kind of rust auto-simd thing, did they use the opportunity to hand optimize other parts or is it making use of newer optimized libraries, or... other)
- eru 1y ago
  Just speculating: Rust can hand over more hints to the code generator. Eg you don't have to worry about aliasing as much as with C pointers. See https://en.wikipedia.org/wiki/Aliasing_(computing)#Conflicts_with_optimization https://en.wikipedia.org/wiki/Aliasing_(computing)#Conflicts...
  - MBCook 1y ago
    This makes a lot of sense to me, though I don’t know the official answer so I’m just sort of guessing along too. Linked from the article is another on how they used c2rust to do the initial translation. https://trifectatech.org/blog/translating-bzip2-with-c2rust/ https://trifectatech.org/blog/translating-bzip2-with-c2rust/ For our purposes, it points out places where the code isn’t very optimal because the C code has no guarantees on the ranges of variables, etc. It also points out a lot of people just use ‘int’ even when the number will never be very big. But with the proper type the Rust compiler can decide to do something else if it will perform better. So I suspect your idea that it allows unlocking better optimizations though more knowledge is probably the right answer.
  - Too 1y ago
    Ergonomics of using the right data structures and algorithms can also play a big role. In C, everything beyond a basic array is too much hassle.
    - littlestymaar 1y ago
      Yeah, that was Brian Cantrill's realization when for the sake of learning he rewrote a part of dtrace in Rust and was shocked when he saw his naive reimplementation being significantly faster than his original code, and the answer boiled down to “I used a BTreeMap" in Rust because it's in std”.
xvilka 1y ago
I hope they or Prossimo will also look and reimplement in the similar fashion the core Internet protocols - BGP, OSPF and RIP, other routing implementations, DNS servers, and so on.
- dataking 1y ago
  https://www.memorysafety.org/initiative/ https://www.memorysafety.org/initiative/ this page mentions TLS and DNS which goes some way towards your suggestion.
  - throw10920 1y ago
    Is that domain actually about memory safety or about Rust?
- glowiefedposter 1y ago
  [dead]
- nickpsecurity 1y ago
  One guy did Ironsides DNS in SPARK Ada which has stronger proofs.
  - xvilka 1y ago
    Nothing against Ada, it's a good language. The only problem would be finding contributors in that case.
- everfrustrated 1y ago
  Check out https://nlnet.nl/project/current.html https://nlnet.nl/project/current.html https://www.sovereign.tech/programs/fund https://www.sovereign.tech/programs/fund There's been good support over the last couple of years to fund rewriting critical internet & OS tools into safer languages like Rust. Eg BGP in Rust https://www.nlnetlabs.nl/projects/routing/rotonda/ https://www.nlnetlabs.nl/projects/routing/rotonda/
broken_broken_ 1y ago
About not having perf on macOS: you can get quite far with dtrace for profiling. That’s what the original flame graph script in Perl mentions using and what the flame graph Rust reimplementation also uses. It does not have some metrics like cache misses or micro instructions retired but still it can be very useful.
zoobab 1y ago
Lbzip2 had much faster decompressing speed, using all available CPU cores. It's 2025, and most programs like Python are stuck at one CPU core.
- guappa 1y ago
  Thanks for showing us you have no understanding of python's situation.
  - pdimitar 1y ago
    Feel free to enlighten future readers?
tephra 1y ago
I like Rust and have an ambition to learn it as well (I've had a few false starts...). One of my issues that I have is that every (slight exaggeration) library that I seem to come across is still at version 0.x.y. Take this library as an example. 0.1.0 was released in 2014 and it still hasn't had a 1.0.0 release, is there an aversion to get to 1.0.0 in the rust community?
- liambigelow 1y ago
  https://0ver.org/#notable-zerover-projects https://0ver.org/#notable-zerover-projects
  - liambigelow 1y ago
    Serious answer: For some, they do change semi-often and don't feel compelled to declare stability. In other cases, it's a stable + widely used 0.x package, and bumping it to 1.0 usually implies _some_ kind of breaking change. (I don't know if that _should_ be the case, but I know that if I see a dependency has bumped from 0.x to 1.0 I'm going to be cautious and wait to update it until I have more time). In general: People usually aren't too concerned about it.
  - sramsay64 1y ago
    This list's Zig as an entry, despite the Zig project having very clear plans[0] for a 1.0 release. That's not 0ver, it's just the beta stage of semver. [0] https://github.com/ziglang/zig/milestone/2 https://github.com/ziglang/zig/milestone/2
- deleted 1y ago
  [deleted]
- scripturial 1y ago
  Yes, in rust, the package manager has built in rules about when to update a package. It won’t auto update a major version change because it implies a change that breaks something. As long as your package is safe to auto update you don’t want to change the major version number.
wiz21c 1y ago
FTA: > Why bother working on this algorithm from the 90s that sees very little use today? What's in use nowadays ? zstd ? ahh saw this: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/
Aissen 1y ago
Does anyone know if it supports parallel decompression, lbzip2-style? (or just iterators doing pre-scanning for the block magic that allow doing parallel decompression on top). Edit : it probably doesn't.
agumonkey 1y ago
rust aside, I really enjoy seeing all these different implementation benchmarks, very satisfying to read
nullbyte 1y ago
We should rewrite Rust in Javascript

336 points by Bogdanp 1y ago

How realistic is it for the Trifecta Tech implementation to start displacing the "official" implementation used by linux distros, which hasn't seen an upstream release since 2019? Fedora recently swapped the original Adler zlib implementation with zlib-ng, so that sort of thing isn't impossible. You just need to provide a C ABI compatible with the original one.

masfuerte 1y ago
They do provide a compatible C ABI. Someone "just" needs to do the work to make it happen.
rlpb 1y ago
> You just need to provide a C ABI compatible with the original one. How does this interact with dynamic linking? Doesn't the current Rust toolchain mandate static linking?
- sedatk 1y ago
  No. https://doc.rust-lang.org/reference/linkage.html#r-link.dylib https://doc.rust-lang.org/reference/linkage.html#r-link.dyli...
- arcticbull 1y ago
  Rust lets you generate dynamic C-linkage libraries. Use crate-type=["cdylib"]
- nicoburns 1y ago
  Dynamic linking works fine if you target the C ABI.
- timeon 1y ago
  You can use dynamic linking in Rust with C ABI. Which means going through `unsafe` keyword - also known as 'trust me bro'. Static linking directly to Rust source means it is checked by compiler so there is no need for unsafe.
- bluGill 1y ago
  Rust cannot dynamic link to rust. It can dynamic link to C and be dynamicly linked by C - if you combine the two you can cheat but it is still C that you are dealing with not rust even if rust is on both sides.
  - deleted 1y ago
    [deleted]
  - mjevans 1y ago
    It might help to think of it as two IPC 'servers' written in rust that happen to have the C ABI interfaces as their communication protocol.
  - filmor 1y ago
    Rust can absolutely link to Rust libraries dynamically. There is no stable ABI, so it has to be the same compiler version, but it will still be dynamically linked.
- conradev 1y ago
  Rust importing Rust must be statically linked, yes. You can statically link Rust into a dynamic library that other libraries link to, though!
- alxhill 1y ago
  The commenters below are confusing two things - Rust binaries can be dynamically linked, but because Rust doesn’t have a stable ABI you can’t do this across compiler versions the way you would with C. So in practice, everything is statically linked.
  - eru 1y ago
    Static linking also produces smaller binaries and lets you do link-time-optimisation.
    - emidln 1y ago
      Static linking doesn't produce smaller binaries. You are literally adding the symbols from a library into your executable rather than simply mentioning them and letting the dynamic linker figure out how to map those symbols at runtime. The sum size of a dynamic binary plus the dynamic libraries may be larger than one static linked binary, but whether that holds for more static binaries (2, 3, or 100s) depends on the surface area your application uses of those libraries. It's relatively common to see certain large libraries only dynamically linked, with the build going to great lengths to build certain libraries as shared objects with the executables linking them using a location-relative RPATH (using the $ORIGIN feature) to avoid the extra binary size bloat over large sets of binaries.
      - IshKebab 1y ago
        Static linking does produce smaller binaries when you bundle dependencies. You're conflating two things - static vs dynamic linking, and bundled vs shared dependencies. They are often conflated because you can't have shared dependencies with static linking, and bundling dynamically linked libraries is uncommon in FOSS Linux software. It's very common on Windows or with commercial software on Linux though.
        guappa 1y ago
        You know how the page cache works? Static linking makes it not work. So 3000 processes won't share the same pages for the libc but will have to load it 3000 times.
wmf 1y ago
Ubuntu is using Rust sudo so it's definitely possible.
- egorfine 1y ago
  It's not. At least not yet. It's planned for 25.10, but thankfully sudo will be packaged and available for a few versions after that as promised [1]. [1] https://discourse.ubuntu.com/t/adopting-sudo-rs-by-default-in-ubuntu-25-10/60583 https://discourse.ubuntu.com/t/adopting-sudo-rs-by-default-i...
tiffanyh 1y ago
I think that is the goal of uutils. https://uutils.github.io/ https://uutils.github.io/
- cocoa19 1y ago
  I hope some are improved too. The performance boost in tools like ripgrep and tokei is insane compared to the tools they replace (grep and cloc respectively).
  - egorfine 1y ago
    I absolutely hate it when people call their tools a "replacement" for something that is part of core standards, something, that did just fine for decades. ripgrep is an excellent tool. But it's not a grep replacement. And should not ever be.
    - mprovost 1y ago
      The GNU utils were a replacement for the BSD utils which were a replacement for the original AT&T utils. Every replacement added new functionality and improvements, and every time someone complained that they didn't stick closer to the thing they replaced. Looking specifically at grep, there used to be new versions like egrep and fgrep that added functionalities beyond standard grep's, but those were eventually pulled into "standard" grep (GNU or BSD). If we stuck with standards we'd all still be using the Bourne shell. The GNU utilities have been around long enough that they feel like the standard now, but I'm glad that we're coming into a new phase of innovation in command-line utilities. And this didn't start with Rust - the new generation of search utilities started with ack (Perl) and then ag (C).
      - egorfine 1y ago
        > which were a replacement for the You have a point here. I have to agree. "X but rewritten in Z" is a terrible marketing, though. Makes me instantly want to hate the tool and its authors. (Love rust. Hate the vibe).
deknos 1y ago
i wait until they come to the hard stuff like awk, sed and grep.
- GuB-42 1y ago
  ripgrep is one of the best grep replacement you can find, maybe even the best, and also one of the most famous Rust projects. I don't know of a sed equivalent, but I guess that would be easy to implement as Rust has good regex support (see ripgrep), and 90%+ of sed usage is search-and-replace. The other commands don't look hard to implement and because they are not used as much, optimizing these is less of a priority. I don't know about awk, it is a full programming language, but I guess it is far from an impossible task to implement. Now the real hard part is making a true, bug-for-bug compatible replacement of the GNU version of these tools, but while good to have, it is not strictly necessary. For example, Busybox is very popular, maybe even more so than GNU in terms of number of devices, and it has its own (likely simplified) version of grep, sed and awk.
  - scns 1y ago
    There is sd, not a drop in replacement though. https://github.com/chmln/sd https://github.com/chmln/sd
- egorfine 1y ago
  What would be the point?
kpcyrd 1y ago
I briefly looked a this and there's already cargo-c configuration, which is good, but it's currently namespaced differently, so it won't get automatically detected by C programs as `libbz2`: https://github.com/trifectatechfoundation/libbzip2-rs/blob/8214be22d32e91d31a122409eba5f3c67dc80659/libbz2-rs-sys-cdylib/Cargo.toml#L30-L39 https://github.com/trifectatechfoundation/libbzip2-rs/blob/8... I'm not familiar enough with the symbols of bzip2 to say anything about ABI compatibility. I have a toy project to explore things like that, but it's difficult to set aside the amount of time needed to maintain an implementation of the GNU operating system. I would welcome pull requests though: https://github.com/kpcyrd/platypos https://github.com/kpcyrd/platypos
Pesthuf 1y ago
If it hasn't seen an upstream release since 2019, doesn't that mean the implementation is just... finished? Maybe there's no more bugs to fix and features to add. And in that case, I don't see what's wrong with it.
- LinusU 1y ago
  Isn't 10-15% faster compression, and 5-10% faster decompression, a very nice "feature"? > [...] doesn't that mean the implementation is just... finished? I don't think that it _necessarily_ means that, e.g. all projects that haven't had a release since 2019 aren't finished? Probably most of them are simply abandoned? On the other hand, a finished implementation is certainly a _possible_ explanation for why there have been no releases. In this specific case, there are a handful of open bugs on their issue tracker. So that would indicate that the project isn't finished. ref: https://sourceware.org/bugzilla/buglist.cgi?product=bzip2 https://sourceware.org/bugzilla/buglist.cgi?product=bzip2
  - restalis 1y ago
    "Isn't 10-15% faster compression, and 5-10% faster decompression, a very nice «feature»?" Although much code can be optimized to get it to run 10-15% faster, if that comes at the expense of legibility then such "feature" get rejected nowadays. Translating an existing codebase into a language that makes things more difficult¹ and (because of that) has (and most likely will have) fewer engineers willing to working in it looks very much akin to applying legibility-affecting optimizations to me. ¹ ...and not only. I've already described in other comments some of the issues I see with it as a software development tool, here https://news.ycombinator.com/item?id=31565024 https://news.ycombinator.com/item?id=31565024 here https://news.ycombinator.com/item?id=33390634 https://news.ycombinator.com/item?id=33390634 and here https://news.ycombinator.com/item?id=42241516 https://news.ycombinator.com/item?id=42241516
    - LinusU 1y ago
      > Although much code can be optimized to get it to run 10-15% faster, if that comes at the expense of legibility then such "feature" get rejected nowadays. Makes sense, and I'd probably make the same call if I was a maintainer and someone submitted a patch which increased performance at the cost of maintainability... > Translating an existing codebase into a language that makes things more difficult¹ and (because of that) has (and most likely will have) fewer engineers willing to working in it looks very much akin to applying legibility-affecting optimizations to me. Here I have to personally disagree. I think that Rust is easier than both C and C++. Especially when coming into an already existing project. The chance of me contributing to a Rust project is higher than to a C project, because I feel more comfortable in knowing that my code is "correct". It's also easier to match the "style" since most Rust projects follow the same structures and patterns, whereas in C it can vary wildly. E.g. I contributed my first feature to Valkey (Redis fork, C codebase) recently, and figuring out how the tests worked took me quite some time. In fact, in the end I couldn't figure out how to run a specific test so I just ran all tests and grepped the output for my test name. But the tests take tens of minutes to run so this was sub-optimal. On the other hand, 99% of all Rust projects use `cargo test`, and to run a single test I can just click the play button in my editor (Zed) that shows up next to the test (or `cargo test "testname"`). (with this said, I think that Valkey is a really well structured code base! Great work on their part) Anyhow, this is just to illustrate my experience. I'm sure that for someone more used to C or C++ they would be more productive in that. And I could go on for ages on all the features that make me miss Rust every day at work when I have to work in other languages, especially algebraic data types!

_zagj 1y ago

[flagged]

jeffbee 1y ago
You should of course verify these results in your scenario. However, I somewhat doubt that the person exists who cares greatly about performance, and is still willing to consider bzip2. There isn't a point anywhere in the design space where bzip2 beats zstd. You can get smaller outputs from zstd in 1/20th the time for many common inputs, or you can spend the same amount of time and get a significantly smaller output, and zstd decompression is again 20-50x faster depending. So the speed of your bzip2 implementation hardly seems worth arguing over.
- MBCook 1y ago
  Sure there is: someone provided you bzip2 files. Or required you give them files in that format. Then you don’t have a choice. And if you have to use it, 14% is a really nice speed up.
vlovich123 1y ago
> After the uutils debacle Which debacle?
- _zagj 1y ago
  [flagged]
  - 0cf8612b2e1e 1y ago
    So what was I supposed to get from that 4chan wannabe site? That the project is not currently at fast as GNU? Where is the lying?
    - _zagj 1y ago
      [flagged]

dale_huevo 1y ago

A lot of this "rewrite X in Rust" stuff feels like burning your own house down so you can rebuild and paint it a different color. Counting CPU cycles as if it's an accomplishment seems irrelevant in a world where 50% of modern CPU resources are allocated toward UI eye candy.

0cf8612b2e1e 1y ago
Every cycle saved is longer battery life. Someone paid the one time cost of porting it, and now we can enjoy better performance forever.
- dale_huevo 1y ago
  They kicked off the article saying that no one uses bzip2 anymore. A million cycles saved for something no one uses (according to them) is still 0% battery life saved. If modern CPUs are so power efficient and have so many spare cycles to allocate to e.g. eye candy no one asked for, then no one is counting and the comparison is irrelevant.
  - jimktrains2 1y ago
    Isn't bzip used quite a bit, especially for tar files?
    - jeffbee 1y ago
      If so, only by misguided users. Why would anyone choose bz2 in 2025?
      - 0x457 1y ago
        To unpack an archive made from the time when bz2 was used?
      - ben-schaaf 1y ago
        Of course no one uses systems, tools and files created before 2025!

firesteelrain 1y ago

Anyone know if this will by default resolve the 11 outstanding CVEs? Ironically there is one CVE reported in the bzip2 crate [1] https://app.opencve.io/cve/?product=bzip2&vendor=bzip2_project https://app.opencve.io/cve/?product=bzip2&vendor=bzip2_proje...

Philpax 1y ago
> The bzip2 crate before 0.4.4 They're releasing 0.6.0 today :>
tialaramex 1y ago
There's certainly a contrast between the "Oops a huge file causes a runtime failure" reported for that crate and a bunch of "Oops we have bounds misses" in C. I wonder how hard anybody worked on trying to exploit the bounds misses to get code execution. It may or may not be impossible to achieve that escalation.
HackerThemAll 1y ago
[flagged]
- jorams 1y ago
  But it does apply to the bzip2 crate, which is the topic of discussion. Its new pure-rust implementation is libbz2-rs-sys, not bzip2-rs. The last sentence is irrelevant.
- debugnik 1y ago
  This article is about the bzip2 crate, not the bzip2-rs crate, despite the repo for the former having the name of the latter.
  - conorjh 1y ago
    [dead]

rwaksmunski 1y ago

I use this crate to process 100s of TB of Common Crawl data, I appreciate the speedups.

malux85 1y ago
Yeah came here to say a 14% speed up in compression is pretty good!
- koakuma-chan 1y ago
  It's blazingly fast
- aidenn0 1y ago
  bzip2 (particularly parallel implementations thereof) are already relatively competitive for compression. The decompression time is where it lags behind because lz77 based algorithms can be incredibly fast at decompression.
viraptor 1y ago
What's the reason for using bz2 here? Wouldn't it be faster to do a one off conversion to zstd? It beats bzip2 in every metric at higher compression levels as far as I know.
- declan_roberts 1y ago
  That assumes you're processing the data more than once.
  - deleted 1y ago
    [deleted]
- rwaksmunski 1y ago
  Common Crawl delivers the data as bz2. Indeed I store intermediate data in zstd with ZFS.
1y ago

solarized 1y ago

Do they use any llm to transpile the C to Rust ?

solarized 1y ago
So, have the "toxic" people from StackExchange migrated here too? I just asked out of curiosity and got downvoted. :Sigh
nightfly 1y ago
Task that requires precision and potentially hard to audit? Exactly where I'd use an LLM /s
- CGamesPlay 1y ago
  Without commenting on whether an LLM is the right approach, I don't think this task is particularly hard to audit. There is almost assuredly a huge test suite for bzip2 archives; fuzzing file formats is very easy; and you can restrict / audit the use of unsafe by the translator.
  - MBCook 1y ago
    You’re right, there is a large existing test suite. It’s mentioned in an article linked from this one. https://trifectatech.org/blog/translating-bzip2-with-c2rust/ https://trifectatech.org/blog/translating-bzip2-with-c2rust/ I suspect attempting to debug it would be a nightmare though. Given the LLM could hallucinate anything anywhere you’d likely waste a ton of time. I suspect it would be faster to just try and write a new implementation based on the spec and debug that against the test suite. You’d likely be closer. In fact, since they used c2rust, they had a perfectly working version from the start. From there they just had to clean up the Rust code and make sure it didn’t break anything. Clearly the best of the three options.
  - j16sdiz 1y ago
    > and you can restrict / audit the use of unsafe by the translator. No. You need to audit for correctness in additional to safety.

a-dub 1y ago

i'd be curious if they're using the same llvm codegen (with the same optimization) backend for the c and rust versions. if so, where the speedups are coming from? (ie, is it some kind of rust auto-simd thing, did they use the opportunity to hand optimize other parts or is it making use of newer optimized libraries, or... other)

eru 1y ago
Just speculating: Rust can hand over more hints to the code generator. Eg you don't have to worry about aliasing as much as with C pointers. See https://en.wikipedia.org/wiki/Aliasing_(computing)#Conflicts_with_optimization https://en.wikipedia.org/wiki/Aliasing_(computing)#Conflicts...
- MBCook 1y ago
  This makes a lot of sense to me, though I don’t know the official answer so I’m just sort of guessing along too. Linked from the article is another on how they used c2rust to do the initial translation. https://trifectatech.org/blog/translating-bzip2-with-c2rust/ https://trifectatech.org/blog/translating-bzip2-with-c2rust/ For our purposes, it points out places where the code isn’t very optimal because the C code has no guarantees on the ranges of variables, etc. It also points out a lot of people just use ‘int’ even when the number will never be very big. But with the proper type the Rust compiler can decide to do something else if it will perform better. So I suspect your idea that it allows unlocking better optimizations though more knowledge is probably the right answer.
- Too 1y ago
  Ergonomics of using the right data structures and algorithms can also play a big role. In C, everything beyond a basic array is too much hassle.
  - littlestymaar 1y ago
    Yeah, that was Brian Cantrill's realization when for the sake of learning he rewrote a part of dtrace in Rust and was shocked when he saw his naive reimplementation being significantly faster than his original code, and the answer boiled down to “I used a BTreeMap" in Rust because it's in std”.

xvilka 1y ago

I hope they or Prossimo will also look and reimplement in the similar fashion the core Internet protocols - BGP, OSPF and RIP, other routing implementations, DNS servers, and so on.

dataking 1y ago
https://www.memorysafety.org/initiative/ https://www.memorysafety.org/initiative/ this page mentions TLS and DNS which goes some way towards your suggestion.
- throw10920 1y ago
  Is that domain actually about memory safety or about Rust?
glowiefedposter 1y ago
[dead]
nickpsecurity 1y ago
One guy did Ironsides DNS in SPARK Ada which has stronger proofs.
- xvilka 1y ago
  Nothing against Ada, it's a good language. The only problem would be finding contributors in that case.
everfrustrated 1y ago
Check out https://nlnet.nl/project/current.html https://nlnet.nl/project/current.html https://www.sovereign.tech/programs/fund https://www.sovereign.tech/programs/fund There's been good support over the last couple of years to fund rewriting critical internet & OS tools into safer languages like Rust. Eg BGP in Rust https://www.nlnetlabs.nl/projects/routing/rotonda/ https://www.nlnetlabs.nl/projects/routing/rotonda/

broken_broken_ 1y ago

About not having perf on macOS: you can get quite far with dtrace for profiling. That’s what the original flame graph script in Perl mentions using and what the flame graph Rust reimplementation also uses. It does not have some metrics like cache misses or micro instructions retired but still it can be very useful.

zoobab 1y ago

Lbzip2 had much faster decompressing speed, using all available CPU cores. It's 2025, and most programs like Python are stuck at one CPU core.

guappa 1y ago
Thanks for showing us you have no understanding of python's situation.
- pdimitar 1y ago
  Feel free to enlighten future readers?

tephra 1y ago

I like Rust and have an ambition to learn it as well (I've had a few false starts...). One of my issues that I have is that every (slight exaggeration) library that I seem to come across is still at version 0.x.y. Take this library as an example. 0.1.0 was released in 2014 and it still hasn't had a 1.0.0 release, is there an aversion to get to 1.0.0 in the rust community?

liambigelow 1y ago
https://0ver.org/#notable-zerover-projects https://0ver.org/#notable-zerover-projects
- liambigelow 1y ago
  Serious answer: For some, they do change semi-often and don't feel compelled to declare stability. In other cases, it's a stable + widely used 0.x package, and bumping it to 1.0 usually implies _some_ kind of breaking change. (I don't know if that _should_ be the case, but I know that if I see a dependency has bumped from 0.x to 1.0 I'm going to be cautious and wait to update it until I have more time). In general: People usually aren't too concerned about it.
- sramsay64 1y ago
  This list's Zig as an entry, despite the Zig project having very clear plans[0] for a 1.0 release. That's not 0ver, it's just the beta stage of semver. [0] https://github.com/ziglang/zig/milestone/2 https://github.com/ziglang/zig/milestone/2
deleted 1y ago
[deleted]
scripturial 1y ago
Yes, in rust, the package manager has built in rules about when to update a package. It won’t auto update a major version change because it implies a change that breaks something. As long as your package is safe to auto update you don’t want to change the major version number.

wiz21c 1y ago

FTA: > Why bother working on this algorithm from the 90s that sees very little use today? What's in use nowadays ? zstd ? ahh saw this: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/

Aissen 1y ago

Does anyone know if it supports parallel decompression, lbzip2-style? (or just iterators doing pre-scanning for the block magic that allow doing parallel decompression on top). Edit : it probably doesn't.

agumonkey 1y ago

rust aside, I really enjoy seeing all these different implementation benchmarks, very satisfying to read

nullbyte 1y ago

We should rewrite Rust in Javascript