3 ms·

The port was line for line and full of unsafe code. It didn't prevent any bugs. They're not using Rust for its strengths.

skeledrew 2mo ago
You should give the article[0] a read. "A large percentage of bugs from that list are use-after-free, double-free, and "forgot to free" in an error path. In safe Rust, these are compiler errors and RAII-like automatic cleanup with Drop. Compiler errors are a better feedback loop than a style guide." "At the time of writing, about 4% of Bun's Rust code sits inside an unsafe block (~13,000 unsafe keywords across ~27,000 lines / ~780,000 lines), and 78% of those blocks are a single line — a pointer that came from C++, or one call into a C library. I expect this number to go down over time as we refactor from a faithful Zig port (which had no greppable unsafe keyword) to idiomatic Rust, but we are going to continue using C & C++ libraries like JavaScriptCore so it will always have more unsafe than pure Rust projects." [0] https://bun.com/blog/bun-in-rust https://bun.com/blog/bun-in-rust
- bunderbunder 2mo ago
  That one’s interesting to me because it speaks to both sides of the debate. Yes, safe Rust is good for catching those kinds of errors at compile time. But also, it should theoretically be very easy for a compiler to avoid those problems in Zig, too, if you are using the language the way it wants to be used. Which, from what I’ve experienced so far, does seem to take a whole lot more effort if you’re using a coding agent. I spend an incredible amount of time making sure mine doesn’t bloat our codebase with Java-flavored Python, and all the noise and defects and performance problems that it brings.
  - skeledrew 2mo ago
    > using the language the way it wants to be used. This sounds like just the coding conventions dependency they're trying to avoid.
    - bunderbunder 2mo ago
      Yup. But that also gets to the “they’re both right” angle. Zig’s convention may not be how the Bun developers wanted to work, but it’s a good choice for the kinds of projects Zig is designed for. Which overlaps a bit with, but isn't the same as, Rust’s intended use cases. For example, I might rather go with the Zig philosophy if I’ve got hard memory constraints and want to avoid dynamic memory allocation. Which, ironically, is something I’d love for my browser’s JavaScript runtime to do. But I can also respect others not wanting that.
    - TheGoddessInari 2mo ago
      IMO it's the same problem with Rust. LLMs are trained on vast quantities of code that violate Rust soundness in safe code. LLMs are prediction engines: you can move the needle (heavily) on the predictions from lazy to correct by construction, but the defaults even on high end models like Opus are always riddled with shortcuts. LLMs can produce coherent & safe Rust 1.92, LLMs can produce coherent Zig 0.16, but that's dependent on in context learning & instruction following.
      - maleldil 2mo ago
        > LLMs are trained on vast quantities of code that violate Rust soundness in safe code. Why is your reasoning behind this? Most Rust code in the wild does not use unsafe. Most projects don't have a single line of unsafe.
      - skeledrew 2mo ago
        > that's dependent on in context learning & instruction following Not just that. For Rust in particular, since unsafe code is uncompilable code (unless explicitly marked "unsafe"), the LLM is essentially forced to continue iterating even if the code already works but is unsafe, until it finds an implementation that's both correct (per the spec) and safe (per the compiler). That's the difference from Zig, where "make the code safe" is essentially part of the spec, and so may be missed by the LLM and not be discovered by a human until there's an observable runtime issue, or CVE.
  - dnautics 2mo ago
    > But also, it should theoretically be very easy for a compiler to avoid those problems in Zig, too, if you are using the language the way it wants to be used. agreed. in my side project im tinkering with a memory safety checker in zig that intercepts a compiler artifact, and it works better with idiomatic zig, problematic code is when you try to write c-isms in zig -- so i basicallt tell the checker to reject c-isms that create ambiguity for safety checking.
- sfink 2mo ago
  The way I'm thinking of it is that they ported Bun from Zig to Unsafe Rust, and are still working on now porting it from Unsafe Rust to Rust (not 100% safe, but a reasonable amount).
- Dylan16807 2mo ago
  I'm optimistic for that refactoring. But if one in 30 lines sprinkled all over everywhere is unsafe then you haven't actually done much to improve safety. When you have a large percent of fully safe functions and modules, that's when you're making a difference. Counting the lines is not a very helpful metric.
  - deleted 2mo ago
    [deleted]
  - skeledrew 2mo ago
    > one in 30 lines > Counting the lines is not a very helpful metric. Seems self-contradictory. And they did say they're working to reduce the unsafe code over time. That huge percentage of memory bugs that they were able to better triage and fix due to the port was also them improving safety.
    - Dylan16807 2mo ago
      The other part of that line is important. Measuring lines is better than nothing but the distribution is much more important. If they're scattered all over that's a bigger downside than the line reduction is an upside. If you want good metrics, measure functions and data structures without unsafe. Not lines.
      - skeledrew 2mo ago
        > If you want good metrics, measure functions and data structures without unsafe. I doubt it, as it's possible a substantial amount of those unsafe areas can't even be removed, given the project's dependent on C/C++ libraries. Without proper calibration those metrics won't tell the full story, and with calibration the incentive for the story will decrease over time with the unsafe refactoring. I doubt the distribution of the unsafe code really matters that much; what matters is proper handling, wherever it is, so stability and security isn't compromised.
samwillis 2mo ago
They have made it very clear that the mechanical translation is the starting point, and are refactoring to remove as many unsafe declarations as possible.
neuronexmachina 2mo ago
That may have been the case for the initial iteration a few months ago, but is it still true?
insanitybit 2mo ago
They turned every hidden memory safety bug into a grep'able memory safety bug. That's step 1 and a massive win.
atombender 2mo ago
The original Zig code is metaphorically one big unsafe block. Even if the Rust port is made up of 3% unsafe blocks, that still means 97% of the original Zig code has been made safe (in the Rust sense).
- audunw 2mo ago
  Since they did a one-to-one translation to start with, hypothetically, if these were the true numbers, you could probably get a static analysis tool to guarantee you that 97% of the zig code was safe and tell you which of the 3% were unsafe. Zig has a lot more syntax and conventions to encourage safe code compare to C after all. It just doesn’t have way to mark which parts are safe or unsafe. An LLM could probably also trivially give you accurate reviews saying which parts were unsafe and in need of tests or reviews. I mean, considering their LLM budgets they could probably have had nightly reviews running every night for years before spending more than their Rust rewrite. Not that I think Rust rewrite was a bad idea. Rust is a good fit for this kind of project. I say that as a Zig enthusiast. It’s just that their stated motivation and reported results are kinda BS. If they just wrote “we just like Rust and thought it’d be cool to see of LLM could do the whole rewrite”, and left it at that, I think it’d be a more honest description of the motivation. The rest is just rationalisation.
  - Zakis1 2mo ago
    No you cannot get a static analysis tool to guarantee 97% of zig code was safe. Well, you can - the tool is called the borrow checker. Do you know what's infinitely cheaper and faster than an LLM running nightly reviews of all of the code and infact mathematically provable that the code (and any new code) is memory safe? It's called the borrow checker.
    - dnautics 2mo ago
      it seems like a reasonable enterprise to intercept the data dependency graph from the zig compiler and subject it to analysis to prove memory safety parameters: (my wip experiment: ) https://github.com/ityonemo/clr https://github.com/ityonemo/clr

skeledrew 2mo ago

You should give the article[0] a read. "A large percentage of bugs from that list are use-after-free, double-free, and "forgot to free" in an error path. In safe Rust, these are compiler errors and RAII-like automatic cleanup with Drop. Compiler errors are a better feedback loop than a style guide." "At the time of writing, about 4% of Bun's Rust code sits inside an unsafe block (~13,000 unsafe keywords across ~27,000 lines / ~780,000 lines), and 78% of those blocks are a single line — a pointer that came from C++, or one call into a C library. I expect this number to go down over time as we refactor from a faithful Zig port (which had no greppable unsafe keyword) to idiomatic Rust, but we are going to continue using C & C++ libraries like JavaScriptCore so it will always have more unsafe than pure Rust projects." [0] https://bun.com/blog/bun-in-rust https://bun.com/blog/bun-in-rust

bunderbunder 2mo ago
That one’s interesting to me because it speaks to both sides of the debate. Yes, safe Rust is good for catching those kinds of errors at compile time. But also, it should theoretically be very easy for a compiler to avoid those problems in Zig, too, if you are using the language the way it wants to be used. Which, from what I’ve experienced so far, does seem to take a whole lot more effort if you’re using a coding agent. I spend an incredible amount of time making sure mine doesn’t bloat our codebase with Java-flavored Python, and all the noise and defects and performance problems that it brings.
- skeledrew 2mo ago
  > using the language the way it wants to be used. This sounds like just the coding conventions dependency they're trying to avoid.
  - bunderbunder 2mo ago
    Yup. But that also gets to the “they’re both right” angle. Zig’s convention may not be how the Bun developers wanted to work, but it’s a good choice for the kinds of projects Zig is designed for. Which overlaps a bit with, but isn't the same as, Rust’s intended use cases. For example, I might rather go with the Zig philosophy if I’ve got hard memory constraints and want to avoid dynamic memory allocation. Which, ironically, is something I’d love for my browser’s JavaScript runtime to do. But I can also respect others not wanting that.
  - TheGoddessInari 2mo ago
    IMO it's the same problem with Rust. LLMs are trained on vast quantities of code that violate Rust soundness in safe code. LLMs are prediction engines: you can move the needle (heavily) on the predictions from lazy to correct by construction, but the defaults even on high end models like Opus are always riddled with shortcuts. LLMs can produce coherent & safe Rust 1.92, LLMs can produce coherent Zig 0.16, but that's dependent on in context learning & instruction following.
    - maleldil 2mo ago
      > LLMs are trained on vast quantities of code that violate Rust soundness in safe code. Why is your reasoning behind this? Most Rust code in the wild does not use unsafe. Most projects don't have a single line of unsafe.
    - skeledrew 2mo ago
      > that's dependent on in context learning & instruction following Not just that. For Rust in particular, since unsafe code is uncompilable code (unless explicitly marked "unsafe"), the LLM is essentially forced to continue iterating even if the code already works but is unsafe, until it finds an implementation that's both correct (per the spec) and safe (per the compiler). That's the difference from Zig, where "make the code safe" is essentially part of the spec, and so may be missed by the LLM and not be discovered by a human until there's an observable runtime issue, or CVE.
- dnautics 2mo ago
  > But also, it should theoretically be very easy for a compiler to avoid those problems in Zig, too, if you are using the language the way it wants to be used. agreed. in my side project im tinkering with a memory safety checker in zig that intercepts a compiler artifact, and it works better with idiomatic zig, problematic code is when you try to write c-isms in zig -- so i basicallt tell the checker to reject c-isms that create ambiguity for safety checking.
sfink 2mo ago
The way I'm thinking of it is that they ported Bun from Zig to Unsafe Rust, and are still working on now porting it from Unsafe Rust to Rust (not 100% safe, but a reasonable amount).
Dylan16807 2mo ago
I'm optimistic for that refactoring. But if one in 30 lines sprinkled all over everywhere is unsafe then you haven't actually done much to improve safety. When you have a large percent of fully safe functions and modules, that's when you're making a difference. Counting the lines is not a very helpful metric.
- deleted 2mo ago
  [deleted]
- skeledrew 2mo ago
  > one in 30 lines > Counting the lines is not a very helpful metric. Seems self-contradictory. And they did say they're working to reduce the unsafe code over time. That huge percentage of memory bugs that they were able to better triage and fix due to the port was also them improving safety.
  - Dylan16807 2mo ago
    The other part of that line is important. Measuring lines is better than nothing but the distribution is much more important. If they're scattered all over that's a bigger downside than the line reduction is an upside. If you want good metrics, measure functions and data structures without unsafe. Not lines.
    - skeledrew 2mo ago
      > If you want good metrics, measure functions and data structures without unsafe. I doubt it, as it's possible a substantial amount of those unsafe areas can't even be removed, given the project's dependent on C/C++ libraries. Without proper calibration those metrics won't tell the full story, and with calibration the incentive for the story will decrease over time with the unsafe refactoring. I doubt the distribution of the unsafe code really matters that much; what matters is proper handling, wherever it is, so stability and security isn't compromised.

samwillis 2mo ago

They have made it very clear that the mechanical translation is the starting point, and are refactoring to remove as many unsafe declarations as possible.

neuronexmachina 2mo ago

That may have been the case for the initial iteration a few months ago, but is it still true?

insanitybit 2mo ago

They turned every hidden memory safety bug into a grep'able memory safety bug. That's step 1 and a massive win.

atombender 2mo ago

The original Zig code is metaphorically one big unsafe block. Even if the Rust port is made up of 3% unsafe blocks, that still means 97% of the original Zig code has been made safe (in the Rust sense).

audunw 2mo ago
Since they did a one-to-one translation to start with, hypothetically, if these were the true numbers, you could probably get a static analysis tool to guarantee you that 97% of the zig code was safe and tell you which of the 3% were unsafe. Zig has a lot more syntax and conventions to encourage safe code compare to C after all. It just doesn’t have way to mark which parts are safe or unsafe. An LLM could probably also trivially give you accurate reviews saying which parts were unsafe and in need of tests or reviews. I mean, considering their LLM budgets they could probably have had nightly reviews running every night for years before spending more than their Rust rewrite. Not that I think Rust rewrite was a bad idea. Rust is a good fit for this kind of project. I say that as a Zig enthusiast. It’s just that their stated motivation and reported results are kinda BS. If they just wrote “we just like Rust and thought it’d be cool to see of LLM could do the whole rewrite”, and left it at that, I think it’d be a more honest description of the motivation. The rest is just rationalisation.
- Zakis1 2mo ago
  No you cannot get a static analysis tool to guarantee 97% of zig code was safe. Well, you can - the tool is called the borrow checker. Do you know what's infinitely cheaper and faster than an LLM running nightly reviews of all of the code and infact mathematically provable that the code (and any new code) is memory safe? It's called the borrow checker.
  - dnautics 2mo ago
    it seems like a reasonable enterprise to intercept the data dependency graph from the zig compiler and subject it to analysis to prove memory safety parameters: (my wip experiment: ) https://github.com/ityonemo/clr https://github.com/ityonemo/clr