8 ms·
A year of Rust in ClickHouse
- mplanchard 2y agoI love this. We are a rust shop and we use clickhouse a fair bit. We’ve been quite impressed with its speed and flexibility. I’m glad to see this kind of direct, real-world feedback around both the benefits and difficulties of mixing rust and C++, which we have also had to do a bit of (albeit in the opposite direction: a smattering of C++ in a sea of rust). I’m not sure if the poster here is the post author, but it would be great if the author would consider filling out this survey that was recently released asking for feedback on the future of rust’s vision: https://blog.rust-lang.org/2025/04/04/vision-doc-survey.html https://blog.rust-lang.org/2025/04/04/vision-doc-survey.html I’d love to see rust become the de facto standard for cross-language portable code by virtue of its ease of use, but as this and our experience highlights, there’s some way to go yet!
- mplanchard 2y agoOh one more note, regarding hermetic builds: I have tried to package clickhouse in nix for our dev environment and CI, but its build is pretty complicated, so I resorted to just pulling the precompiled binaries. Nix, via the standard rust integration or via something like crane, is actually quite nice for building rust/C++ combo projects, so it’d be awesome if the team might consider this as a means of achieving reproducibility. I’d imagine they’d have an easier time of it than I did, given they are more familiar with their own build process.
- IshKebab 2y agoThis guy seems to be both very positive about Rust and unfairly cynical about it at the same time... Rust is a really fantastic language but having worked on a mixed C++/Rust codebase I can see why they had so many issues. Rust just wasn't really designed with C++ interop in mind so it's kind of painful to use them together. Impressive that they made it work.
- uecker 2y ago"unfairly"? A lot of issues are obvious deficiencies of Rust, including immaturity of the ecosystem, integration issues, complexity, monomorphization bloat, supply chain issues. Now, all languages have issues, and Rust is certainly a nice language overall. The main issue with Rust is that is has been oversold as a panacea for safety using exaggerated arguments. So a bit of cynicism seems entirely fair.
- IshKebab 2y agoYes. I was thinking of this: > If you do an experiment and say "C++" anywhere on the Internet, in a minute someone will chime in and educate you about the existence of Rust. > I know examples when engineers rewrite code from Rust in Rust if they like to rewrite everything in Rust. > our engineers become too nauseous from Rust poisoning > So now they [Rust devs] can write something other than new versions of old terminal applications. > someone shows PRQL, everyone else thinks "What a wonderful idea, and, also, Rust" and gives this project a star on GitHub. This is, by the way, how most of Rust projects get their stars on GitHub. It doesn't look like someone wants to use this language, but what we want is to ride the hype. > we started to understand that it would be hard to get rid of Rust, and we could tolerate it. It's a very shitty attitude and not even accurate. You see this attitude from old C/C++ devs quite a lot, it's just very weird that he has that attitude and then also seems to be simultaneously quite keen to use Rust. Very weird! Anyway those are just the non-technical things. On the technical side: > Fully offline builds They solved it by vendoring but this is the obvious solution and also applies to C++. > Segfault in Rust They tried to do a null-terminated read of a string that wasn't null-terminated. Nothing to do with Rust. That would be an error in C++ too. In fact this is a strong argument for Rust. > Panic C/C++ code aborts. Or more commonly it crashes in a very difficult to debug way. I'll take panics any day. > Sanitizers require nightly Ok fair enough but this seems relatively minor. > Rust's OpenSSL links with the system library by default and you have to set an environment variable to statically link it. They set the environment variable. Frankly this is a million times easier than doing the same thing in C++. I'll stop there, but overall it seems like a lot of "this is a problem we had with Rust" where it should really be more like "this is something we had to do when using C++ with Rust". Weird vibe anyway.
- ChocolateGod 2y ago> If you do an experiment and say "C++" anywhere on the Internet, in a minute someone will chime in and educate you about the existence of Rust. Many people see this as a problem. The response to TypeScript choosing Go over Rust was pretty gross imho, no one should be abused for choosing a language.
- pjmlp 2y agoThe actual issue wasn't Go over Rust, rather having key people responsible for C# design, on a Microsoft project, going for a Google language. While at the same time, the .NET team routinely talks about .NET image problem outside traditional Microsoft shops, which naturally decisions like this aren't helping a tiny bit.
- ChocolateGod 2y ago.NET isn't famous for producing portable binaries and I don't think it's being any better now other than some experimental modes that break most code.
- pjmlp 2y agoDo you want more portable than bytecode with a dynamic compiler? Apparenty the greatest thing on Earth as per WebAssembly folks. As for AOT compilation, there have been multiple approaches since the early days, and the latest, Native AOT is good enough for everything required to write a TypeScript compiler, including better WebAssembly support than the Go compiler, thanks to Blazor infrastructure.
- dep_b 2y agoYeah loved the language and the IDE, but only boring Enterprise stuff built with it. Also C# got so complex, since it had to absorb every idea from F# rather than making F# a viable programming language on itself and improving interop... At a given point after being a C# programmer for years I still encountered patterns that were completely unreadable to me.
- pjmlp 2y agoI love that it goes both ways, about plus and minus of both languages, including rewriting back into C++ when it made sense, and the side joke about rewriting existing CLI tools in Rust. While C++ isn't perfect, has the warts of a 50 year's old language, and probably will never match Rust's safety, we would already be in a much better place if at least everyone used the tools at their disposal from the last 30 years. While I would advise to use Rust for some security critical scenarios, there are many others where it is still getting there, and there are other requirements to take into account other than affine types.
- simonask 2y agoComing from C++, my favorite take on Rust is that it is fundamentally about productivity. Avoiding UB is a serious drain on productivity in C++, and every new language or library feature comes with additional pitfalls, increasing the mental load. This is to say: The benefit of Rust is not actually about "security critical scenarios", but much more generally about delivering the same quality of code in a fraction of the time.
- pjmlp 2y agoWhile I agree with the general sentiment, while it doesn't sort out all safety issues, actually programming in C++ instead of C with C++ compiler, and making use of the pleothora of analysers would already prevent many issues. Better be 80% safer than none at all.
- selfmodruntime 2y agoIt's not only avoid UB in one area, it's about hidden footguns which you may not know about. I have been using C++ for a decade now and there are still UB causes I learn about every week or so.
- DeathArrow 2y agoC++ can be safe enough if you proceed with care. What I dislike of C++ is that it grew to become a monster of a language, containing all programming paradigms and ideas, good or bad, known to mankind. It's so monstrously huge no human can hold its entire complexity in his head. C++ allows you to do things in 10000 different ways and developer would do just that. Often in the same code base. That being said, I would use a sane subset of C++ every day over Rust. It's not that I hate Rust or that I don't think is good, technically sound and capable. It just doesn't fit the way I think and I like to work. I like to keep a simple model in mind. For me, the memory is just a huge array from which we copy data to CPU cache, move some to CPU registers, execute instructions and fetch data from the registers and put it again in some part of that huge array, to be used later. Rust adds a lot of complexity over this simple mental model of mine.
- chenhoey1211 2y agoLove this approach — Rust in the right places. I’ve been wondering if using Wasm modules (e.g. from MoonBit) for isolated components might offer a similar balance: memory safety without full rewrite.
- j16sdiz 2y agoThose github pr linked from the blog don't give me much confident: Links to "Better C++" is a PR for removing c++ template for build time. Unwinding stack in a "funny" way. PR comment saying something shouldn't be public and go on merging anyway.
- deleted 2y ago[deleted]
- dathinab 2y agoOne thing I often see pop up in larger projects, which in the article is likely the fault of way to large symbols, is overuse of generics/type state/etc. Or you could formulate this as needless obsession with not using `dyn`. And sure generics are more powerful, dyn has limitations, etc. etc. It's one of this "Misconceptions Programmers believe about Monomorphisation vs. Virtual Calls" things as in: TL;DR: dyn isn't as bad as some people make it out to be; Weather perf. or convenience it can be the better choice. Any absolute recommendation of always use this or that is wrong. - wrong: monomorphisation is always faster; reason: monomorphisation pollutes the instruction cache way worse, as such in some situations switching some parts (not all parts) to virtual calls and similar approaches can lead to major performance improvements. Good example here are various experiments about how to implement something like serde but faster and with less binary size. - wrong: monomorphisation was picked in rust because it's better for rust; right: it was picked because it is reasonable good and was viable to implement with available resources. (for low level languages it's still better then only using vtables, but technically transparent hybrid solutions are even more desirable) - wrong: virtual calls are always slow in microbenchmarks; right: while they are more work to do modern cpus have gotten very very good at optimizing them, under the right conditions the might be literally as fast as normal function calls (but most times they are slightly slower until mono. trashes icache too much) - wrong: monomorphisation is always better for the optimizer; right: monomorphisation gives the optimizer more choices, but always relevant or useful choices but they always add more work it has to do, so slower compiler times and if you are unlucky it will miss more useful optimizations due to noise - wrong: in rust generics are always more convenient to use; right: Adding a generic (e.g. to accomodate a return position impl trait) in the wrong place can lead you to having to write generic parameters all through the code base. But `dyn` has much more limitations/constraints, so for both convenience and performance it's a trade of which more often favors monomorphisation, but not as much as many seem to believe. - wrong: always using dyn works; right: dyn doesn't work for all code and even if it would using it everywhere can put too much burden on the branch predictor and co. making vcalls potentially as slow as some people thing they are (it's kinda similar to how to much monomorphisation is bad for the icache and it's predictors, if we gloss over a ton of technical details) So all in all understand what your tools entail, instead of just blindly using them. And yes that's not easy. It's on of the main differences between a junior and a senior skill level. As a junior you follow rules, guidelines (or imitate other) when to use which tool. As a senior you deeply understand why the rules, guidelines, actions of other people are the way they are and in turn know when to diverge from it.
- rokob 2y agoGiven the date on the post, I can’t tell if this is real.
- epage 2y ago> As a downside, Rust libraries typically have a large fan-out of dependencies, much like Node.js. This requires taking care to avoid the blow-up of dependencies, and to deal with annoyances of dependabot. In the linked situation, the were using the library of a binary. This get into the tension between "make it easy for `cargo install` (and have a `cli` feature be default) and "make it easy for `cargo add` (and make `cli` opt-in). This is not a great experience and we should improve it. There was an RFC to auto-enable features when a build-target is built (allowing `cli` to be opt-in but `cargo install` to auto-opt-in), rather than skip it, but the dev experience needed work, The maintainer can split the package which helps with semver for the two sides but needs to break one side to do so and if its the bin, people need to discover the suffix (`-bin`, `-cli`, etc). Current workarounds: - `cargo add skim` will show the `cli` feature is enabled and you can re-run with `--no-default-features` - if `cli` wasn't a default, `cargo install skim` would suggest adding `--features cli`
- deleted 2y ago[deleted]
- alkonaut 2y agoRe: panics: If you have a single long lived process that must do multiple short-lived things (web requests, say) and a panic in one of them MUST NOT take down the whole process, is that extremely difficult to pull off in Rust? I thought you could set up panic boundaries much like you would use catch-all exception handlers around e.g. each web request or similar, in other languages?
- sph 2y agoSince Rust is not a managed/high-level language, panics are unrecoverable crashes so they need to be dealt at a higher-level, i.e the OS, with appropriate supervisor systems like systemd, or having a master Rust process that spawns subprocesses, and react when one of them abnormally terminates with regular POSIX APIs. On a platform like Elixir, for example, you can deal with process crashes because everything runs on top of a VM, which is at all effects and purposes your OS, and provides process supervision APIs.
- tuetuopay 2y agoThis is factually incorrect. The behavior you describe with Elixir (sic) is precisely what most Rust async runtimes do. (sic because it's Erlang that's to thank) IMHO that is the sensible thing to do for pretty much any green thread or highly concurrent application. e.g. Golang does the same: panicking will only bring down the goroutine and not the whole process.
- pornel 2y agoRust can be optionally compiled in a panic=abort mode, but by default panics are recoverable. From implementation perspective Rust panics are almost identical to C++ exceptions. For servers that must not suddenly die, it's wise to use panic=unwind and catch_unwind at task/request boundaries (https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.h...) In very early pre-1.0 prototypes Rust was meant to have isolated tasks that are killed on panic. As Rust became more low-level, it turned into terminating a whole OS thread on panic, and since Rust 1.9.0, it's basically just a try/catch with usage guidelines.
- 2y ago
- loicalleyne 2y agohttps://github.com/UoCCS/project-GROS https://github.com/UoCCS/project-GROS
- pod_krad 1y agoIn early versions (before 1.0) Rust already had GC. But then it was removed.
- hexo 2y agoThis web is impossible to scroll. Sadly, when I see that I don't think I need to read about any technical stuff they did anymore.