17 ms·
How safe is Zig?
- dleslie 4y agoAnd here is the table with Nim added; though potentially many GC'd languages would be similar to Nim: https://uploads.peterme.net/nimsafe.html https://uploads.peterme.net/nimsafe.html Edit: noteworthy addendum: the ARC/ORC features have been released, so the footnote is now moot.
- jewpfko 4y agoThanks! I'd love to see a Dlang BetterC column too
- Snarwin 4y agoHere's a version with D included: https://gist.github.com/pbackus/0e9c9d0c83cd7d3a46365c054129c554 https://gist.github.com/pbackus/0e9c9d0c83cd7d3a46365c054129... The only difference in BetterC is that you lose access to the GC, so you have to use RC if you want safe heap allocation.
- kzrdude 4y agoI guess rust should say "wraps" for integer overflow as well, as that's what it does in default release compile.
- 3a2d29 4y agoSeeing Nim danger made me think, shouldn’t rust unsafe be added? Seems inaccurate to display rust as safe and not include what actually allows memory bugs to be found in public crates.
- IshKebab 4y agoI don't know why Rust gets "runtime" and Nim gets "compile time" for type confusion?
- shirleyquirk 4y agoyes, for tagged unions specifically, (which the linked post refers to for that row) Nim raises an exception at runtime when trying to access the wrong field, (or trying to change the discriminant)
- ArrayBoundCheck 4y agoI like zig but this is taking a page out of rust book and exaggerating C and C++ clang and gcc will both tell you at runtime if you go out of bounds, have an integer overflow, use after free etc. You need to turn on the sanitizer. You can't have them all on at the same time because code will be unnecessarily slow (ex: having thread sanitizer on in a single threaded app is pointless)
- lijogdfljk 4y agoWhat is the cause of all those notorious C bugs then?
- CodeSgt 4y ago> at runtime
- deleted 4y ago[deleted]
- woodruffw 4y agoNeither Clang nor GCC has perfect bounds or lifetime analysis, since the language semantics forbid it: it's perfectly legal at compile time to address at some offset into a supplied pointer, because the compiler has no way of knowing that the memory there isn't owned and initialized. Sanitizers are great; I love sanitizers. But you can't run them in production without a significant performance hit, and that's where they're needed most. I don't believe this post blows that problem out of proportion, and is correct in noting that we can solve it without runtime instrumentation and overhead.
- AlotOfReading 4y agoState of the art sanitizing is pretty consistently in the <50% overhead range (e.g. SANRAZOR), with things like UBSAN coming in under 10%. If you can't afford even that, tools like ASAP have been around for 7-ish years now to make overhead arbitrarily low by trading off increased false-negatives in hot codepaths. Yes, the "just-enable-the-compiler-flags" approach can be expensive, but the tools exist to allow most people to be sanitizing most of the time. Devs simply don't know what's available to them.
- woodruffw 4y agoThis was a great read, with an important point: there's always a tradeoff to be made, and we can make it (e.g. never freeing memory to obtain temporal memory safety without static lifetime checking). One thought: > Never calling free (practical for many embedded programs, some command-line utilities, compilers etc) This works well for compilers and embedded systems, but please don't do it command-line tools that are meant to be scripted against! It would be very frustrating (and a violation of the pipeline spirit) to have a tool that works well for `N` independent lines of input but not `N + 1` lines.
- anonymoushn 4y agoI've been writing a toy `wordcount` recently, and it seems like if I wanted to support inputs much larger than the ~5GB file I'm testing against, or inputs that contain a lot more unique strings per input file size, I would need to realloc, but I would not need to free.
- woodruffw 4y agoIs that `wordcount` in Zig? My understanding (which could be wrong) is that reallocation in Zig would leave the old buffer "alive" (from the allocator's perspective) if it couldn't be expanded, meaning that you'd eventually OOM if a large enough contiguous region couldn't be found.
- anonymoushn 4y agoIt's in zig but I just call mmap twice at startup to get one slab of memory for the whole file plus all the space I'll need. I am not sure whether Zig's GeneralPurposeAllocator or PageAllocator currently use mremap or not, but I do know that when realloc is not implemented by a particular allocator, the Allocator interface provides it as alloc + memcpy + free. So I think I would not OOM. In safe builds when using GeneralPurposeAllocator, it might be possible to exhaust the address space by repeatedly allocating and freeing memory, but I wouldn't expect to run into this on accident.
- 4y ago
- anonymoushn 4y agoI would like Zig to do more to protect users from dangling stack pointers somehow. I am almost entirely done writing such bugs, but I catch them in code review frequently, and I recently moved these lines out of main() into some subroutine: var fba = std.heap.FixedBufferAllocator.init(slice_for_fba); gpa = fba.allocator(); slice_for_fba is a heap-allocated byte slice. gpa is a global. fba was local to main(), which coincidentally made it live as long as gpa, but then it was local to some setup subroutine called by main(). gpa contains an internal pointer to fba, so you run into trouble pretty quickly when you try allocating memory using a pointer to whatever is on that part of the stack later, instead of your FixedBufferAllocator. Many of the dangling stack pointers I've caught in code review don't really look like the above. Instead, they're dangling pointers that are intended to be internal pointers, so they would be avoided if we had non-movable/non-copyable types. I'm not sure such types are worth the trouble otherwise though. Personally, I've just stopped making structs that use internal pointers. In a typical case, instead of having an internal array and a slice into the array, a struct can have an internal heap-allocated slice and another slice into that slice. like I said, I'd like these thorns to be less thorny somehow.
- throwawaymaths 4y agoThis. I believe it is in the works, but postponed to finish up self-hosted. https://github.com/ziglang/zig/issues/2301 https://github.com/ziglang/zig/issues/2301
- 10000truths 4y agoAlternatively, use offset values instead of internal pointers. Now your structs are trivially relocatable, and you can use smaller integer types instead of pointers, which allows you to more easily catch overflow errors.
- anonymoushn 4y agoThis is a good idea, but native support for slices tempts one to stray from the path.
- alphazino 4y ago
- verdagon 4y agoA lot of embedded devices and safety critical software sometimes don't even use a heap, and instead use pre-allocated chunks of memory whose size is calculated beforehand. It's memory safe, and has much more deterministic execution time. This is also a popular approach in games, especially ones with entity-component-system architectures. I'm excited about Zig for these use cases especially, it can be a much easier approach with much less complexity than using a borrow checker.
- brundolf 4y agoRust's borrow checker would be much calmer in these scenarios too, wouldn't it? If there are no lifetimes, there are no lifetime errors
- thecatster 4y agoRust is definitely different (and calmer imho) on bare metal. That said (as much of a Rust fanboy I am), I also enjoy Zig.
- the__alchemist 4y agoYep! We've entered a grey areas, where some Rust embedded libs are expanding the definitions of memory safety, and what the borrow checker should evaluate beyond what you might guess. Eg, structs that represent peripherals, that are now checked for ownership; the intent being to prevent race conditions. And Traits being used to enforce pin configuration.
- _vvhw 4y agoThis is almost what we do for TigerBeetle, a new distributed database being written in Zig. All memory is statically allocated at startup [1]. Thereafter, there are zero calls to malloc() or free(). We run a single-threaded control plane for a simple concurrency model, and because we use io_uring—multithreaded I/O is less of a necessary evil than it used to be. I find that the design is more memory efficient because of these constraints, for example, our new storage engine can address 100 TiB of storage using only 1 GiB of RAM. Latency is predictable and gloriously smooth, and the system overall is much simpler and fun to program. [1] “Let's Remix Distributed Database Design” https://www.youtube.com/channel/UC3TlyQ3h6lC_jSWust2leGg https://www.youtube.com/channel/UC3TlyQ3h6lC_jSWust2leGg
- afdbcreid 4y agoDo compilers really can never call `free()`? Simple compiler probably can. Most complex probably cannot (I don't want to imagine a Rust compiler without freeing memory: it has 7 layers of lowering (source code->tokens->ast->HIR->THIR->MIR->monomorphized MIR, excluding the final LLVM IR) and also allocates a lot while type-checking or borrow-checking). What is most interesting to me is the average compiler. Does somebody have statistics on the average amount compilers allocate and free?
- com2kid 4y ago> Do compilers really can never call `free()`? I worked on, one of the many, Microsoft compiler teams, though as a software engineer in test not directly on the compiler itself, and I believe the lead dev told me they don't free any memory, though I could be misremembering since it was my first job out of college. Remember C compilers are often one file at a time (and a LOLWTF # of includes), and the majority of work goes into making a single output file, and then you are done. Freeing memory would just take time, better to just hand it all back to the OS. Also compilers are obsessed with correctness, generating incorrect code is to be avoided at all costs. Dealing with memory management is just one more place where things can go wrong. So why bother? I do remember running out of memory using link time code gen though, back when everything was 32bit. Related, I miss the insane dedication to quality that team had. Every single bug had a regression test created for it. We had regression tests 10-15 years old that would find a bug that would have otherwise slipped through. It was a great way to start my career off, just sad I haven't seen testing done at that level since then!
- kaba0 4y agoBootstrapping aside, a compiler written in a GCd language would make perfect sense. It really doesn’t have any reason to go lower level than that (other than of course, if one wants to bootstrap it in the same language that happens to be a low-level one)
- com2kid 4y agoThere is no reason to free memory. Your process is going to hard exit after a set workload. If you wrote a compiler in a GCd language, you'd want to disable the collector because that just takes time, and compilers are slow enough as it is!
- belter 4y ago1 year ago, 274 comments. "How Safe Is Zig?": https://news.ycombinator.com/item?id=26537693 https://news.ycombinator.com/item?id=26537693
- nwellnhof 4y agoUBSan has a -fsanitize-minimal-runtime flag which is supposedly suitable for production: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#minimal-runtime https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#... So it seems that null-pointer dereferences and integer overflows can be checked at runtime in C. Besides, there should be production-ready C compilers that offer bounds checking.
- pjmlp 4y agoThere should but there aren't, GCC had a couple of extensions on a branch like 20 years ago that never got merged. The best is to use C++ instead, with bounds checked library types.
- pjmlp 4y agoThis is why for me, Zig is mostly a Modula-2 with C syntax in regards to safety. All the runtime tooling it offers, already exists for C and C++ for at least 30 years, going back to stuff like Purify (1992).
- lmh 4y agoQuestion for Zig experts: Is it possible, in principle, to use comptime to obtain Rust-like safety? If this was a library, could it be extended to provide even stronger guarantees at compile time, as in a dependent type system used for formal verification? Of course, this does not preclude a similar approach in Rust or C++ or other languages; but comptime's simplicity and generality seem like they might be beneficial here.
- ptato 4y agoNot an expert by any means, but my gut says that it would be very cumbersome and not practical for general use.
- pron 4y agoNot as it is (it would require mutating the type's "state"), but hypothetically, comptime could be made to support even more programmable types. But could doesn't mean should. Zig values language simplicity and explicitness above many other things.
- lmh 4y agoThanks, that's informative. This was meant to clarify the bounds of Zig's design rather than as a research proposal. Otherwise, one might read it as an open invitation to just the sort of demonic meta-thinking that its users abhor.
- anonymoushn 4y agoIt is possible in principle to write a Rust compiler in comptime Zig, but the real answer is "no."
- kristoff_it 4y agoSomebody implemented part of it in the past, but it was based on the ability to observe the order of execution of comptime blocks, which is going to be removed from the language (probably already is). https://github.com/DutchGhost/zorrow https://github.com/DutchGhost/zorrow It's not a complete solution, among other things, because it only works if you use it to access variables, as the language has no way of forcing you.
- tptacek 4y ago"Temporal" and "spatial" is a good way to break this down, but it might be helpful to know the subtext that, among the temporal vulnerabilities, UAF and, to an extent, type confusion are the big scary ones. Race conditions are a big ugly can of worms whose exploitability could probably be the basis for a long, tedious debate. When people talk about Zig being unsafe, they're mostly reacting to the fact that UAFs are still viable in it.
- _vvhw 4y agoI see your UAF and raise you a bleed! As you know, buffer bleeds like Heartbleed and Cloudbleed can happen even in a memory safe language, they're hard to defend against (padding is everywhere in most formats!), easier to pull off than a UAF, often remotely accessible, difficult to detect, remain latent for a long time, and the impact is devastating. All your RAM are belong to us. For me, this can of worms is the one that sits on top of the dusty shelf, it gets the least attention, and memory safe languages can be all the more vulnerable as they lull one into a false sense of safety.
- tptacek 4y agoHas an exploitable buffer bleed (I'm happy with this coinage!) happened in any recent memory safe codebase?
- _vvhw 4y agoI worked on a static analysis tool to detect bleeds in outgoing email attachments, looking for non-zero padding in the ZIP file format. It caught different banking/investment systems written in memory safe languages leaking server RAM. You could sometimes see the whole intranet web page, that the teller or broker used to generate and send the statement, leaking through. Bleeds terrify me, no matter the language. The thing with bleeds is that they're as simple as a buffer underflow, or forgetting to zero padding. Not even the borrow checker can provide safety against that.
- tptacek 4y ago
- dkersten 4y agoI’m not sure I understand the value of an allocator that doesn’t reuse allocations, as a bug prevention thing. Is it just for performance? (Since its never reused, allocation can simply be incrementing an offset by the size of the allocation)? Because beyond that, you can get the same benefit in C by simply never calling free on the memory you want to “protect” against use-after-free.
- kaba0 4y agoI believe it is only for performance, as malloc will have to find place for the allocation, while it is a pointer bump only for a certain kind of allocator.
- anonymoushn 4y agoThe allocations are freed and the addresses are never reused. So heap use-after-frees are segfaults.
- dkersten 4y agoDoes that mean that each allocation is always page-aligned?
- anonymoushn 4y agoNot sure on the details here. I'd have to try it out and see. Larger allocations will be page-aligned, but if you make a bunch of very small allocations, they may go into the same pages, and freeing all but one per page of them may leave you with the pages still mapped. I've skimmed the GeneralPurposeAllocator code and know it has this sort of behavior at least sometimes, but I'm not really familiar with which things change in safe builds.
- dkersten 4y agoThanks for the reply! I wasn't sure if you'd see mine, since it was two days late :) So I suppose the "use after free" protection is "best effort": it will try to trap such use by unmapping the page, but there's no guarantee that it will definitely be the case. I suppose that's ok, since use after free is a bug anyway. We want to catch the bugs and protect against data corruption, but the programmer is still on the hook for not making the bugs in the first place, so if the protection does miss one, well, at least it tried -- its just an extra layer. It would certainly be wasteful if allocating a few bytes uses up an entire page, just so it can be unmapped when the memory is freed.
- ajross 4y ago> In practice, it doesn't seem that any level of testing is sufficient to prevent vulnerabilities due to memory safety in large programs. So I'm not covering tools like AddressSanitizer that are intended for testing and are not recommended for production use. I closed the window right there. Digs like this (the "not recommended" bit is a link to a now famous bomb thrown by Szabolcs on the oss-sec list, not to any kind of industry consensus piece) tell me that the author is grinding an axe and not taking the subject seriously. Security is a spectrum. There are no silver bullets. It's OK to say something like "Rust is better than Zig+ASan because", it's quite another to refuse to even treat the comparison and pretend that hardening tools don't exist. This is fundamentally a strawman, basically. The author wants to argue against a crippled toolchain that is easier to beat instead of one that gets used in practice.
- klyrs 4y agoAs a Zig fan, I disagree. I think it's really important to examine the toolchain that beginners are going to use. > I'm also focusing on software as it is typically shipped, ignoring eg bounds checking compilers like tcc or quarantining allocators like hardened_malloc which are rarely used because of the performance overhead. To advertize that Zig is perfectly safe because things like ASan exist would be misleading, because that's not what users get out of the box. Zig is up-front and honest about the tradeoffs between safety and performance, and this evaluation of Zig doesn't give any surprises if you're familiar with how Zig describes itself.
- ajross 4y ago> To advertize that Zig is perfectly safe because things like ASan exist would be misleading Exactly! And for the same reason. You frame your comparison within the bounds of techniques that are used in practice. You don't refuse to compare a tool ahead of time, especially when doing so reinforces your priors. To be blunt: ASan is great. ASan finds bugs. Everyone should use ASan. Everyone should advocate for ASan. But doing that cuts against the point the author is making (which is basically the same maximalist Rust screed we've all heard again and again), so... he skipped it. That's not good faith comparison, it's spin.
- AndyKelley 4y agoI have one trick up my sleeve for memory safety of locals. I'm looking forward to experimenting with it during an upcoming release cycle of Zig. However, this release cycle (0.10.0) is all about polishing the self-hosted compiler and shipping it. I'll be sure to make a blog post about it exploring the tradeoffs - it won't be a silver bullet - and I'm sure it will be a lively discussion. The idea is (1) escape analysis and (2) in safe builds, secretly heap-allocate possibly-escaped locals with a hardened allocator and then free the locals at the end of their declared scope.
- skullt 4y agoDoes that not contradict the Zig principle of no hidden allocations?
- kristoff_it 4y agoI don't know the precise details of what Andrew has in mind but the compiler can know how much memory is required for this kind of operation at compile time. This is different from normal heap allocation where you only know how much memory is needed at the last minute. At least in simple cases, this means that the memory for escaped variables could be allocated all at once at the beginning of the program not too differently to how the program allocates memory for the stack.
- messe 4y agoStatic allocation at the beginning of the program like that can only work for single threaded programs with non-recursive functions though, right? I’d hazard a guess that the implementation will rely on use-after-free faulting, meaning that the use of any escaped variable will fault rather than corrupting the stack.
- wavesquid 4y agoNo need to limit to single-threaded: as long as we reserve enough space in the TLS area it's possible to work with multiple threads. Zig has future plans to require recursive functions to declare their maximum stack memory usage up-front, so that will provide the rest.
- einpoklum 4y agoOne-liner summary: Zig has run-time protection against out-of-bounds heap access and integer overflow, and partial run-time protection against null pointer dereferencing and type mixup (via optionals and tagged unions); and nothing else.
- avgcorrection 4y agoA meta point to make here but I don’t quite understand the pushback that Rust has gotten. How often does a language come around that flat out eliminates certain errors statically, and at the same time manages to stay in that low-level-capable pocket? And doesn’t require a PhD (or heck, a scholarly stipend) to use? Honestly that might be a once in a lifetime kind of thing. But not requiring a PhD (hyperbole) is not enough: it should be Simple as well. But unfortunately Rust is (mamma mia) Complex and only pointy-haired Scala type architects are supposed to gravitate towards it. But think of what the distinction between no-found-bugs (testing) and no-possible-bugs (a certain class of bugs) buys you; you don’t ever have to even think about those kinds of things as long as you trust the compiler and the Unsafe code that you rely on. Again, I could understand if someone thought that this safety was not worth it if people had to prove their code safe in some esoteric metalanguage. And if the alternatives were fantastic. But what are people willing to give up this safety for? A whole bunch of new languages which range from improved-C to high-level languages with low-level capabilities. And none of them seem to give some alternative iron-clad guarantees. In fact, one of their selling point is mere optionality: you can have some safety and/or you can turn it off in release. So runtime checks which you might (culturally/technically) be encouraged to turn off when you actually want your code to run out in the wild, where users give all sorts of unexpected input (not just your “asdfg” input) and get your program into weird states that you didn’t have time to even think of. (Of course Rust does the same thing with certain non-memory-safety bug checks like integer overflow.)
- kristoff_it 4y ago> Of course Rust does the same thing with certain non-memory-safety bug checks like integer overflow. The problem with getting lost too much in the ironclad certainties of Rust is that you start forgetting that simplicity (papa pia) protects you from other problems. You can get certain programs in pretty messed up states with an unwanted wrap around. Programming is hard. Rust is cool, very cool, but it's not a universal silver bullet.
- avgcorrection 4y agoNothing Is Perfect is a common refrain and non-argument. If option A has 20 defects and option B has the superset of 25 defects then option A is better—the fact that option A has defects at all is completely besides the point with regards to relative measurements.
- LAC-Tech 4y agoSafe enough. You can use `std.testing.allocator` and it will report leaks etc in your test cases. What rust does sounds like a good idea in theory. In practice it rejects too many valid programs, over-complicates the language, and makes me feel like a circus animal being trained to jump through hoops. Zigs solution is hands down better for actually getting work done, plus it's so dead simple to use arena allocation and fixed buffers that you're likely allocating a lot less in the first place. Rust tries to make allocation implicit, leaving you confused when it detects an error. Zig makes memory management explicit but gives you amazing tools to deal with it - I have a much clearer mental model in my head of what goes on. Full disclaimer, I'm pretty bad at systems programming. Zig is the only one I've used where I didn't feel like memory management was a massive headache.
- Klonoar 4y ago>Zigs solution is hands down better for actually getting work done Rust has seen significant usage in large companies; they wouldn't be using it unless it was usable for "real work". >Full disclaimer, I'm pretty bad at systems programming. Zig is the only one I've used where I didn't feel like memory management was a massive headache. I'd say this about Rust, though. Rust's mental model is very straightforward if you accept the borrow-checker and stop fighting it. Can you list any examples of what you think is a headache...? >In practice it rejects too many valid programs, over-complicates the language, and makes me feel like a circus animal being trained to jump through hoops. I've found that jumping through those hoops leads to things running in production that don't make me get up in the middle of the night. Can you show me a "valid program" that Rust rejects?
- LAC-Tech 4y agoRust has seen significant usage in large companies; they wouldn't be using it unless it was usable for "real work". I didn't say it wasn't usable. I said I found Zig more usable. I'd say this about Rust, though. Rust's mental model is very straightforward if you accept the borrow-checker and stop fighting it. Can you list any examples of what you think is a headache...? Mate, I didn't start learning rust in order to wage war against the borrow checker. I had no idea what the hell it wanted a lot of the time. Each time I fixed an error I thought I got it, and each time I was wrong. The grind got boring. As for specific examples no, I've tried to put rust out of my mind. I certainly can't remember specific issues from 3 months ago. I've found that jumping through those hoops leads to things running in production that don't make me get up in the middle of the night. Can you show me a "valid program" that Rust rejects? Yeah that's how rust as sold, the compiler is your friend and stuff will compile and it will never fail. In reality the compiler was so irritating I hardly got anything done at all. The output wasn't super reliable software, it was no software.
- TinkersW 4y agoFalsely representing the state of C & C++ doesn't really lead to a convincing argument. All those safety checks Zig supports are easily enabled in C++, and widely used. Sometimes they are even on by default.
- baby 4y agoSo zig has runtime integer overflow protection by default? That's interesting considering Rust lost that battle
- longrod 4y agoI think Zig has a lot more footguns due to it's explicit nature. When you don't hide away the details from the human, you are increasing the risk of writing bad code and it becomes increasingly harder to make the compiler detect each potentially bad decision. Rust did it but they had to rethink the whole problem from the ground up. Rust is safe but that safety had quite the learning cost as compared to, say, Zig or Go. The good thing about this is that you can't use many of the bad practices from other languages that have become habits. But Zig is still in beta/alpha stage so let's see how they increase the overall safety in the coming months/years. My experience with Zig has left me quite satisfied, especially the comptime features but the explicitness sometimes gets in the way of readability.
- galangalalgol 4y agowhy does go always get brought up when talking about rust or nim or now zig? Its got gc, a large std lib, less latency than java but noticably slower, and its not suited for embedded or drivers. Its a completely different language for a completely different niche than zig or rist, though maybe I could see a comparison to nim... Maybe. I do like go for what it is, batteries included back end language with syntax nicer than java, but I mostly do embedded hpc, so I don't reach for go often. I can believe that abstractions prevent errors, but can't a language allow footguns but have syntax that makes them easy to detect?
- longrod 4y agoGo is a compiled language with comparable performance to Zig/Rust/C in a lot of use cases. It's heavily ergonomic for network/server side programming, it's safe, easy to use with a tiny footprint. Rust is not only or even primarily used in embedded or driver programming. Indeed, I see more user space software built in rust than in embedded nowadays. As for Zig, it's so new and alpha that it didn't even find a niche yet. Rust also hasn't yet found a specific niche. It's all over the place in GUI, networking, embedded etc. which isn't a bad thing, to be clear. It might be harder to do embedded in Go but that's not the point here, is it? It's about safety. Go has a special place in that it is compiled, easy to learn, quite performant for most tasks, and safe to boot. > I can believe that abstractions prevent errors, but can't a language allow footguns but have syntax that makes them easy to detect? What would be the point though? If a language can detect footguns, it's time to prevent them which is what Rust does, essentially. Footguns are rarely useful and there's always an alternative way. For these rare cases, Rust includes the unsafe escape hatch but then all bets are off. Don't expect the compiler to help you if you are intent on going down that road.