5 ms·

There's no need for uncertainty about what Zig is going for. Rust code blows up when the heap runs out regardless of how carefully you write it unless you want

There's no need for uncertainty about what Zig is going for. Rust code blows up when the heap runs out regardless of how carefully you write it unless you want to forego the standard library and whatever depends on it. Zig gives you control over memory allocation and hence at least the potential to write correct code.

scottlamb 8y ago
> There's no need for uncertainty about what Zig is going for. Rust code blows up when the heap runs out regardless of how carefully you write it unless you want to forego the standard library and whatever depends on it. Zig gives you control over memory allocation and hence at least the potential to write correct code. I hope that's not it. Talk about throwing out the baby with the bathwater. It's true that Rust's standard library panics on ENOMEM, but not only is this not inherent to the language, it's possible to extend the standard library, and folks are doing so. See https://github.com/rust-lang/rust/issues/48043 https://github.com/rust-lang/rust/issues/48043 for support of "fallible allocations".
- gradschool 8y ago
  I agree that Rust has a lot of good ideas in it. The notion of ownership isn't far off my own way thinking about pointers when writing code with pointers in it, and I welcome any help from the compiler at avoiding my stupid mistakes, but if worse comes to worst, at least my mistakes are fixable. In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age.
  - steveklabnik 8y ago
    The decision was made with pragmatism in mind. What do most programs in Rust want to do? Most of them are fine with panic on an error. Making your code robust against this, at least in Rust, makes things a lot more cumbersome, and most programs would panic upon such an error anyway. Most programs that care about this possibility aren't using the standard library anyway. Therefore, for the standard library, the decision was made to panic on OOM, and to not block the release of Rust 1.0 on something that, at the time, wasn't itself a stable feature. Three years later, I'm comfortable with this decision; three more years of waiting on Rust 1.0 would have killed it as a language. Many, many useful programs have been written in Rust since then. Of course, you can disagree with this decision; I mostly want to communicate that it was not a flippant "who needs to care about that," but a careful decision made after weighing the tradeoffs. In the old days, those tradeoffs were very different.
    - AnIdiotOnTheNet 8y ago
      It seems like a weird choice to me. Zig's standard library functions that require allocation take the allocator as a parameter, and return oom errors back up to the caller to do with as they please. Usually, the caller will also just kick that error up until it reaches the top of the call stack and gets caught by the default panic handler, but at any point they can choose to handle it however they want. Handling the error by kicking it up the call stack is a single keyword at the callsite. As a bonus, this means that when you're reading code you can easily spot the function calls that have a possibility of failing.
      - steveklabnik 8y ago
        Yeah, I specifically added the "in Rust" here because I know Zig really cares about this problem. I haven't actually used contemporary Zig enough to be able to say anything about its approach, though. > Zig's standard library functions that require allocation take the allocator as a parameter, and return oom errors back up to the caller to do with as they please. You could do something like this in Rust. Here's a function that accepts a vector and pushes something onto it: fn foo(v: &mut Vec<i32>, item: i32) { v.push_back(item); } (Keeping it non-generic to simplify.) Does the data structure hold the allocator, or the function? What if I created a list with one allocator, but called push_back with another? It'd have to be in the structure, right? Given that, here's what this feature would look like, roughly, if Vec<T> supported it: fn foo<A: Allocator>(v: &mut Vec<i32, A>, item: i32) -> Result<(), Box<Error>> { v.push_back(item)?; } This is significantly more boilerplate. The type signature is more than twice as long! You'd also have to adjust the caller: // before foo(&mut v, 5); // after foo(&mut v, 5)?; Of course, it'd be possible to maybe reduce this with langauge features. What does this look like in Zig?
        AndyKelley 8y ago
        Here's an excerpt from std.ArrayList[1]: pub fn ArrayList(comptime T: type) type { return struct { const Self = @This(); raw_items: []T, len: usize, allocator: *Allocator, // ... pub fn append(self: *Self, item: T) !void { const new_item_ptr = try self.addOne(); new_item_ptr.* = item; } pub fn addOne(self: *Self) !*T { const new_length = self.len + 1; try self.ensureCapacity(new_length); return self.addOneAssumeCapacity(); } // ... }; } test "std.ArrayList.basic" { var bytes: [1024]u8 = undefined; const allocator = &std.heap.FixedBufferAllocator.init(bytes[0..]).allocator; var list = ArrayList(i32).init(allocator); try list.append(1); try list.append(2); try list.append(3); assert(list.len == 3); } In summary, functions that can fail have an inferred error set, or an explicit one[2]. At the callsite of functions that can fail will either be `if`, `while`, `catch`, or `try`, in order to deal with the error case. [1]: https://github.com/ziglang/zig/blob/e3d8cae35a5d194386eacd9afa4f496111a37e16/std/array_list.zig https://github.com/ziglang/zig/blob/e3d8cae35a5d194386eacd9a... [2]: https://ziglang.org/documentation/master/#Error-Set-Type https://ziglang.org/documentation/master/#Error-Set-Type
  - pcwalton 8y ago
    > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Ever since Unix had copy-on-write data pages, it has had effectively the same behavior around OOM. The semantics of fork() require it, unless you're OK with the idea of a process that uses more than half of the available memory being unable to launch any child processes. So the idea of killing processes on OOM is very old indeed.
    - AndyKelley 8y ago
      Windows doesn't overcommit. On Linux, you can turn it off. And on Linux, we have vfork.
      - pcwalton 8y ago
        Because Windows (roughly speaking) doesn't use the fork/exec model for process spawning, it doesn't need overcommit. If you use fork/exec, then overcommit is the least bad of all the bad options. Anyway, whether overcommit is a good idea is somewhat tangential. The point is that portable programs cannot rely on being able to detect and recover from OOM in the same process. That's why you see so few programs try. (The programs that do try frequently suffer from security vulnerabilities in their OOM handling paths, ironically enough.)
        AndyKelley 8y ago
        > If you use fork/exec, then overcommit is the least bad of all the bad options. vfork is better than fork/exec with overcommit. It's interesting to me that you mention portability as an argument for why an out of memory situation should cause a crash rather than be handled like any other error. Let's say I make a library and I want this library to be maximally portable. That means it should run in some of the following environments: * Microsoft Windows * embedded software * operating system kernels * drivers * real-time software In all of these environments, a library that handles out of memory correctly would be more portable than a library that panics. If only there was a programming language that helped programmers solve this problem, while preventing the pitfalls that result in these security vulnerabilities that you mention...
  - scottlamb 8y ago
    > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Do you run on Linux? Do you leave vm.overcommit_memory at the default 0? Do you run on 64-bit platforms? Most people here I imagine would say yes, yes, and yes. If so, ENOMEM is not a thing that happens. Some process on the machine will get OOM-killed instead. It's silly to write a bunch of dead code.
  - scottlamb 8y ago
    > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Ignoring my other point (that ENOMEM doesn't happen on the most common deployment environment), what do you think the correct ENOMEM behavior should be? In many situations, I think it's to crash (what Rust does anyway). For example, at work I write cloud servers, which are replicated. If there's say a memory leak that is causing it to have not quite enough RAM, I'd rather it go down and restart cleanly that try to limp on. The other servers will take the load in the meantime. (They might all crash in waves, which is sad but better that than all being unable to serve well and lingering in that state. If you don't crash, then you need some external agent that decides when to restart them, and you end up with a pretty similar outcome with more complexity and rarely exercised logic.) There are certainly specialized environments where you want to do something else, but it sounds like you're saying it's thoughtless/wrong for the program to crash, and in general I disagree.

scottlamb 8y ago

> There's no need for uncertainty about what Zig is going for. Rust code blows up when the heap runs out regardless of how carefully you write it unless you want to forego the standard library and whatever depends on it. Zig gives you control over memory allocation and hence at least the potential to write correct code. I hope that's not it. Talk about throwing out the baby with the bathwater. It's true that Rust's standard library panics on ENOMEM, but not only is this not inherent to the language, it's possible to extend the standard library, and folks are doing so. See https://github.com/rust-lang/rust/issues/48043 https://github.com/rust-lang/rust/issues/48043 for support of "fallible allocations".

gradschool 8y ago
I agree that Rust has a lot of good ideas in it. The notion of ownership isn't far off my own way thinking about pointers when writing code with pointers in it, and I welcome any help from the compiler at avoiding my stupid mistakes, but if worse comes to worst, at least my mistakes are fixable. In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age.
- steveklabnik 8y ago
  The decision was made with pragmatism in mind. What do most programs in Rust want to do? Most of them are fine with panic on an error. Making your code robust against this, at least in Rust, makes things a lot more cumbersome, and most programs would panic upon such an error anyway. Most programs that care about this possibility aren't using the standard library anyway. Therefore, for the standard library, the decision was made to panic on OOM, and to not block the release of Rust 1.0 on something that, at the time, wasn't itself a stable feature. Three years later, I'm comfortable with this decision; three more years of waiting on Rust 1.0 would have killed it as a language. Many, many useful programs have been written in Rust since then. Of course, you can disagree with this decision; I mostly want to communicate that it was not a flippant "who needs to care about that," but a careful decision made after weighing the tradeoffs. In the old days, those tradeoffs were very different.
  - AnIdiotOnTheNet 8y ago
    It seems like a weird choice to me. Zig's standard library functions that require allocation take the allocator as a parameter, and return oom errors back up to the caller to do with as they please. Usually, the caller will also just kick that error up until it reaches the top of the call stack and gets caught by the default panic handler, but at any point they can choose to handle it however they want. Handling the error by kicking it up the call stack is a single keyword at the callsite. As a bonus, this means that when you're reading code you can easily spot the function calls that have a possibility of failing.
    - steveklabnik 8y ago
      Yeah, I specifically added the "in Rust" here because I know Zig really cares about this problem. I haven't actually used contemporary Zig enough to be able to say anything about its approach, though. > Zig's standard library functions that require allocation take the allocator as a parameter, and return oom errors back up to the caller to do with as they please. You could do something like this in Rust. Here's a function that accepts a vector and pushes something onto it: fn foo(v: &mut Vec<i32>, item: i32) { v.push_back(item); } (Keeping it non-generic to simplify.) Does the data structure hold the allocator, or the function? What if I created a list with one allocator, but called push_back with another? It'd have to be in the structure, right? Given that, here's what this feature would look like, roughly, if Vec<T> supported it: fn foo<A: Allocator>(v: &mut Vec<i32, A>, item: i32) -> Result<(), Box<Error>> { v.push_back(item)?; } This is significantly more boilerplate. The type signature is more than twice as long! You'd also have to adjust the caller: // before foo(&mut v, 5); // after foo(&mut v, 5)?; Of course, it'd be possible to maybe reduce this with langauge features. What does this look like in Zig?
      - AndyKelley 8y ago
        Here's an excerpt from std.ArrayList[1]: pub fn ArrayList(comptime T: type) type { return struct { const Self = @This(); raw_items: []T, len: usize, allocator: *Allocator, // ... pub fn append(self: *Self, item: T) !void { const new_item_ptr = try self.addOne(); new_item_ptr.* = item; } pub fn addOne(self: *Self) !*T { const new_length = self.len + 1; try self.ensureCapacity(new_length); return self.addOneAssumeCapacity(); } // ... }; } test "std.ArrayList.basic" { var bytes: [1024]u8 = undefined; const allocator = &std.heap.FixedBufferAllocator.init(bytes[0..]).allocator; var list = ArrayList(i32).init(allocator); try list.append(1); try list.append(2); try list.append(3); assert(list.len == 3); } In summary, functions that can fail have an inferred error set, or an explicit one[2]. At the callsite of functions that can fail will either be `if`, `while`, `catch`, or `try`, in order to deal with the error case. [1]: https://github.com/ziglang/zig/blob/e3d8cae35a5d194386eacd9afa4f496111a37e16/std/array_list.zig https://github.com/ziglang/zig/blob/e3d8cae35a5d194386eacd9a... [2]: https://ziglang.org/documentation/master/#Error-Set-Type https://ziglang.org/documentation/master/#Error-Set-Type
- pcwalton 8y ago
  > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Ever since Unix had copy-on-write data pages, it has had effectively the same behavior around OOM. The semantics of fork() require it, unless you're OK with the idea of a process that uses more than half of the available memory being unable to launch any child processes. So the idea of killing processes on OOM is very old indeed.
  - AndyKelley 8y ago
    Windows doesn't overcommit. On Linux, you can turn it off. And on Linux, we have vfork.
    - pcwalton 8y ago
      Because Windows (roughly speaking) doesn't use the fork/exec model for process spawning, it doesn't need overcommit. If you use fork/exec, then overcommit is the least bad of all the bad options. Anyway, whether overcommit is a good idea is somewhat tangential. The point is that portable programs cannot rely on being able to detect and recover from OOM in the same process. That's why you see so few programs try. (The programs that do try frequently suffer from security vulnerabilities in their OOM handling paths, ironically enough.)
      - AndyKelley 8y ago
        > If you use fork/exec, then overcommit is the least bad of all the bad options. vfork is better than fork/exec with overcommit. It's interesting to me that you mention portability as an argument for why an out of memory situation should cause a crash rather than be handled like any other error. Let's say I make a library and I want this library to be maximally portable. That means it should run in some of the following environments: * Microsoft Windows * embedded software * operating system kernels * drivers * real-time software In all of these environments, a library that handles out of memory correctly would be more portable than a library that panics. If only there was a programming language that helped programmers solve this problem, while preventing the pitfalls that result in these security vulnerabilities that you mention...
- scottlamb 8y ago
  > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Do you run on Linux? Do you leave vm.overcommit_memory at the default 0? Do you run on 64-bit platforms? Most people here I imagine would say yes, yes, and yes. If so, ENOMEM is not a thing that happens. Some process on the machine will get OOM-killed instead. It's silly to write a bunch of dead code.
- scottlamb 8y ago
  > In the old days it would have been considered bad practice to postpone considerations of storage limitations to some future release, but maybe I'm just showing my age. Ignoring my other point (that ENOMEM doesn't happen on the most common deployment environment), what do you think the correct ENOMEM behavior should be? In many situations, I think it's to crash (what Rust does anyway). For example, at work I write cloud servers, which are replicated. If there's say a memory leak that is causing it to have not quite enough RAM, I'd rather it go down and restart cleanly that try to limp on. The other servers will take the load in the meantime. (They might all crash in waves, which is sad but better that than all being unable to serve well and lingering in that state. If you don't crash, then you need some external agent that decides when to restart them, and you end up with a pretty similar outcome with more complexity and rarely exercised logic.) There are certainly specialized environments where you want to do something else, but it sounds like you're saying it's thoughtless/wrong for the program to crash, and in general I disagree.