6 ms·

It would be interesting to know how such a panic-inducing bug makes it passed Rust's much vaunted borrow-checker and other correctness validations. Or does the

by tux1968 10y ago

It would be interesting to know how such a panic-inducing bug makes it passed Rust's much vaunted borrow-checker and other correctness validations. Or does the regex library include "unsafe" code blocks?

corndoge 10y ago
The borrow checker eliminates data races and hard memory errors, such as access violations. Panics are well within the bounds of safety. From the docs: Unlike C, Undefined Behavior is pretty limited in scope in Rust. All the core language cares about is preventing the following things: * Dereferencing null or dangling pointers * Reading uninitialized memory * Breaking the pointer aliasing rules * Producing invalid primitive values: dangling/null references, a bool that isn't 0 or 1, an undefined enum discriminant, a char outside the ranges [0x0, 0xD7FF] and [0xE000, 0x10FFFF], A non-utf8 str * Unwinding into another language * Causing a data race <snip> Rust is otherwise quite permissive with respect to other dubious operations. Rust considers it "safe" to: * Deadlock * Have a race condition * Leak memory * Fail to call destructors * Overflow integers * Abort the program * Delete the production database https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html
- dom0 10y ago
  Deadlocks and race conditions are two IMHO typical examples of what many developers consider shortcomings of the platform / language, while they are actually application logic bugs that are hard to catch at a language level for all but the most trivial cases. So this is a very nice listing right there, starting with two things that some people may find surprising, then moving towards things that no one should find surprising :)
  - geofft 10y ago
    Note that it's possible to use Rust's typesystem to build things that are deadlock-free and race-condition-free; see, for instance, https://aturon.github.io/blog/2015/08/27/epoch/ https://aturon.github.io/blog/2015/08/27/epoch/ for an API for building lock-free data structures. The assistance you get from the Rust language is much better than the assistance you'd get from C or C++. (Rust has a lot of features other than memory safety!) It's just that Rust doesn't consider it unsafe to build a design that permits deadlocks and race conditions.
  - BuuQu9hu 10y ago
    There are languages like E and Monte where deadlocks aren't possible unless there is an actual loop in the data flow dependencies and race conditions are impossible, but they aren't commonly known.
    - dom0 10y ago
      > unless there is an actual loop in the data flow Hence trivial. Making something provably deadlock-free where the message flow is a DAG is a trivial case. > race conditions are impossible I very much doubt that since race conditions are not only a technical issue (i.e. "is updating that counter atomic?"), but largely a specification issue. Recall that issues like lost updates (or all those file system races) are race conditions, as well as the many issues with non-atomic REST APIs.
burntsushi 10y ago
This bug has nothing to do with memory safety or borrowing. It's just an old fashion logic bug, and the panic prevents any memory unsafety from happening. This bug was in the regex-syntax crate, which doesn't use any unsafe. The regex crate itself does have two uses of unsafe however (one for eliding bounds checks in the inner DFA loop and another for eliding bounds checks in Teddy).
usernam 10y ago
The borrow checker can be [ab]used to guard for several forms of resource usage patterns, but make no mistake here: Rust doesn't mark the end of program bugs (or security bugs). It prevents some forms of bugs. There will still be plenty of bugs to go around.
mattnewton 10y ago
Panics could be a safe way to handle faults at runtime. To me, "Unsafe operations" are more like ones that continue executing in a comprimised state, such as a buffer overflow, or that crash incompletely, segfault, locking up resources in a live lock, etc. Bailing and freeing all your resources seems harder to exploit and sometimes the only sane response.
Manishearth 10y ago
Panics have nothing to do with Rust's safety guarantees or unsafe. A panic is a way of signalling "uh oh the program made an assumption that doesn't hold, there's something horribly wrong, please exit". A fatal exception. Most languages, including Rust, will not let you encode all invariants of a program into the type system. Only languages that come bundled with proof checkers can do that, and that's tedious. So you will have situations where you have to assume an invariant to be true, and if it isn't, there's something horribly wrong and the program should exit. That's a panic. Rust makes sure that wrong assumptions about invariants can't lead to memory unsafety. It can't magically make bugs go away.
chousuke 10y ago
"Safety" doesn't mean your program will never crash. It means that when you mistakenly assume an invariant that does not hold in safe code, the worst that happens is a panic. This is much better than what could happen, since it stops the process in a well defined manner. In a memory-unsafe language, you'll be lucky to get a crash. Instead, you might suffer silent data corruption or a potential exploit, or both. The code here made an assumption that didn't hold, and predictably hit a panic when that assumption was tested.
- Manishearth 10y ago
  Just to clarify here, "crash" in this context means "abort". Some folks use "crash" to mean "segfault", which is related to safety. This can get confusing :)
  - deleted 10y ago
    [deleted]
  - benchaney 10y ago
    I believe a segfault is a type of abort. It is just enforced by the kernel rather than being enforced in userspace. Or is there something else you are getting at?
    - kibwen 10y ago
      A segfault has similarities to an abort, but the importance of the distinction isn't so much in how they're expressed as in the circumstances under which they occur. When someone says "my Rust program crashed", that usually indicates the program deliberately terminating because of a detected or deliberately-unhandled logic bug (like divide-by-zero), which casts no aspersions on the memory-safety of the program. Less frequently "my Rust program crashed" can mean an abort (e.g. because of panicking during unwinding, or because of deliberately turning off unwinding to translate all panics to aborts), but that also has no repercussions for the memory-safety of the program. But if you do get a segfault from a Rust program, that's a big deal, and in Rust terms one would be explicit and say "my Rust program segfaulted" rather than "my Rust program crashed", which leaves no ambiguity as to the severity of the implications.
kibwen 10y ago
I don't think this comment warrants the downvotes it's receiving, as it's entirely reasonable to conflate panics with unsafety if one is only used to "crashing" in the context of C. It's a misconception that we get relatively frequently, so we may just need to make sure this is highlighted in our official tutorials.
- carols10cents 10y ago
  We have a whole chapter on error handling in the second edition of The Rust Programming Language[1] that goes into these issues in detail. When we revise chapter 1 (we're doing that chapter last), we're planning on defining what Rust means by "safe". [1] - http://rust-lang.github.io/book/second-edition/ch09-00-error-handling.html http://rust-lang.github.io/book/second-edition/ch09-00-error...
- j1f4 10y ago
  > reasonable to conflate panics with unsafety if one is only used to "crashing" in the context of C. I suppose it's possible that's what generates most instances of this question, but I had the same question when I was first introduced to Rust and it came from a different root. I had been reading about Erlang, and from that context I expected "safety" to be about avoiding system downtime (e.g. gracefully handling failed asserts) as much as anything.
lmm 10y ago
The code chose to panic. What's interesting is why - it suggests the regex library has some invalid states that are representable, perhaps because the type system wasn't powerful enough to express the constraints that the regex library wanted to apply to its values.
- deleted 10y ago
  [deleted]
- burntsushi 10y ago
  Is there a Turing complete language suitable for writing a fast regex engine with a type system that can prevent all possible invalid states? And if so, how much overhead is associated with said process?
  - lmm 10y ago
    > Is there a Turing complete language suitable for writing a fast regex engine with a type system that can prevent all possible invalid states? I suspect the only way to answer that for certain is to do it. I'd be vaguely interested in trying with Idris[1] or similar. Looking at the code from Reddit it seems like the important distinctions in this instance - between whitespace and not, between the raw input string and the input string with whitespace skipped, between a character that has been confirmed to lie between 0 and 7 and an unknown input character - ought to be trackable in Rust as well though. > And if so, how much overhead is associated with said process? In theory it should be zero overhead at runtime. In practice of course one should have to see how it turned out in benchmarks. [1] Which, sure, would require explicitly choosing possible nontermination when running the regex, if that's the desired behaviour. At one job my team deliberately added a step counter and limit to a regex matcher (abort after 10000 or so) as that was better behaviour than running indefinitely.
    - burntsushi 10y ago
      Sorry, by overhead I meant developer overhead. But I think me and you have had this conversation before, and we fundamentally disagree. Putting a limit on the regex engine won't fly. How big should that limit be? I use this particular regex engine to search files that are many GB in size. Some people might search bigger files. What should the limit be? Limits only work when you can restrict your use cases significantly, which is not a luxury I can enjoy. The broader point I was trying to make with my question is that "invalid states aren't represented" isn't particularly interesting to me on its own. What's interesting to me is what it takes to achieve it. i.e., What do I have to give up to make it so? Developer time? Performance? Code complexity? Other practical things? Keep in mind that I'm after things that work, and not a hypothetically advanced-enough tool.

by tux1968 10y ago

corndoge 10y ago

The borrow checker eliminates data races and hard memory errors, such as access violations. Panics are well within the bounds of safety. From the docs: Unlike C, Undefined Behavior is pretty limited in scope in Rust. All the core language cares about is preventing the following things: * Dereferencing null or dangling pointers * Reading uninitialized memory * Breaking the pointer aliasing rules * Producing invalid primitive values: dangling/null references, a bool that isn't 0 or 1, an undefined enum discriminant, a char outside the ranges [0x0, 0xD7FF] and [0xE000, 0x10FFFF], A non-utf8 str * Unwinding into another language * Causing a data race <snip> Rust is otherwise quite permissive with respect to other dubious operations. Rust considers it "safe" to: * Deadlock * Have a race condition * Leak memory * Fail to call destructors * Overflow integers * Abort the program * Delete the production database https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html

dom0 10y ago
Deadlocks and race conditions are two IMHO typical examples of what many developers consider shortcomings of the platform / language, while they are actually application logic bugs that are hard to catch at a language level for all but the most trivial cases. So this is a very nice listing right there, starting with two things that some people may find surprising, then moving towards things that no one should find surprising :)
- geofft 10y ago
  Note that it's possible to use Rust's typesystem to build things that are deadlock-free and race-condition-free; see, for instance, https://aturon.github.io/blog/2015/08/27/epoch/ https://aturon.github.io/blog/2015/08/27/epoch/ for an API for building lock-free data structures. The assistance you get from the Rust language is much better than the assistance you'd get from C or C++. (Rust has a lot of features other than memory safety!) It's just that Rust doesn't consider it unsafe to build a design that permits deadlocks and race conditions.
- BuuQu9hu 10y ago
  There are languages like E and Monte where deadlocks aren't possible unless there is an actual loop in the data flow dependencies and race conditions are impossible, but they aren't commonly known.
  - dom0 10y ago
    > unless there is an actual loop in the data flow Hence trivial. Making something provably deadlock-free where the message flow is a DAG is a trivial case. > race conditions are impossible I very much doubt that since race conditions are not only a technical issue (i.e. "is updating that counter atomic?"), but largely a specification issue. Recall that issues like lost updates (or all those file system races) are race conditions, as well as the many issues with non-atomic REST APIs.

burntsushi 10y ago

This bug has nothing to do with memory safety or borrowing. It's just an old fashion logic bug, and the panic prevents any memory unsafety from happening. This bug was in the regex-syntax crate, which doesn't use any unsafe. The regex crate itself does have two uses of unsafe however (one for eliding bounds checks in the inner DFA loop and another for eliding bounds checks in Teddy).

usernam 10y ago

The borrow checker can be [ab]used to guard for several forms of resource usage patterns, but make no mistake here: Rust doesn't mark the end of program bugs (or security bugs). It prevents some forms of bugs. There will still be plenty of bugs to go around.

mattnewton 10y ago

Panics could be a safe way to handle faults at runtime. To me, "Unsafe operations" are more like ones that continue executing in a comprimised state, such as a buffer overflow, or that crash incompletely, segfault, locking up resources in a live lock, etc. Bailing and freeing all your resources seems harder to exploit and sometimes the only sane response.

Manishearth 10y ago

Panics have nothing to do with Rust's safety guarantees or unsafe. A panic is a way of signalling "uh oh the program made an assumption that doesn't hold, there's something horribly wrong, please exit". A fatal exception. Most languages, including Rust, will not let you encode all invariants of a program into the type system. Only languages that come bundled with proof checkers can do that, and that's tedious. So you will have situations where you have to assume an invariant to be true, and if it isn't, there's something horribly wrong and the program should exit. That's a panic. Rust makes sure that wrong assumptions about invariants can't lead to memory unsafety. It can't magically make bugs go away.

chousuke 10y ago

"Safety" doesn't mean your program will never crash. It means that when you mistakenly assume an invariant that does not hold in safe code, the worst that happens is a panic. This is much better than what could happen, since it stops the process in a well defined manner. In a memory-unsafe language, you'll be lucky to get a crash. Instead, you might suffer silent data corruption or a potential exploit, or both. The code here made an assumption that didn't hold, and predictably hit a panic when that assumption was tested.

Manishearth 10y ago
Just to clarify here, "crash" in this context means "abort". Some folks use "crash" to mean "segfault", which is related to safety. This can get confusing :)
- deleted 10y ago
  [deleted]
- benchaney 10y ago
  I believe a segfault is a type of abort. It is just enforced by the kernel rather than being enforced in userspace. Or is there something else you are getting at?
  - kibwen 10y ago
    A segfault has similarities to an abort, but the importance of the distinction isn't so much in how they're expressed as in the circumstances under which they occur. When someone says "my Rust program crashed", that usually indicates the program deliberately terminating because of a detected or deliberately-unhandled logic bug (like divide-by-zero), which casts no aspersions on the memory-safety of the program. Less frequently "my Rust program crashed" can mean an abort (e.g. because of panicking during unwinding, or because of deliberately turning off unwinding to translate all panics to aborts), but that also has no repercussions for the memory-safety of the program. But if you do get a segfault from a Rust program, that's a big deal, and in Rust terms one would be explicit and say "my Rust program segfaulted" rather than "my Rust program crashed", which leaves no ambiguity as to the severity of the implications.

kibwen 10y ago

I don't think this comment warrants the downvotes it's receiving, as it's entirely reasonable to conflate panics with unsafety if one is only used to "crashing" in the context of C. It's a misconception that we get relatively frequently, so we may just need to make sure this is highlighted in our official tutorials.

carols10cents 10y ago
We have a whole chapter on error handling in the second edition of The Rust Programming Language[1] that goes into these issues in detail. When we revise chapter 1 (we're doing that chapter last), we're planning on defining what Rust means by "safe". [1] - http://rust-lang.github.io/book/second-edition/ch09-00-error-handling.html http://rust-lang.github.io/book/second-edition/ch09-00-error...
j1f4 10y ago
> reasonable to conflate panics with unsafety if one is only used to "crashing" in the context of C. I suppose it's possible that's what generates most instances of this question, but I had the same question when I was first introduced to Rust and it came from a different root. I had been reading about Erlang, and from that context I expected "safety" to be about avoiding system downtime (e.g. gracefully handling failed asserts) as much as anything.

lmm 10y ago

The code chose to panic. What's interesting is why - it suggests the regex library has some invalid states that are representable, perhaps because the type system wasn't powerful enough to express the constraints that the regex library wanted to apply to its values.

deleted 10y ago
[deleted]
burntsushi 10y ago
Is there a Turing complete language suitable for writing a fast regex engine with a type system that can prevent all possible invalid states? And if so, how much overhead is associated with said process?
- lmm 10y ago
  > Is there a Turing complete language suitable for writing a fast regex engine with a type system that can prevent all possible invalid states? I suspect the only way to answer that for certain is to do it. I'd be vaguely interested in trying with Idris[1] or similar. Looking at the code from Reddit it seems like the important distinctions in this instance - between whitespace and not, between the raw input string and the input string with whitespace skipped, between a character that has been confirmed to lie between 0 and 7 and an unknown input character - ought to be trackable in Rust as well though. > And if so, how much overhead is associated with said process? In theory it should be zero overhead at runtime. In practice of course one should have to see how it turned out in benchmarks. [1] Which, sure, would require explicitly choosing possible nontermination when running the regex, if that's the desired behaviour. At one job my team deliberately added a step counter and limit to a regex matcher (abort after 10000 or so) as that was better behaviour than running indefinitely.
  - burntsushi 10y ago
    Sorry, by overhead I meant developer overhead. But I think me and you have had this conversation before, and we fundamentally disagree. Putting a limit on the regex engine won't fly. How big should that limit be? I use this particular regex engine to search files that are many GB in size. Some people might search bigger files. What should the limit be? Limits only work when you can restrict your use cases significantly, which is not a luxury I can enjoy. The broader point I was trying to make with my question is that "invalid states aren't represented" isn't particularly interesting to me on its own. What's interesting to me is what it takes to achieve it. i.e., What do I have to give up to make it so? Developer time? Performance? Code complexity? Other practical things? Keep in mind that I'm after things that work, and not a hypothetically advanced-enough tool.