12 ms·

Pre-Pooping Your Pants with Rust

129 points by Manishearth 12y ago

VeejayRampay 12y ago
A quick observation from an outsider: The article goes into a lot of depth, seems highly technical, well crafted and researched but the whole "pre-poop your pants" thing just ruins all that and makes it looks childish. I understand that not everything about development has to be serious, but I also think that aligning form and content is the surest way to make sure your message comes across.
- golergka 12y ago
  I guess it's a personal thing. I don't know much about Rust, but the title explained the severity of the problem pretty good right away.
  - digitalzombie 12y ago
    I'm hoping this design pattern name catch on. PPYP design pattern. Sometime programming can be boring and then you come across PPYP and it makes it fun again.
    - Dewie2 12y ago
      First it was dry and boring, and then someone mentioned "poop" and then the same subject was hilarious! I don't get it...
      - WhatWouldLogDo 12y ago
        Analytic. Human beings still feel funny about Depth Psycho(techno)logy. Parallel processing pathos of human life terms with the logos apath programming words, represents multiple realities, versus the monotheistic standard robo swears robo cares.
- pluma 12y ago
  First Ruby, then JS, now Rust.
  - th0br0 12y ago
    What is your point?
    - iagooar 12y ago
      I guess the parent means something that has to do with the community being a bit hipster / infantile? Only guessing though.
      - WhatWouldLogDo 12y ago
        I'd rather have rules to let you guess free (than have the memory leak of voting insight by iff it makes you/language feel good about problems.)
      - pluma 12y ago
        The entire infantile humour / brogrammer / hipster / drama phenomenon. It started in Ruby (_why, Zed Shaw, GitHub, etc) then moved into JS as Node and CoffeeScript became popular with Rails users, now it's in Rust (again via Ruby). I don't think it's always the same people or that it has anything to do with Ruby itself, it just seems like there's a certain culture that first grew in the Ruby / Rails community.
        steveklabnik 12y ago
        FWIW I actually thought the title detracted from the otherwise excellent contents of the post.
        dbaupp 12y ago
        This post is the first infantile thing in this vein I can remember seeing related to Rust, and, as far as I know, the author has no connection to the Ruby community you describe, they're just choosing poor titles of their own accord (I agree with Steve that the title detracts).
- lifthrasiir 12y ago
  I'll reuse my reply on /r/rust to defend the naming: > I like the naming and feeling of "Pre-Pooping Your Pants" pattern. We are doing the intentional leak only to be (hopefully) collected later, which may not happen but it still remains memory-safe even in that case. Still this is not very desirable and the name clearly indicates so.
  - lighthawk 12y ago
    A term for code written in an undesirable way to tie you over in the short-term is a "hack". Not all hacks are bad. But, if I see "// this is a hack to..." or "# this is a hack to..." or similar in code, that is a signal to me that the developer (1) either didn't have time to perform the right thing but could have with more time (within their lifetime), or (2) could not feasibly perform the right thing under constraints given (within their lifetime). Over the past ~8 years, "hack"'s use has switched back to a much more positive sense- possibly because of HN, or possibly because people read Eric Raymond http://www.catb.org/esr/faqs/hacker-howto.html http://www.catb.org/esr/faqs/hacker-howto.html, but "hack" != "hacker". The negative sense is still used and understood by most developers, and it comes from two things: "HACK... HACK... HACK...", like the sound of an ax attempting to chop down a tree (brute force), and the other definition of a hack not used as much these days, which is "work for hire especially with mediocre professional standards". I think we should at least consider PPYP to be a subclass of "hack", or perhaps just use the word "hack" instead of PPYP.
    - jdub 12y ago
      kludge is a better term for an undesirable short-term fix. hack implies clever or elegant.
- mrbig4545 12y ago
  I couldn't agree more. the article is well written, but the title puts me off a lot. How am I going to show this to my boss, and have him take it seriously with such an infantile title?
  - kyberias 12y ago
    Why would you want to show it to your boss?
    - mrbig4545 12y ago
      I tend to keep him updated with rust news, so that when the day comes, I've laid enough groundwork to be able to do something in it without having to explain what it is and why etc. And part of that is showing flaws, and how they're resolved/worked around and so on.
  - nosefrog 12y ago
    You have the right to your own feelings, and I agree that the title may be off putting. The audience for the article is rust core contributors and rust fans, and they're a very informal bunch. If rusters are like similar communities I know, talking about poop makes a blog post more likely to spread. EDIT: I do not want to argue with Dewey2, so I'm putting this here: the women in the rust community that feel uncomfortable when they are addressed as "guys" have every right to feel that way, and I commend the rust community on making that stand.
    - Dewie2 12y ago
      A community that is informal enough to give analogies involving poop, but too uptight to stomach the gender-neutral use of the word "guys"[1]. Makes sense. [1] http://www.reddit.com/r/rust/comments/1nvsdh/a_note_on_conduct_please_read/ccmnjdj http://www.reddit.com/r/rust/comments/1nvsdh/a_note_on_condu...
      - Ygg2 12y ago
        Because toilet humor and gender discrimination are the same?
- nosefrog 12y ago
  I think Pre-Poop Your Pants is a really clever way to package the core idea. I agree that the article is well crafted, and I don't think it's childish to talk about poop in this context.
- Manishearth 12y ago
  It's not an "official" article, just a blog post by a contributor. Informal is okay IMO.
radulemnaru 12y ago
This title made my day!
bgdnpn 12y ago
I really like the title and the first sentence. "Much existential anguish and ennui was recently triggered.."
PieSquared 12y ago
I hope that the Rust devs end up doing the "right thing", which from this article seems like going back and fixing Rc, instead of just marking mem::forget as safe. They're still pre-1.0, and anything they do now they will be stuck with for a long, long time, especially if Rust succeeds as much as I hope it does. Delaying 1.0 by a few weeks may seem like a big deal, but ultimately it is a self-imposed deadline. It's great to have those, but following them dogmatically might not be the best strategy. In cases like this, I generally lean towards slowing down and doing things right: otherwise you will pay the price ten times over later. (That said, while I understand this issue, I don't know very much about the context in the Rust community, so I'm not actually sure that mem::forget should be unsafe. It was just the impression from the article and from previous Rust code I've read/written.)
- Jweb_Guru 12y ago
  There isn't an acceptable way to fix Rc. It wouldn't be hard to require it not to have cycles, but cyclic data is actually a major usecase for Rc. A cycle collector wouldn't fix any of the unsafety here unless it was run on literally every drop, which would be far too slow to be acceptable in a systems language. An opt-in trait to prevent leaking specific types would be practical, but the implementation details aren't settled and it can be done backwards-compatibly, so there's a strong feeling that it shouldn't block the release. Any other attempts to determine whether cyclic data structures always eventually drop their contents enter very-active-research territory and would be more like a ten year than a few week thing.
  - deleted 12y ago
    [deleted]
  - pcwalton 12y ago
    Precisely. Rc is not easy to "fix": the fact that it allows cycles is part of its design.
    - mcguire 12y ago
      There's a reason why finalizers aren't a widely used technique in Java and why Python's cycle detector simply stacks things with destructors in an out-of-the-way location.
zenojevski 12y ago
Why on Earth are we still using such unsafe languages? If the author had written this in a modern, safe language like Rust this would not have happened.
- Ygg2 12y ago
  Funny. But the problem here is complex and hard to solve. Cycles and memory leaks are nigh impossible to solve completely without sacrificing on that altar ability to create custom data structures or no-GC by default. Which was non-starter for Rust.
  - krick 12y ago
    Well, I'm still not quite proficient with Rust, so maybe my feelings are misguided, but the whole situation seems really terrifying to me. Like, way too terrifying than that we could just make a point about "not doing like that again" and calmly move on to the 1.0. After all, why Rust? Because we're all tired of problems with C++ and stuff. And the point isn't really about making a language that would be safe "as often as possible", it would be quite terrible goal, actually. The point is, I believe, to make it nearly impossible to write unsafe code accidentally, without even noticing it. And again, maybe it's just me, but what I understood from that post is totally non-intuitive to me. I don't really understand what are the guidelines and how many more gotchas like that there could be. "Not to mess with unsafe code" is not really good guideline, because it turns out that it can be not quite obvious that the code is unsafe.
    - Ygg2 12y ago
      Well, the thing is, until Rust is formally tested, as in a subset of Rust is proven in CoQ to be safe, you don't have any guarantees. I assume much of it is safe, with few probably hidden caveats, hidden in more complex logic. In C++ writing code that causes this behavior is trivial, here you need to jump through a lot of hoops. So it offers you comparatively more safety, but this kind of bugs are potential gold mine for crackers. Anyway core developers are on the scene deciding how to tackle this issue.
garethrees 12y ago
There is a general problem with combining destructors and automatic collection of reference cycles, and its one that several languages have unexpectedly found themselves facing. For example, Java has the function System.runFinalizersOnExit, but if you look at the documentation you'll see that it now says, "Deprecated. This method is inherently unsafe. It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in erratic behavior or deadlock." http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#runFinalizersOnExit(boolean) http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.h... The problem is discussed in detail in Hans Boehm's 2002 technical report “Destructors, Finalizers, and Synchronization”. http://www.hpl.hp.com/techreports/2002/HPL-2002-335.pdf http://www.hpl.hp.com/techreports/2002/HPL-2002-335.pdf Boehm's key points are: (1) When a destructor on an object O runs normally, anything that O points to is still alive (because of the reference from O), but when a cycle is collected, not everyone can go first: that is, all but one destructors in the cycle have to run after one or more of their references are dead. It is hard to write destructor code that is safe in all cases. (2) Any destructor that needs to update a concurrently accessed data structure has to take a lock, but destructors on cycles run asynchronously with respect to the rest of the program and so an unlucky timing leads to deadlock. If this problem was better appreciated then language designers wouldn't go down the rabbit hole of trying to figure out how to make destructors work together with automatic collection of reference cycles, and instead try the alternative approach of providing mechanisms for the program to run destructors synchronously. In the Memory Pool System we use a message-passing interface: http://www.ravenbrook.com/project/mps/master/manual/html/topic/finalization.html http://www.ravenbrook.com/project/mps/master/manual/html/top...
charlieflowers 12y ago
Call Scott Meyers. So you cannot safely assume your destructors will be called, and you need to proactively take steps like this in case they aren't. Nothing in the natural use of the language cues you to do this, so you need a moral guide like "Effective C++" for Rust. This was the kind of accidental complexity that Rust had hoped to avoid. It seems like Rust has avoided a lot of it. But I guess it was entirely too optimistic to hope Rust could avoid all of it.
- Gankro 12y ago
  Note that this is only something you really need to think about when writing `unsafe` code, which basically unlocks a whole new set of rules that you need to worry about (zero-sized types in offsets, manual allocations, safety boundaries, GEP/aliasing rules, uninit memory, double destructors, etc). This is a drop in the bucket that anyone venturing to use `unsafe` will have to face in some kind of "guide to unsafety".
  - mcguire 12y ago
    The problem is that all of this is safe code. The problem, to my mind, is that you can write mem::forget (as in the code here's safe_forget)* without ever using the unsafe keyword*. Edit: I was wrong. The problem isn't with forget. This is hairy.
    - steveklabnik 12y ago
      The 'mem::forget can be written in safe code' meme comes from https://github.com/rust-lang/rust/issues/24456 https://github.com/rust-lang/rust/issues/24456 Which, as you can see, uses Rc, which uses unsafe code in a way that leads to the soundness bug. So that's not strictly true, or rather, doesn't really change anything, as it relies on the same bug.
      - mcguire 12y ago
        nikomatsakis, in that thread: "1. There were other ways to forget even before Rc, at least in some cases. For example, if T:Send holds, you could send the value to a thread that runs an infinite loop, or which is deadlocked on a port. "2. It is true that one cannot assume that a destructor will run, and hence that forget is not itself unsafe (rather, it is unsafe to write a dtor that must run)...." And alexcrichton: "I commented on #24292, but the gist is that there are multiple ways to leak memory today (e.g. #14875 and #16135), so a targeted solution at Rc may not cover all use cases. Although as I mention in #24292 these other bugs can also be considered separate bugs on their own which need to be fixed regardless (but sometimes is quite difficult to do so)." I especially liked dgrunwald's comment: "A safe mem::forget has the advantage that it makes it easier to write the counterexample proving thread::JoinGuard unsafe. Safe mem::forget makes it more likely that people will know that destructors are not realiable, so they can avoid repeating this mistake." But then, "Put a big freaking spike in the middle of the steering wheel and get rid of the airbags and seat belts" has always appealed to me as an automotive safety approach.
Gankro 12y ago
Boring edition for people who have feels about maturity: http://cglab.ca/~abeinges/blah/everyone-peaches/ http://cglab.ca/~abeinges/blah/everyone-peaches/ ( people getting mad about PPYP is really a victory, though; the second last bullet was the most important: > It's vaguely incoherent and meaningless on its own: this is a property inherited from its connection to the venerable Resource Acquisition Is Initialization (RAII) pattern. I'm mad about RAII and not going to take it anymore! )
mcguire 12y ago
Largely unrelated question: In the code about a third of the way into the article, there is, impl<'a> Drop for Foo<'a> { fn drop(&mut self) { *self.0 += 1; } } What does the line "self.0 += 1;" do? Specifically, the ".0"? Foo is a basically a reference to an i32, so I would have thought that would be "self += 1;".
- steveklabnik 12y ago
  The .0 is a tuple access. Foo is a 'tuple struct' of length 1: http://doc.rust-lang.org/nightly/book/tuple-structs.html http://doc.rust-lang.org/nightly/book/tuple-structs.html (though the syntax is explained in the section on tuples: http://doc.rust-lang.org/nightly/book/primitive-types.html#tuple-indexing http://doc.rust-lang.org/nightly/book/primitive-types.html#t... )
  - mcguire 12y ago
    Ah! Cool. Thanks!
sirclueless 12y ago
Putting stuff in a HashMap that you'll never ask for again is a kind of leak. Allocating stuff on the stack and then looping forever is a kind of leak. The case where you put it on the heap and forget to ever free it is a specific kind of leak that people seem particularly terrified of for whatever reason. I too am particularly terrified of this kind of leak. The reason is that it is the only kind of leak mentioned that is not clearly a programmer mistake. C++ has a whole class of "mistakes" that one cannot be aware of simply by reading the language specification. Common ones such as writing a constructor and destructor but not a copy-constructor are now compiler warnings, but many will pass under the radar. As a result, to write correct code one needs to be aware of multiple conventions and rules and patterns that are external to the language. When reviewing code one needs to do more than observe that it compiles and accomplishes its stated task, there are probably several organizational standards that need to be satisfied as well to be safe. This is precisely the kind of thing that Rust was supposed to solve. The goal was to have a language that didn't require an "Effective <xyz>" book to be memorized to do code review. A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error. So yes, I am particularly put off by this error.
- steveklabnik 12y ago
  Well, it's worth noting that this error happened because unsafe library code relied on a property that wasn't actually true. The language itself is working just fine, Rust cannot and will not prevent all bugs.
  - sirclueless 12y ago
    Scrapping the guarantee that destructors will be called is tantamount to scrapping RAII. thread::scoped made the assumption that destructors were called before a borrowed value was returned, and it led to a memory-safety error. If you don't use unsafe, you won't get memory-safety errors, but you can get plenty of other nastiness. The guarantee that destructors are called when values leave scope is fantastic. It means that one can leverage memory-safety into arbitrary-resource-acquisition-safety. It violates the principle of least astonishment in a brutal way if Rust's borrow checker can statically guarantee that a value no longer exists, but can't know whether or not its destructor has been called.
aturon 12y ago
I want to clarify a couple of points: Most important, this API problem does not break Rust's basic safety guarantee. That is, you must write unsafe code to violate memory safety. If Rust's basic memory safety guarantee were at risk, the core team would absolutely consider delaying the release, or doing whatever else was needed to address it! We must never allow memory safety violations in safe code, no matter how obscure the bug that leads to them. So this is a question of writing "safe" APIs that hide uses of `unsafe`. What precise assumptions are you allowed to make within such code? As it is, we are taking this API issue very seriously, and making sure that we explore all of the options available. We have already yanked the affected APIs, while we determine the best way forward. There are known ways to safely re-introduce the (relatively few) places in the standard library where unsafe code was using this pattern. Gankro talks about one in the post; you can see a proposal here (https://github.com/rust-lang/rfcs/pull/1084 https://github.com/rust-lang/rfcs/pull/1084) for the `scoped` API. The basic issue here is about a tension between a couple of different APIs, as Gankro explained in the post. In particular: - `Rc` has been part of Rust for a long, long time, and is a fundamental systems programming tool. - APIs like `scoped` and `drain_range` would like to use RAII for ergonomics and consistency, but this is either not possible (`scoped`) or subtle (`drain_range`) given `Rc`. In either case, these APIs use `unsafe` internally, so it's all about what that unsafe code can assume. Furthermore, there is not a strong consensus about how best to resolve these issues, even if we had all the time in the world. Currently there are at least three camps: - Stick with the status quo (perhaps marking `mem::forget` safe to reflect that reality). Safe code can assume no leakage, but unsafe code needs to take extra care (and sometimes avoid the RAII pattern). This is already the case for a large number of other properties, as Gankro points out. Leakage in safe code can never lead to memory unsafety; it is just a vanilla bug, and Rust doesn't prevent you from making bugs. - Introduce something like `Leak`, meaning that you have to explicitly ask for a given type to be guaranteed not to leak through things like `Rc` cycles. While that allows you to write APIs like `scoped` and `drain_range` easily using RAII, you need to be aware of the marker, and make sure to use it for such types. Worse, though, is the interaction with trait objects: depending on the design, you may have to write `Box<MyTrait + Leak>` to be able to store the trait object behind an `Rc`, and that `Leak` bound needs to be present for the entire chain of APIs leading up to that point. - Restrict `Rc` in some way, perhaps to `'static` data, thereby ruling out (a class of) leaks for all types. It's not completely clear how much fallout this would involve. The compiler currently relies on non-`'static` reference cells, and this change would likely force channels to use a `'static` bound as well, thereby defeating much of the purpose of the `scoped` API in the first place. Finally, it's worth noting that at least one version of the `Leak` proposal can be added backwards-compatibly, later on, so there is potentially plenty of time to explore that approach if we feel the complexity is worth it. I believe that Niko Matsakis (also on the core team) is planning to write a blog post explaining all of the above in much greater detail.

129 points by Manishearth 12y ago

A quick observation from an outsider: The article goes into a lot of depth, seems highly technical, well crafted and researched but the whole "pre-poop your pants" thing just ruins all that and makes it looks childish. I understand that not everything about development has to be serious, but I also think that aligning form and content is the surest way to make sure your message comes across.

golergka 12y ago
I guess it's a personal thing. I don't know much about Rust, but the title explained the severity of the problem pretty good right away.
- digitalzombie 12y ago
  I'm hoping this design pattern name catch on. PPYP design pattern. Sometime programming can be boring and then you come across PPYP and it makes it fun again.
  - Dewie2 12y ago
    First it was dry and boring, and then someone mentioned "poop" and then the same subject was hilarious! I don't get it...
    - WhatWouldLogDo 12y ago
      Analytic. Human beings still feel funny about Depth Psycho(techno)logy. Parallel processing pathos of human life terms with the logos apath programming words, represents multiple realities, versus the monotheistic standard robo swears robo cares.
pluma 12y ago
First Ruby, then JS, now Rust.
- th0br0 12y ago
  What is your point?
  - iagooar 12y ago
    I guess the parent means something that has to do with the community being a bit hipster / infantile? Only guessing though.
    - WhatWouldLogDo 12y ago
      I'd rather have rules to let you guess free (than have the memory leak of voting insight by iff it makes you/language feel good about problems.)
    - pluma 12y ago
      The entire infantile humour / brogrammer / hipster / drama phenomenon. It started in Ruby (_why, Zed Shaw, GitHub, etc) then moved into JS as Node and CoffeeScript became popular with Rails users, now it's in Rust (again via Ruby). I don't think it's always the same people or that it has anything to do with Ruby itself, it just seems like there's a certain culture that first grew in the Ruby / Rails community.
      - steveklabnik 12y ago
        FWIW I actually thought the title detracted from the otherwise excellent contents of the post.
      - dbaupp 12y ago
        This post is the first infantile thing in this vein I can remember seeing related to Rust, and, as far as I know, the author has no connection to the Ruby community you describe, they're just choosing poor titles of their own accord (I agree with Steve that the title detracts).
lifthrasiir 12y ago
I'll reuse my reply on /r/rust to defend the naming: > I like the naming and feeling of "Pre-Pooping Your Pants" pattern. We are doing the intentional leak only to be (hopefully) collected later, which may not happen but it still remains memory-safe even in that case. Still this is not very desirable and the name clearly indicates so.
- lighthawk 12y ago
  A term for code written in an undesirable way to tie you over in the short-term is a "hack". Not all hacks are bad. But, if I see "// this is a hack to..." or "# this is a hack to..." or similar in code, that is a signal to me that the developer (1) either didn't have time to perform the right thing but could have with more time (within their lifetime), or (2) could not feasibly perform the right thing under constraints given (within their lifetime). Over the past ~8 years, "hack"'s use has switched back to a much more positive sense- possibly because of HN, or possibly because people read Eric Raymond http://www.catb.org/esr/faqs/hacker-howto.html http://www.catb.org/esr/faqs/hacker-howto.html, but "hack" != "hacker". The negative sense is still used and understood by most developers, and it comes from two things: "HACK... HACK... HACK...", like the sound of an ax attempting to chop down a tree (brute force), and the other definition of a hack not used as much these days, which is "work for hire especially with mediocre professional standards". I think we should at least consider PPYP to be a subclass of "hack", or perhaps just use the word "hack" instead of PPYP.
  - jdub 12y ago
    kludge is a better term for an undesirable short-term fix. hack implies clever or elegant.
mrbig4545 12y ago
I couldn't agree more. the article is well written, but the title puts me off a lot. How am I going to show this to my boss, and have him take it seriously with such an infantile title?
- kyberias 12y ago
  Why would you want to show it to your boss?
  - mrbig4545 12y ago
    I tend to keep him updated with rust news, so that when the day comes, I've laid enough groundwork to be able to do something in it without having to explain what it is and why etc. And part of that is showing flaws, and how they're resolved/worked around and so on.
- nosefrog 12y ago
  You have the right to your own feelings, and I agree that the title may be off putting. The audience for the article is rust core contributors and rust fans, and they're a very informal bunch. If rusters are like similar communities I know, talking about poop makes a blog post more likely to spread. EDIT: I do not want to argue with Dewey2, so I'm putting this here: the women in the rust community that feel uncomfortable when they are addressed as "guys" have every right to feel that way, and I commend the rust community on making that stand.
  - Dewie2 12y ago
    A community that is informal enough to give analogies involving poop, but too uptight to stomach the gender-neutral use of the word "guys"[1]. Makes sense. [1] http://www.reddit.com/r/rust/comments/1nvsdh/a_note_on_conduct_please_read/ccmnjdj http://www.reddit.com/r/rust/comments/1nvsdh/a_note_on_condu...
    - Ygg2 12y ago
      Because toilet humor and gender discrimination are the same?
nosefrog 12y ago
I think Pre-Poop Your Pants is a really clever way to package the core idea. I agree that the article is well crafted, and I don't think it's childish to talk about poop in this context.
Manishearth 12y ago
It's not an "official" article, just a blog post by a contributor. Informal is okay IMO.

radulemnaru 12y ago

This title made my day!

bgdnpn 12y ago

I really like the title and the first sentence. "Much existential anguish and ennui was recently triggered.."

PieSquared 12y ago

I hope that the Rust devs end up doing the "right thing", which from this article seems like going back and fixing Rc, instead of just marking mem::forget as safe. They're still pre-1.0, and anything they do now they will be stuck with for a long, long time, especially if Rust succeeds as much as I hope it does. Delaying 1.0 by a few weeks may seem like a big deal, but ultimately it is a self-imposed deadline. It's great to have those, but following them dogmatically might not be the best strategy. In cases like this, I generally lean towards slowing down and doing things right: otherwise you will pay the price ten times over later. (That said, while I understand this issue, I don't know very much about the context in the Rust community, so I'm not actually sure that mem::forget should be unsafe. It was just the impression from the article and from previous Rust code I've read/written.)

Jweb_Guru 12y ago
There isn't an acceptable way to fix Rc. It wouldn't be hard to require it not to have cycles, but cyclic data is actually a major usecase for Rc. A cycle collector wouldn't fix any of the unsafety here unless it was run on literally every drop, which would be far too slow to be acceptable in a systems language. An opt-in trait to prevent leaking specific types would be practical, but the implementation details aren't settled and it can be done backwards-compatibly, so there's a strong feeling that it shouldn't block the release. Any other attempts to determine whether cyclic data structures always eventually drop their contents enter very-active-research territory and would be more like a ten year than a few week thing.
- deleted 12y ago
  [deleted]
- pcwalton 12y ago
  Precisely. Rc is not easy to "fix": the fact that it allows cycles is part of its design.
  - mcguire 12y ago
    There's a reason why finalizers aren't a widely used technique in Java and why Python's cycle detector simply stacks things with destructors in an out-of-the-way location.

zenojevski 12y ago

Why on Earth are we still using such unsafe languages? If the author had written this in a modern, safe language like Rust this would not have happened.

Ygg2 12y ago
Funny. But the problem here is complex and hard to solve. Cycles and memory leaks are nigh impossible to solve completely without sacrificing on that altar ability to create custom data structures or no-GC by default. Which was non-starter for Rust.
- krick 12y ago
  Well, I'm still not quite proficient with Rust, so maybe my feelings are misguided, but the whole situation seems really terrifying to me. Like, way too terrifying than that we could just make a point about "not doing like that again" and calmly move on to the 1.0. After all, why Rust? Because we're all tired of problems with C++ and stuff. And the point isn't really about making a language that would be safe "as often as possible", it would be quite terrible goal, actually. The point is, I believe, to make it nearly impossible to write unsafe code accidentally, without even noticing it. And again, maybe it's just me, but what I understood from that post is totally non-intuitive to me. I don't really understand what are the guidelines and how many more gotchas like that there could be. "Not to mess with unsafe code" is not really good guideline, because it turns out that it can be not quite obvious that the code is unsafe.
  - Ygg2 12y ago
    Well, the thing is, until Rust is formally tested, as in a subset of Rust is proven in CoQ to be safe, you don't have any guarantees. I assume much of it is safe, with few probably hidden caveats, hidden in more complex logic. In C++ writing code that causes this behavior is trivial, here you need to jump through a lot of hoops. So it offers you comparatively more safety, but this kind of bugs are potential gold mine for crackers. Anyway core developers are on the scene deciding how to tackle this issue.

garethrees 12y ago

There is a general problem with combining destructors and automatic collection of reference cycles, and its one that several languages have unexpectedly found themselves facing. For example, Java has the function System.runFinalizersOnExit, but if you look at the documentation you'll see that it now says, "Deprecated. This method is inherently unsafe. It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in erratic behavior or deadlock." http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#runFinalizersOnExit(boolean) http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.h... The problem is discussed in detail in Hans Boehm's 2002 technical report “Destructors, Finalizers, and Synchronization”. http://www.hpl.hp.com/techreports/2002/HPL-2002-335.pdf http://www.hpl.hp.com/techreports/2002/HPL-2002-335.pdf Boehm's key points are: (1) When a destructor on an object O runs normally, anything that O points to is still alive (because of the reference from O), but when a cycle is collected, not everyone can go first: that is, all but one destructors in the cycle have to run after one or more of their references are dead. It is hard to write destructor code that is safe in all cases. (2) Any destructor that needs to update a concurrently accessed data structure has to take a lock, but destructors on cycles run asynchronously with respect to the rest of the program and so an unlucky timing leads to deadlock. If this problem was better appreciated then language designers wouldn't go down the rabbit hole of trying to figure out how to make destructors work together with automatic collection of reference cycles, and instead try the alternative approach of providing mechanisms for the program to run destructors synchronously. In the Memory Pool System we use a message-passing interface: http://www.ravenbrook.com/project/mps/master/manual/html/topic/finalization.html http://www.ravenbrook.com/project/mps/master/manual/html/top...

charlieflowers 12y ago

Call Scott Meyers. So you cannot safely assume your destructors will be called, and you need to proactively take steps like this in case they aren't. Nothing in the natural use of the language cues you to do this, so you need a moral guide like "Effective C++" for Rust. This was the kind of accidental complexity that Rust had hoped to avoid. It seems like Rust has avoided a lot of it. But I guess it was entirely too optimistic to hope Rust could avoid all of it.

Gankro 12y ago
Note that this is only something you really need to think about when writing `unsafe` code, which basically unlocks a whole new set of rules that you need to worry about (zero-sized types in offsets, manual allocations, safety boundaries, GEP/aliasing rules, uninit memory, double destructors, etc). This is a drop in the bucket that anyone venturing to use `unsafe` will have to face in some kind of "guide to unsafety".
- mcguire 12y ago
  The problem is that all of this is safe code. The problem, to my mind, is that you can write mem::forget (as in the code here's safe_forget)* without ever using the unsafe keyword*. Edit: I was wrong. The problem isn't with forget. This is hairy.
  - steveklabnik 12y ago
    The 'mem::forget can be written in safe code' meme comes from https://github.com/rust-lang/rust/issues/24456 https://github.com/rust-lang/rust/issues/24456 Which, as you can see, uses Rc, which uses unsafe code in a way that leads to the soundness bug. So that's not strictly true, or rather, doesn't really change anything, as it relies on the same bug.
    - mcguire 12y ago
      nikomatsakis, in that thread: "1. There were other ways to forget even before Rc, at least in some cases. For example, if T:Send holds, you could send the value to a thread that runs an infinite loop, or which is deadlocked on a port. "2. It is true that one cannot assume that a destructor will run, and hence that forget is not itself unsafe (rather, it is unsafe to write a dtor that must run)...." And alexcrichton: "I commented on #24292, but the gist is that there are multiple ways to leak memory today (e.g. #14875 and #16135), so a targeted solution at Rc may not cover all use cases. Although as I mention in #24292 these other bugs can also be considered separate bugs on their own which need to be fixed regardless (but sometimes is quite difficult to do so)." I especially liked dgrunwald's comment: "A safe mem::forget has the advantage that it makes it easier to write the counterexample proving thread::JoinGuard unsafe. Safe mem::forget makes it more likely that people will know that destructors are not realiable, so they can avoid repeating this mistake." But then, "Put a big freaking spike in the middle of the steering wheel and get rid of the airbags and seat belts" has always appealed to me as an automotive safety approach.

Gankro 12y ago

Boring edition for people who have feels about maturity: http://cglab.ca/~abeinges/blah/everyone-peaches/ http://cglab.ca/~abeinges/blah/everyone-peaches/ ( people getting mad about PPYP is really a victory, though; the second last bullet was the most important: > It's vaguely incoherent and meaningless on its own: this is a property inherited from its connection to the venerable Resource Acquisition Is Initialization (RAII) pattern. I'm mad about RAII and not going to take it anymore! )

mcguire 12y ago

Largely unrelated question: In the code about a third of the way into the article, there is, impl<'a> Drop for Foo<'a> { fn drop(&mut self) { *self.0 += 1; } } What does the line "self.0 += 1;" do? Specifically, the ".0"? Foo is a basically a reference to an i32, so I would have thought that would be "self += 1;".

steveklabnik 12y ago
The .0 is a tuple access. Foo is a 'tuple struct' of length 1: http://doc.rust-lang.org/nightly/book/tuple-structs.html http://doc.rust-lang.org/nightly/book/tuple-structs.html (though the syntax is explained in the section on tuples: http://doc.rust-lang.org/nightly/book/primitive-types.html#tuple-indexing http://doc.rust-lang.org/nightly/book/primitive-types.html#t... )
- mcguire 12y ago
  Ah! Cool. Thanks!

sirclueless 12y ago

Putting stuff in a HashMap that you'll never ask for again is a kind of leak. Allocating stuff on the stack and then looping forever is a kind of leak. The case where you put it on the heap and forget to ever free it is a specific kind of leak that people seem particularly terrified of for whatever reason. I too am particularly terrified of this kind of leak. The reason is that it is the only kind of leak mentioned that is not clearly a programmer mistake. C++ has a whole class of "mistakes" that one cannot be aware of simply by reading the language specification. Common ones such as writing a constructor and destructor but not a copy-constructor are now compiler warnings, but many will pass under the radar. As a result, to write correct code one needs to be aware of multiple conventions and rules and patterns that are external to the language. When reviewing code one needs to do more than observe that it compiles and accomplishes its stated task, there are probably several organizational standards that need to be satisfied as well to be safe. This is precisely the kind of thing that Rust was supposed to solve. The goal was to have a language that didn't require an "Effective <xyz>" book to be memorized to do code review. A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error. So yes, I am particularly put off by this error.

steveklabnik 12y ago
Well, it's worth noting that this error happened because unsafe library code relied on a property that wasn't actually true. The language itself is working just fine, Rust cannot and will not prevent all bugs.
- sirclueless 12y ago
  Scrapping the guarantee that destructors will be called is tantamount to scrapping RAII. thread::scoped made the assumption that destructors were called before a borrowed value was returned, and it led to a memory-safety error. If you don't use unsafe, you won't get memory-safety errors, but you can get plenty of other nastiness. The guarantee that destructors are called when values leave scope is fantastic. It means that one can leverage memory-safety into arbitrary-resource-acquisition-safety. It violates the principle of least astonishment in a brutal way if Rust's borrow checker can statically guarantee that a value no longer exists, but can't know whether or not its destructor has been called.

aturon 12y ago

I want to clarify a couple of points: Most important, this API problem does not break Rust's basic safety guarantee. That is, you must write unsafe code to violate memory safety. If Rust's basic memory safety guarantee were at risk, the core team would absolutely consider delaying the release, or doing whatever else was needed to address it! We must never allow memory safety violations in safe code, no matter how obscure the bug that leads to them. So this is a question of writing "safe" APIs that hide uses of `unsafe`. What precise assumptions are you allowed to make within such code? As it is, we are taking this API issue very seriously, and making sure that we explore all of the options available. We have already yanked the affected APIs, while we determine the best way forward. There are known ways to safely re-introduce the (relatively few) places in the standard library where unsafe code was using this pattern. Gankro talks about one in the post; you can see a proposal here (https://github.com/rust-lang/rfcs/pull/1084 https://github.com/rust-lang/rfcs/pull/1084) for the `scoped` API. The basic issue here is about a tension between a couple of different APIs, as Gankro explained in the post. In particular: - `Rc` has been part of Rust for a long, long time, and is a fundamental systems programming tool. - APIs like `scoped` and `drain_range` would like to use RAII for ergonomics and consistency, but this is either not possible (`scoped`) or subtle (`drain_range`) given `Rc`. In either case, these APIs use `unsafe` internally, so it's all about what that unsafe code can assume. Furthermore, there is not a strong consensus about how best to resolve these issues, even if we had all the time in the world. Currently there are at least three camps: - Stick with the status quo (perhaps marking `mem::forget` safe to reflect that reality). Safe code can assume no leakage, but unsafe code needs to take extra care (and sometimes avoid the RAII pattern). This is already the case for a large number of other properties, as Gankro points out. Leakage in safe code can never lead to memory unsafety; it is just a vanilla bug, and Rust doesn't prevent you from making bugs. - Introduce something like `Leak`, meaning that you have to explicitly ask for a given type to be guaranteed not to leak through things like `Rc` cycles. While that allows you to write APIs like `scoped` and `drain_range` easily using RAII, you need to be aware of the marker, and make sure to use it for such types. Worse, though, is the interaction with trait objects: depending on the design, you may have to write `Box<MyTrait + Leak>` to be able to store the trait object behind an `Rc`, and that `Leak` bound needs to be present for the entire chain of APIs leading up to that point. - Restrict `Rc` in some way, perhaps to `'static` data, thereby ruling out (a class of) leaks for all types. It's not completely clear how much fallout this would involve. The compiler currently relies on non-`'static` reference cells, and this change would likely force channels to use a `'static` bound as well, thereby defeating much of the purpose of the `scoped` API in the first place. Finally, it's worth noting that at least one version of the `Leak` proposal can be added backwards-compatibly, later on, so there is potentially plenty of time to explore that approach if we feel the complexity is worth it. I believe that Niko Matsakis (also on the core team) is planning to write a blog post explaining all of the above in much greater detail.