5 ms·

Without the exclusivity, it would be easy for a reference to be invalidated; the classic example is let nut v: Vec<T> = ...; let r: &T = &v[0]; v.pus

by dbaupp 7y ago

Without the exclusivity, it would be easy for a reference to be invalidated; the classic example is let nut v: Vec<T> = ...; let r: &T = &v[0]; v.push(...); // A foo(r); // B The reference passed at B might be invalidated by A, if the vector is reallocated. It’s the shared vs exclusive borrowing that defends against this, by flagging the above program as an error.

LessDmesg 7y ago
And this is another point in favor of compacting GCs: non-GC languages are moving things around almost as much (i.e. growable arrays are ubiquitous, plus hashmaps may be based on them too etc). Without a GC the simple task of moving references becomes dangerous and requires the pain demonstrated to us by Rust (or leads to pervasively crashy software culture demonstrated to us by C++). With GC, this type of thing is not something the developer needs to care about.
- jfkebwjsbx 7y ago
  > pervasively crashy software culture demonstrated to us by C++ Please don’t spread FUD. The majority of commercial C++ software out there is rock solid. Vulnerabilities are the actual problem in today's Internet connected world and why Rust has grown so fast. As for GCs: they are not good enough for some fields, so they cannot be used. Even state of the art ones like Go's.
  - zelphirkalt 7y ago
    Why the "commercial"? Are other software types not worthy to be considered in this? Are open source non-commercial projects inherently worse code and more crashy? Should we not look at all kinds of software, when talking about programming languages in general? It has been my experience, that even the well known softwares written in C (not sure about C++) are indeed kind of crashy. Example: Open Broadcast Studio, used by many people to stream on various platforms. It is possible there to go into the settings and change them so that OBS crashes and cannot be restarted again, unless you guess, that the configuration is faulty and OBS is unable to deal with a faulty config. You can only get it to run again by removing the config or reinstalling. Great quality. VLC is another example of such crashy software. Not only did it crash many times in the past, but also cannot properly deal with unplayable files in a looping playlist and will simply bring all your cores to 100% usage. So far some anecdotes from the well known C software world. I am not sure, whether the people in the respective community consider these softwares to be good quality.
    - erik_seaberg 7y ago
      Apart from maybe seL4 (which I only know by reputation, and was translated from a formally verified spec) I can't name any software that I would call reliable. If we ever want computers to actually work, the whole stack needs to be flensed down to the metal and replaced, and I hope Rust is the language to do it with.
  - littlestymaar 7y ago
    The op is talking about compacting GCs and Go's isn't a compacting one. Go's GC is a variation of a regular mark and sweep GC: the GC starts exploring the heap to find all reachable memory, and then clears the unreachable one, meanwhile compacting GCs move the reachable memory from one place (the “from space”) to another (the “to space”), which become the new heap. Theses two garbage collectors famillies are completely different. BTW, calling go's GC “state of the art” is debatable at best. See : https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e?gi=15496e3661e1 https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...
    - jfkebwjsbx 7y ago
      And I was talking about GCs overall. The point was that Go's GC is pretty much the best GC out there used in major projects, designed to try to compete with systems programming languages and come as closely as possible to it. A lot of money has been put to make it as fast and deterministic as possible. It does not mean it is the latest in research, of course.
      - littlestymaar 7y ago
        This is a real misunderstanding. Go's GC is not “pretty much the best GC out there” by most GC standards (it's already quite good and keeps getting better, though). And it hasn't been “designed to try to compete with systems programming languages”. Go as a language was designed like that, and special attention was made to allow as much value types as possible, to allocate as much things as you can on the stack, thus reducing GC pressure. For the first five years of Go or something, Go's GC was actually pretty bad (it was probably the most basic GC you could find in any somewhat popular language), but it wasn't too much of a deal because you can avoid it most of the time when it goes in your way (much more easily than in Java for instance). After some time, they decided to enhance it, but they were on a budget (no lots of money spent on it” actually), so because in go you can avoid allocating memory on the heap, they decided to focus on GC latency instead of throughput (if the GC's throughput isn't good enough for you, you better reduce your allocations). Overall go is a pretty fast language, and it's an engineering success, but it's in spite of its GC and thanks to other parts of the language's design, not because Go's GC is exceptionally good (it's not, and if you read my link you'd understand how).
  - LessDmesg 7y ago
    Oh please. I've used Windows since 2000, and have seen lots of commercial C++ software (games, editors, Windows itself) give me that characteristic sound and dialog box. I stopped using Plasma desktop and KDE because of the crashes. "Error. Memory could not be written", segmentation fault, DLL error - I naively thought these were an inevitable consequence of using a computer, until I realized that the problem was that all this software was written in C++.
    - jfkebwjsbx 7y ago
      1. KDE is not commercial AFAIK. 2. Windows was (is?) not coded in modern C++ but a bastard C or very old C++ from what I understand. Bugs were also mainly by third-party drivers and software. The industry for home user software had very bad quality, that’s is true, but that had nothing to do with C++. C, Fortran, Delphi and others were used too. 3. Videogames are in an industry where deadlines are more important than bugs and companies really don’t care about fixing stuff before release.
  - pjmlp 7y ago
    Microsoft, Google and Apple happen to disagree with you. https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/ https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-s... https://security.googleblog.com/2019/08/adopting-arm-memory-tagging-extension.html https://security.googleblog.com/2019/08/adopting-arm-memory-... https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Pr... https://developer.apple.com/swift/#safety https://developer.apple.com/swift/#safety Modern real time GC are good enough for military use, https://www.ptc.com/en/products/developer-tools/perc https://www.ptc.com/en/products/developer-tools/perc
    - jfkebwjsbx 7y ago
      Do not mix up vulnerabilities (security) with normal usage (generally buggy SW), which is what I talked about. I explicitly called out vulnerabilities as the actual problem that calls for safe languages. For instance, you can have a library that works perfectly for any non-malicious/valid PNG image, yet have vulnerabilities for invalid, crafted files. In fact, this is almost always the case for any product. Only really bad quality software has issues with the happy expected path. But many more will have vulnerabilities in the unexpected paths.
- jimktrains2 7y ago
  GCs don't magically allow you to not move things around. You still need to grow arrays and hashmaps. GCs also don't magically provide data race guarantes, which is where a good portion of the complexity with the borrow checker comes to play.
  - LessDmesg 7y ago
    My point was the opposite: if data is to be copied about anyway, why not use GC which makes this copying safe (i.e. it handles reference updates) and extracts additional profits (compactization) while it's at it? As opposed to manual languages where objects are expected to be pinned (and memory-fragmenting) but often aren't, leading to headaches or runtime errors.
    - winstonewert 7y ago
      Do any languages actually do this? Do any languages somehow use their GC to update references when a growable array is reallocated? I'm not aware of any language that does. Furthermore, its hard to see how it could. In the case of a simple copying/compacting collector the whole world is stopped and the pointer graph traced. But you obviously can't do that every time a growable array needs to be reallocated. I understand that there are more complex solutions that avoid that pause, but still not any I'm aware of that would be exploitable for reallocating an array.
- steveklabnik 7y ago
  This still happens in GC'd languages, for example, iterating over a collection while modifying it. In Java you may get a ConcurrentModifiedException. In Ruby and JS you get strange behavior. In Rust you get a compile error.
fauigerzigerk 7y ago
So the reason why this causes a problem is that mutating an outer value could invalidate the lifetime of an inner value. Could this not be covered by the lifetime guarantee without imposing read/write exclusivity on all values even where lifetimes are unaffected?
- steveklabnik 7y ago
  It is not clear to me what your question is asking, to be honest.
  - fauigerzigerk 7y ago
    The code posted by dbaupp shows how violating Rust's immutability/exclusivity guarantee can also violate the lifetime guarantee. The implication being that the lifetime guarantee depends on the immutability/exclusivity guarantee (as supermatt claimed earlier). It's a good example, but I'm still not enirely convinced that the implication is necessarily correct. I wonder if a sufficiently smart compiler could not detect this particular special case and relax immutability/exclusivity in other cases where no references to the internal structure of a value exist, or if it could even keep whatever r is referencing alive to uphold the lifetime guarantee. I realise the latter could have some undesirable side-effects wrt memory usage and it raises ownership questions if there are multiple such references. It's essentially what persistent data structures in functional languages do and it may not be a good fit for Rust. The reason why I'm even interested in this is that Rust can sometimes feel restrictive in situations where having both mutable and immutable references or more than one mutable reference to the same thing is reasonably safe, such as in the local scope. Obviously people are working around mutability restrictions by using indexes/handles instead of references, but that doesn't seem any safer.
    - steveklabnik 7y ago
      There's a few different things at play here: 1. In order a sufficiently smart compiler to understand if this is okay, it would have to know how the Vec<T> type actually works. But Vec<T> is a library type, and uses unsafe code, which is kind of definition-ally code that the compiler cannot understand. So, even for this specific case, it is too hard to do so today. A human cannot even tell if this is okay or not, right now, because you'd need to see the code that's in the `...`. 2. Even if it could, it's unclear if it's a good idea. The more complex analysis a compiler can do, the harder it is to explain what it's doing to end-users. Imagine that, for example, we wrote this code, with the `...` being a case where it's safe, but then we modified it to a case where `...` was not safe anymore. What would that diagnostic look like? How could it be communicated to people in a useful way? 3. There's a tension here. If you make it work for very specific reasons, any small changes would likely cause it to break, whereas it breaking up front may lead you toward a design that is overall more robust. > It's essentially what persistent data structures in functional languages do and it may not be a good fit for Rust. We have those! https://crates.io/crates/im https://crates.io/crates/im > Rust can sometimes feel restrictive in situations where having both mutable and immutable references or more than one mutable reference to the same thing is reasonably safe, such as in the local scope. It depends! You're not wrong, but also see my comment over here: https://news.ycombinator.com/item?id=22466587 https://news.ycombinator.com/item?id=22466587 even local scope may not be enough.

by dbaupp 7y ago

LessDmesg 7y ago

And this is another point in favor of compacting GCs: non-GC languages are moving things around almost as much (i.e. growable arrays are ubiquitous, plus hashmaps may be based on them too etc). Without a GC the simple task of moving references becomes dangerous and requires the pain demonstrated to us by Rust (or leads to pervasively crashy software culture demonstrated to us by C++). With GC, this type of thing is not something the developer needs to care about.

jfkebwjsbx 7y ago
> pervasively crashy software culture demonstrated to us by C++ Please don’t spread FUD. The majority of commercial C++ software out there is rock solid. Vulnerabilities are the actual problem in today's Internet connected world and why Rust has grown so fast. As for GCs: they are not good enough for some fields, so they cannot be used. Even state of the art ones like Go's.
- zelphirkalt 7y ago
  Why the "commercial"? Are other software types not worthy to be considered in this? Are open source non-commercial projects inherently worse code and more crashy? Should we not look at all kinds of software, when talking about programming languages in general? It has been my experience, that even the well known softwares written in C (not sure about C++) are indeed kind of crashy. Example: Open Broadcast Studio, used by many people to stream on various platforms. It is possible there to go into the settings and change them so that OBS crashes and cannot be restarted again, unless you guess, that the configuration is faulty and OBS is unable to deal with a faulty config. You can only get it to run again by removing the config or reinstalling. Great quality. VLC is another example of such crashy software. Not only did it crash many times in the past, but also cannot properly deal with unplayable files in a looping playlist and will simply bring all your cores to 100% usage. So far some anecdotes from the well known C software world. I am not sure, whether the people in the respective community consider these softwares to be good quality.
  - erik_seaberg 7y ago
    Apart from maybe seL4 (which I only know by reputation, and was translated from a formally verified spec) I can't name any software that I would call reliable. If we ever want computers to actually work, the whole stack needs to be flensed down to the metal and replaced, and I hope Rust is the language to do it with.
- littlestymaar 7y ago
  The op is talking about compacting GCs and Go's isn't a compacting one. Go's GC is a variation of a regular mark and sweep GC: the GC starts exploring the heap to find all reachable memory, and then clears the unreachable one, meanwhile compacting GCs move the reachable memory from one place (the “from space”) to another (the “to space”), which become the new heap. Theses two garbage collectors famillies are completely different. BTW, calling go's GC “state of the art” is debatable at best. See : https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e?gi=15496e3661e1 https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...
  - jfkebwjsbx 7y ago
    And I was talking about GCs overall. The point was that Go's GC is pretty much the best GC out there used in major projects, designed to try to compete with systems programming languages and come as closely as possible to it. A lot of money has been put to make it as fast and deterministic as possible. It does not mean it is the latest in research, of course.
    - littlestymaar 7y ago
      This is a real misunderstanding. Go's GC is not “pretty much the best GC out there” by most GC standards (it's already quite good and keeps getting better, though). And it hasn't been “designed to try to compete with systems programming languages”. Go as a language was designed like that, and special attention was made to allow as much value types as possible, to allocate as much things as you can on the stack, thus reducing GC pressure. For the first five years of Go or something, Go's GC was actually pretty bad (it was probably the most basic GC you could find in any somewhat popular language), but it wasn't too much of a deal because you can avoid it most of the time when it goes in your way (much more easily than in Java for instance). After some time, they decided to enhance it, but they were on a budget (no lots of money spent on it” actually), so because in go you can avoid allocating memory on the heap, they decided to focus on GC latency instead of throughput (if the GC's throughput isn't good enough for you, you better reduce your allocations). Overall go is a pretty fast language, and it's an engineering success, but it's in spite of its GC and thanks to other parts of the language's design, not because Go's GC is exceptionally good (it's not, and if you read my link you'd understand how).
- LessDmesg 7y ago
  Oh please. I've used Windows since 2000, and have seen lots of commercial C++ software (games, editors, Windows itself) give me that characteristic sound and dialog box. I stopped using Plasma desktop and KDE because of the crashes. "Error. Memory could not be written", segmentation fault, DLL error - I naively thought these were an inevitable consequence of using a computer, until I realized that the problem was that all this software was written in C++.
  - jfkebwjsbx 7y ago
    1. KDE is not commercial AFAIK. 2. Windows was (is?) not coded in modern C++ but a bastard C or very old C++ from what I understand. Bugs were also mainly by third-party drivers and software. The industry for home user software had very bad quality, that’s is true, but that had nothing to do with C++. C, Fortran, Delphi and others were used too. 3. Videogames are in an industry where deadlines are more important than bugs and companies really don’t care about fixing stuff before release.
- pjmlp 7y ago
  Microsoft, Google and Apple happen to disagree with you. https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/ https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-s... https://security.googleblog.com/2019/08/adopting-arm-memory-tagging-extension.html https://security.googleblog.com/2019/08/adopting-arm-memory-... https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Pr... https://developer.apple.com/swift/#safety https://developer.apple.com/swift/#safety Modern real time GC are good enough for military use, https://www.ptc.com/en/products/developer-tools/perc https://www.ptc.com/en/products/developer-tools/perc
  - jfkebwjsbx 7y ago
    Do not mix up vulnerabilities (security) with normal usage (generally buggy SW), which is what I talked about. I explicitly called out vulnerabilities as the actual problem that calls for safe languages. For instance, you can have a library that works perfectly for any non-malicious/valid PNG image, yet have vulnerabilities for invalid, crafted files. In fact, this is almost always the case for any product. Only really bad quality software has issues with the happy expected path. But many more will have vulnerabilities in the unexpected paths.
jimktrains2 7y ago
GCs don't magically allow you to not move things around. You still need to grow arrays and hashmaps. GCs also don't magically provide data race guarantes, which is where a good portion of the complexity with the borrow checker comes to play.
- LessDmesg 7y ago
  My point was the opposite: if data is to be copied about anyway, why not use GC which makes this copying safe (i.e. it handles reference updates) and extracts additional profits (compactization) while it's at it? As opposed to manual languages where objects are expected to be pinned (and memory-fragmenting) but often aren't, leading to headaches or runtime errors.
  - winstonewert 7y ago
    Do any languages actually do this? Do any languages somehow use their GC to update references when a growable array is reallocated? I'm not aware of any language that does. Furthermore, its hard to see how it could. In the case of a simple copying/compacting collector the whole world is stopped and the pointer graph traced. But you obviously can't do that every time a growable array needs to be reallocated. I understand that there are more complex solutions that avoid that pause, but still not any I'm aware of that would be exploitable for reallocating an array.
steveklabnik 7y ago
This still happens in GC'd languages, for example, iterating over a collection while modifying it. In Java you may get a ConcurrentModifiedException. In Ruby and JS you get strange behavior. In Rust you get a compile error.

fauigerzigerk 7y ago

So the reason why this causes a problem is that mutating an outer value could invalidate the lifetime of an inner value. Could this not be covered by the lifetime guarantee without imposing read/write exclusivity on all values even where lifetimes are unaffected?

steveklabnik 7y ago
It is not clear to me what your question is asking, to be honest.
- fauigerzigerk 7y ago
  The code posted by dbaupp shows how violating Rust's immutability/exclusivity guarantee can also violate the lifetime guarantee. The implication being that the lifetime guarantee depends on the immutability/exclusivity guarantee (as supermatt claimed earlier). It's a good example, but I'm still not enirely convinced that the implication is necessarily correct. I wonder if a sufficiently smart compiler could not detect this particular special case and relax immutability/exclusivity in other cases where no references to the internal structure of a value exist, or if it could even keep whatever r is referencing alive to uphold the lifetime guarantee. I realise the latter could have some undesirable side-effects wrt memory usage and it raises ownership questions if there are multiple such references. It's essentially what persistent data structures in functional languages do and it may not be a good fit for Rust. The reason why I'm even interested in this is that Rust can sometimes feel restrictive in situations where having both mutable and immutable references or more than one mutable reference to the same thing is reasonably safe, such as in the local scope. Obviously people are working around mutability restrictions by using indexes/handles instead of references, but that doesn't seem any safer.
  - steveklabnik 7y ago
    There's a few different things at play here: 1. In order a sufficiently smart compiler to understand if this is okay, it would have to know how the Vec<T> type actually works. But Vec<T> is a library type, and uses unsafe code, which is kind of definition-ally code that the compiler cannot understand. So, even for this specific case, it is too hard to do so today. A human cannot even tell if this is okay or not, right now, because you'd need to see the code that's in the `...`. 2. Even if it could, it's unclear if it's a good idea. The more complex analysis a compiler can do, the harder it is to explain what it's doing to end-users. Imagine that, for example, we wrote this code, with the `...` being a case where it's safe, but then we modified it to a case where `...` was not safe anymore. What would that diagnostic look like? How could it be communicated to people in a useful way? 3. There's a tension here. If you make it work for very specific reasons, any small changes would likely cause it to break, whereas it breaking up front may lead you toward a design that is overall more robust. > It's essentially what persistent data structures in functional languages do and it may not be a good fit for Rust. We have those! https://crates.io/crates/im https://crates.io/crates/im > Rust can sometimes feel restrictive in situations where having both mutable and immutable references or more than one mutable reference to the same thing is reasonably safe, such as in the local scope. It depends! You're not wrong, but also see my comment over here: https://news.ycombinator.com/item?id=22466587 https://news.ycombinator.com/item?id=22466587 even local scope may not be enough.