7 ms·

Rust Ownership Explained with Python

149 points by adamnemecek 8y ago

krispbyte 8y ago
I remember seeing a visualization of the variables and values in a python program as it ran and how they get linked to one another and that really helped me understand python variables. Does such a visualization exists for Rust? I'm gonna try to hunt the link for the Python one.
- adamnemecek 8y ago
  Python tutor http://www.pythontutor.com/visualize.html#mode=edit http://www.pythontutor.com/visualize.html#mode=edit?
  - krispbyte 8y ago
    Yes! thank you! This helped a lot for understanding the behavior of list vs other values, for example: http://www.pythontutor.com/visualize.html#code=a%20%3D%204%0Ab%20%3D%20a%0Aa%20%3D%2010%0Aprint%28a,%20b%29%0A%0At%20%3D%20%5B1,2,3%5D%0As%20%3D%20t%0At.append%284%29%0Aprint%28t,%20s%29&cumulative=false&curInstr=0&heapPrimitives=nevernest&mode=display&origin=opt-frontend.js&py=3&rawInputLstJSON=%5B%5D&textReferences=false http://www.pythontutor.com/visualize.html#code=a%20%3D%204%0... I think it would do wonders for ownership.
- quodlibetor 8y ago
  You're probably thinking of http://pythontutor.com/ http://pythontutor.com/ for python. It's one of my absolute favorite instructional resources for any language. Birdseye[1] looks like it is trying to be a similar kind of thing for debugging Python, which is amazing. I don't know of anything for anything similar for Rust or anything with similar characteristics (C/++). [1] https://github.com/alexmojaki/birdseye https://github.com/alexmojaki/birdseye
staticassertion 8y ago
It's definitely harder in the general case to run into these bugs, but the real trick, in my opinion, is building APIs where it's impossible. The 'escape hatch' here is the .by_ref() but you could, if it's critical, write wrappers that don't provide those escape hatches. If it were critical that the reuse never happened you can make it impossible, which is really cool. But, by default, for most APIs, it's just generally more difficult to mess up. There are a bunch of articles on session types out there, where you represent each state in a state machine as a separate type, with transition methods that consume the previous state. If the correctness touted in this article sounds appealing, I highly recommend reading about session types.
bayesian_horse 8y ago
As a Rust beginner I am not quite sure why .min() is supposed to take ownership and change the sequence. I would assume it does nothing with the sequence except read it.
- kilburn 8y ago
  Sequences can be generated at runtime, and it may even be impossible to "regenerate" them (think for instance reading bytes from the network). Therefore, just reading that sequence changes (consumes) it.
  - MaulingMonkey 8y ago
    And of course for things that can be iterated over multiple times, it's likely possible that you can (re)generate the iterator. E.g. this compiles fine: fn main() { let squares : Vec<u32> = (0..10).map(|x| x * x).collect(); println!("{:?}", squares.iter().min()); // Some(0) println!("{:?}", squares.iter().max()); // Some(81) } Edit: Or a smaller change to the original, you can add a single .clone(): fn main() { let squares = (0..10).map(|x| x * x); println!("{:?}", squares.clone().min()); println!("{:?}", squares.max()); } This clones the std::iter::Map<std::ops::Range<i32>>. Unlike the vec version, this will re-evaluate the |x|x*x sequence when .max() is called. There's no technical reason Map<Range<i32>> couldn't be made to implement the Copy trait, which would make the extra .clone() here unnecessary - but Rust's standard library has chosen to force you to make an explicit choice instead.
    - lostmsu 8y ago
      Curiously, C# provides the behavior one would expect. It does so by distinguishing IEnumerable<T> [1], and IEnumerator<T> [2]. The first would be the result of the map call (Select in C#'s LINQ), representing a sequence, that could be generated, and the later is the one, that actually generates it one by one, and has to be consumed. I wonder why Rust did not take C#'s approach. [1] https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.ienumerable-1 https://docs.microsoft.com/en-us/dotnet/api/system.collectio... [2] https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.ienumerator-1 https://docs.microsoft.com/en-us/dotnet/api/system.collectio...
thomasjames 8y ago
Isn't the property of the generator expression not that it is an iterator (like a range or list also are) but specifically that it is a generator and thus inherently has state that is permanently/mutably exhaustible across different scopes? I like the example, and this is the first time I've totally wrapped my head around Rust ownership, so thanks! I just think precise Python terminology might keep people from getting confused about different types of Python iterator objects. Iterators are anything that has a __next__() method: https://www.python.org/dev/peps/pep-0234/ https://www.python.org/dev/peps/pep-0234/
- deathanatos 8y ago
  > Isn't the property of the generator expression not that it is an iterator (like a range or list also are) but specifically that it is a generator and thus inherently has state that is permanently/mutably exhaustible across different scopes? > just think precise Python terminology might keep people from getting confused about different types of Python iterator objects. Iterators are anything that has a __next__() method An iterator will function exactly as poorly as a generator does in his first example: In [5]: l = [1, 2, 3] In [6]: iter_ = iter(l) In [7]: min(iter_) Out[7]: 1 In [8]: max(iter_) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-8-0dab0672ef83> in <module>() ----> 1 max(iter_) ValueError: max() arg is an empty sequence (I chose iter(a list) here as it is not a generator.) And for the same reasons, really. An Iterator in Python (something w/ a __next__ method) also keeps mutable state, and the first call to min() exhausts it (by mutating it). (In Rust, it would likely "consume" the iterator, that is, it would move it, and the article demonstrates an example of this.) The mutable state being kept and used for iteration is the material part here; the generator is just an easy way of obtaining an iterator. Hopefully I've got this right, but in Python, all generators are iterators¹ (they have a __next__); generators also additionally have send(), throw(), and close() methods, making them substantially more powerful. They make it really easy to make just a basic iterator, so they're often used for that. ¹The glossary actually calls the functions (those with "yield") themselves generators, and the objects returned by them "generator iterators".
- rahimnathwani 8y ago
merlincorey 8y ago
The Python problem can be solved with `itertools.tee`[0] from itertools import tee squares_min, squares_max = tee(x * x for x in range(10)) print(min(squares_min)) print(max(squares_max)) [0] https://docs.python.org/3/library/itertools.html#itertools.tee https://docs.python.org/3/library/itertools.html#itertools.t...
- c8g 8y ago
  squares = [x * x for x in range(10)] print(min(squares)) print(max(squares)) this one works and looks good.
  - skbly7 8y ago
    It will have memory impact though with larger range. A better approach might be: squares = lambda: (x * x for x in range(10)) print(min(squares)) print(max(squares()))
    - rahimnathwani 8y ago
      Yes, your solution is in the original post, under the heading 'Python solutions': def squares(): return (x * x for x in range(10))
  - adrianN 8y ago
    It also uses O(n) memory.
    - sitkack 8y ago
      We need a language that can schedule and fuse those folds over an infinite sequence concurrently. Joining a set of futures over an infinite sequence fuses folds
      - adrianN 8y ago
        I'm not sure adding more magic to languages is necessary when the alternative of using explicit generators is not too difficult.
mlthoughts2018 8y ago
I can’t comment on the general case, but I disagree very strongly with the author’s perspective about generator expressions here. First, a generator is a precise type of data structure that can be exhausted. If you don’t want that, don’t use that structure. It’s foolish to say that just because a data structure superficially satisfies some API (like being passable to min), that it “ought to have” some behavior that the author wants. That’s backwards. If you want certain behavior, choose a data structure that supports it. It would be no different from saying if I have a heap and I call sort() it is faster than if I have the same data in a list and call sort(). They both support sort() so why aren’t they equally fast? It’s obviously the wrong way to think about it. Instead, use the appropriate data structure. And if you’re really worried about the operation being safe, use the provided default argument, or the pattern for versions before 3.4, as in [0], or write a wrapper function that catches whatever exceptions and handles them. This isn’t even the biggest issue though. Generators generally are meant to function as coroutines and for complex generators, you can use send() to set the “returned value” of a yield statement inside the generator for when it resumes, and so a single generator could go through stages of being exhausted and empty, then having a value set again so that on resumption it’s no longer empty, and could build back up a whole iterable of data that way. So you absolutely do want min() or max() to throw errors when trying to consume from an empty generator, and you absolutely do want a generator’s state to become modified by the way values are consumed or added, because it meaningfully represents the state of the generator. Suppose for a more complex generator, someone uses send() to feed it a value between your call to min() and your call to max(). Generator expressions that superficially look like list comprehensions are just one use case of the general data structure and in fact one of the “generators 101” pieces of advice you always see is that if you need to iterate the values multiple times, then populate a persistent data structure out of the generator, like by wrapping in list(). I bring this up because it frustrates me to see people acting like a type safety or borrow checking idea is somehow aadding something new or solving some type of problem that the dynamic typing approach doesn’t or can’t solve, but this is emphatically incorrect and a severely myopic way to look at it. The dynamic typing approach solves all the same problems, it just facilitates the solution differently: use the default argument or other patterns for safety on functions that can fail on empty containers, and use custom wrappers with exception handling when you need to. Choose a persistent data structure if you need persistence. Again, not saying this has anything to do with people’s big picture reasons for liking static types or borrow checking... just this particular example strikes a chord to me highlighting the super annoying myopia about it. [0]: < https://stackoverflow.com/questions/36157995/a-safe-max-function-for-empty-lists https://stackoverflow.com/questions/36157995/a-safe-max-func... >
shmageggy 8y ago
Not that it really matters, but since he lamented that the second Python solution was "longer and less elegant", I'll point out that that algorithm can be written in a functional style that is both short and (IMO) somewhat elegant. However with reduce being removed as a built-in since Python 3, this is arguably not Pythonic. from functools import reduce squares = (x * x for x in range(10)) min_and_max = lambda current, x: [min(current[0], x), max(current[1], x)] minimum, maximum = reduce(min_and_max, squares, [next(squares)]*2) print(minimum) print(maximum)
- c8g 8y ago
  squares = [x * x for x in range(10)] print(min(squares)) print(max(squares)) this one works and looks good.
  - wyldfire 8y ago
    range(10) doesn't properly demonstrate the material differences between generators and lists. Like posted elsewhere, tee is an appropriate solution that preserves the generator's benefits over a list. It's too bad that the example is so synthetic.
    - shmageggy 8y ago
      One downside of both solutions is that they both traverse the list twice, which could be expensive when we're talking about real data and not toy problems.
vbsteven 8y ago
For me Rust ownership clicked when I was reading a C++ book. More specifically the sections about references, move and copy.
bogomipz 8y ago
I really enjoyed reading this, I thought the author made these very accessible. Perhaps the author might consider writing posts on both traits and lifetimes? Understanding ownership, traits and lifetimes seem to be the keys to understanding and using Rust effectively. Unfortunately these are also the things I've had the hardest time getting my head around. If anyone could suggest similarly accessible articles on traits and lifetimes I would greatly appreciate it. Cheers.

149 points by adamnemecek 8y ago

I remember seeing a visualization of the variables and values in a python program as it ran and how they get linked to one another and that really helped me understand python variables. Does such a visualization exists for Rust? I'm gonna try to hunt the link for the Python one.

adamnemecek 8y ago
Python tutor http://www.pythontutor.com/visualize.html#mode=edit http://www.pythontutor.com/visualize.html#mode=edit?
- krispbyte 8y ago
  Yes! thank you! This helped a lot for understanding the behavior of list vs other values, for example: http://www.pythontutor.com/visualize.html#code=a%20%3D%204%0Ab%20%3D%20a%0Aa%20%3D%2010%0Aprint%28a,%20b%29%0A%0At%20%3D%20%5B1,2,3%5D%0As%20%3D%20t%0At.append%284%29%0Aprint%28t,%20s%29&cumulative=false&curInstr=0&heapPrimitives=nevernest&mode=display&origin=opt-frontend.js&py=3&rawInputLstJSON=%5B%5D&textReferences=false http://www.pythontutor.com/visualize.html#code=a%20%3D%204%0... I think it would do wonders for ownership.
quodlibetor 8y ago
You're probably thinking of http://pythontutor.com/ http://pythontutor.com/ for python. It's one of my absolute favorite instructional resources for any language. Birdseye[1] looks like it is trying to be a similar kind of thing for debugging Python, which is amazing. I don't know of anything for anything similar for Rust or anything with similar characteristics (C/++). [1] https://github.com/alexmojaki/birdseye https://github.com/alexmojaki/birdseye

staticassertion 8y ago

It's definitely harder in the general case to run into these bugs, but the real trick, in my opinion, is building APIs where it's impossible. The 'escape hatch' here is the .by_ref() but you could, if it's critical, write wrappers that don't provide those escape hatches. If it were critical that the reuse never happened you can make it impossible, which is really cool. But, by default, for most APIs, it's just generally more difficult to mess up. There are a bunch of articles on session types out there, where you represent each state in a state machine as a separate type, with transition methods that consume the previous state. If the correctness touted in this article sounds appealing, I highly recommend reading about session types.

bayesian_horse 8y ago

As a Rust beginner I am not quite sure why .min() is supposed to take ownership and change the sequence. I would assume it does nothing with the sequence except read it.

kilburn 8y ago
Sequences can be generated at runtime, and it may even be impossible to "regenerate" them (think for instance reading bytes from the network). Therefore, just reading that sequence changes (consumes) it.
- MaulingMonkey 8y ago
  And of course for things that can be iterated over multiple times, it's likely possible that you can (re)generate the iterator. E.g. this compiles fine: fn main() { let squares : Vec<u32> = (0..10).map(|x| x * x).collect(); println!("{:?}", squares.iter().min()); // Some(0) println!("{:?}", squares.iter().max()); // Some(81) } Edit: Or a smaller change to the original, you can add a single .clone(): fn main() { let squares = (0..10).map(|x| x * x); println!("{:?}", squares.clone().min()); println!("{:?}", squares.max()); } This clones the std::iter::Map<std::ops::Range<i32>>. Unlike the vec version, this will re-evaluate the |x|x*x sequence when .max() is called. There's no technical reason Map<Range<i32>> couldn't be made to implement the Copy trait, which would make the extra .clone() here unnecessary - but Rust's standard library has chosen to force you to make an explicit choice instead.
  - lostmsu 8y ago
    Curiously, C# provides the behavior one would expect. It does so by distinguishing IEnumerable<T> [1], and IEnumerator<T> [2]. The first would be the result of the map call (Select in C#'s LINQ), representing a sequence, that could be generated, and the later is the one, that actually generates it one by one, and has to be consumed. I wonder why Rust did not take C#'s approach. [1] https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.ienumerable-1 https://docs.microsoft.com/en-us/dotnet/api/system.collectio... [2] https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.ienumerator-1 https://docs.microsoft.com/en-us/dotnet/api/system.collectio...

thomasjames 8y ago

Isn't the property of the generator expression not that it is an iterator (like a range or list also are) but specifically that it is a generator and thus inherently has state that is permanently/mutably exhaustible across different scopes? I like the example, and this is the first time I've totally wrapped my head around Rust ownership, so thanks! I just think precise Python terminology might keep people from getting confused about different types of Python iterator objects. Iterators are anything that has a __next__() method: https://www.python.org/dev/peps/pep-0234/ https://www.python.org/dev/peps/pep-0234/

deathanatos 8y ago
> Isn't the property of the generator expression not that it is an iterator (like a range or list also are) but specifically that it is a generator and thus inherently has state that is permanently/mutably exhaustible across different scopes? > just think precise Python terminology might keep people from getting confused about different types of Python iterator objects. Iterators are anything that has a __next__() method An iterator will function exactly as poorly as a generator does in his first example: In [5]: l = [1, 2, 3] In [6]: iter_ = iter(l) In [7]: min(iter_) Out[7]: 1 In [8]: max(iter_) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-8-0dab0672ef83> in <module>() ----> 1 max(iter_) ValueError: max() arg is an empty sequence (I chose iter(a list) here as it is not a generator.) And for the same reasons, really. An Iterator in Python (something w/ a __next__ method) also keeps mutable state, and the first call to min() exhausts it (by mutating it). (In Rust, it would likely "consume" the iterator, that is, it would move it, and the article demonstrates an example of this.) The mutable state being kept and used for iteration is the material part here; the generator is just an easy way of obtaining an iterator. Hopefully I've got this right, but in Python, all generators are iterators¹ (they have a __next__); generators also additionally have send(), throw(), and close() methods, making them substantially more powerful. They make it really easy to make just a basic iterator, so they're often used for that. ¹The glossary actually calls the functions (those with "yield") themselves generators, and the objects returned by them "generator iterators".
rahimnathwani 8y ago

merlincorey 8y ago

The Python problem can be solved with `itertools.tee`[0] from itertools import tee squares_min, squares_max = tee(x * x for x in range(10)) print(min(squares_min)) print(max(squares_max)) [0] https://docs.python.org/3/library/itertools.html#itertools.tee https://docs.python.org/3/library/itertools.html#itertools.t...

c8g 8y ago
squares = [x * x for x in range(10)] print(min(squares)) print(max(squares)) this one works and looks good.
- skbly7 8y ago
  It will have memory impact though with larger range. A better approach might be: squares = lambda: (x * x for x in range(10)) print(min(squares)) print(max(squares()))
  - rahimnathwani 8y ago
    Yes, your solution is in the original post, under the heading 'Python solutions': def squares(): return (x * x for x in range(10))
- adrianN 8y ago
  It also uses O(n) memory.
  - sitkack 8y ago
    We need a language that can schedule and fuse those folds over an infinite sequence concurrently. Joining a set of futures over an infinite sequence fuses folds
    - adrianN 8y ago
      I'm not sure adding more magic to languages is necessary when the alternative of using explicit generators is not too difficult.

mlthoughts2018 8y ago

I can’t comment on the general case, but I disagree very strongly with the author’s perspective about generator expressions here. First, a generator is a precise type of data structure that can be exhausted. If you don’t want that, don’t use that structure. It’s foolish to say that just because a data structure superficially satisfies some API (like being passable to min), that it “ought to have” some behavior that the author wants. That’s backwards. If you want certain behavior, choose a data structure that supports it. It would be no different from saying if I have a heap and I call sort() it is faster than if I have the same data in a list and call sort(). They both support sort() so why aren’t they equally fast? It’s obviously the wrong way to think about it. Instead, use the appropriate data structure. And if you’re really worried about the operation being safe, use the provided default argument, or the pattern for versions before 3.4, as in [0], or write a wrapper function that catches whatever exceptions and handles them. This isn’t even the biggest issue though. Generators generally are meant to function as coroutines and for complex generators, you can use send() to set the “returned value” of a yield statement inside the generator for when it resumes, and so a single generator could go through stages of being exhausted and empty, then having a value set again so that on resumption it’s no longer empty, and could build back up a whole iterable of data that way. So you absolutely do want min() or max() to throw errors when trying to consume from an empty generator, and you absolutely do want a generator’s state to become modified by the way values are consumed or added, because it meaningfully represents the state of the generator. Suppose for a more complex generator, someone uses send() to feed it a value between your call to min() and your call to max(). Generator expressions that superficially look like list comprehensions are just one use case of the general data structure and in fact one of the “generators 101” pieces of advice you always see is that if you need to iterate the values multiple times, then populate a persistent data structure out of the generator, like by wrapping in list(). I bring this up because it frustrates me to see people acting like a type safety or borrow checking idea is somehow aadding something new or solving some type of problem that the dynamic typing approach doesn’t or can’t solve, but this is emphatically incorrect and a severely myopic way to look at it. The dynamic typing approach solves all the same problems, it just facilitates the solution differently: use the default argument or other patterns for safety on functions that can fail on empty containers, and use custom wrappers with exception handling when you need to. Choose a persistent data structure if you need persistence. Again, not saying this has anything to do with people’s big picture reasons for liking static types or borrow checking... just this particular example strikes a chord to me highlighting the super annoying myopia about it. [0]: < https://stackoverflow.com/questions/36157995/a-safe-max-function-for-empty-lists https://stackoverflow.com/questions/36157995/a-safe-max-func... >

shmageggy 8y ago

Not that it really matters, but since he lamented that the second Python solution was "longer and less elegant", I'll point out that that algorithm can be written in a functional style that is both short and (IMO) somewhat elegant. However with reduce being removed as a built-in since Python 3, this is arguably not Pythonic. from functools import reduce squares = (x * x for x in range(10)) min_and_max = lambda current, x: [min(current[0], x), max(current[1], x)] minimum, maximum = reduce(min_and_max, squares, [next(squares)]*2) print(minimum) print(maximum)

c8g 8y ago
squares = [x * x for x in range(10)] print(min(squares)) print(max(squares)) this one works and looks good.
- wyldfire 8y ago
  range(10) doesn't properly demonstrate the material differences between generators and lists. Like posted elsewhere, tee is an appropriate solution that preserves the generator's benefits over a list. It's too bad that the example is so synthetic.
  - shmageggy 8y ago
    One downside of both solutions is that they both traverse the list twice, which could be expensive when we're talking about real data and not toy problems.

vbsteven 8y ago

For me Rust ownership clicked when I was reading a C++ book. More specifically the sections about references, move and copy.

bogomipz 8y ago

I really enjoyed reading this, I thought the author made these very accessible. Perhaps the author might consider writing posts on both traits and lifetimes? Understanding ownership, traits and lifetimes seem to be the keys to understanding and using Rust effectively. Unfortunately these are also the things I've had the hardest time getting my head around. If anyone could suggest similarly accessible articles on traits and lifetimes I would greatly appreciate it. Cheers.