16 ms·

This is an amazing contribution to the language. A mixture of speed and convenience, probably made by volunteers. As for people criticizing a change to what u

This is an amazing contribution to the language. A mixture of speed and convenience, probably made by volunteers. As for people criticizing a change to what use to be a non-deterministic ordering of a dict iteration; I don't know what to say to them, other than, are you serious? There are people out there who are working for us, they work for free and they did some heavy lifting to give us this. They might read what you wrote and think, "Why bother? Maybe I should spend my weekends playing with my kids instead."

bbrik 7y ago
As far as I know, it was implemented by Raymond Hettinger. This is a very interesting talk about the new tech: https://www.youtube.com/watch?v=p33CVV29OG8&t=1202s https://www.youtube.com/watch?v=p33CVV29OG8&t=1202s
- brettcannon 7y ago
  Raymond came up with the idea, PyPy implemented it, and then INADA Naoki implemented it for CPython.
  - takeda 7y ago
    If I remember correctly his talk. He went to CPython first and it was rejected, but PyPy welcomed it.
- lapink 7y ago
  Indeed, here is the mail from Raymond Hettinger explaining the new design on the python-dev list: https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2012-December/1... Quite amazing that it is faster, more memory efficient and convenient for the end user.
- masklinn 7y ago
  It was proposed by Raymond Hettinger, but was actually first implemented by PHP and Pypy.
wutbrodo 7y ago
> As for people criticizing a change to what use to be a non-deterministic ordering of a dict iteration; I don't know what to say to them, other than, are you serious? There are people out there who are working for us, they work for free and they did some heavy lifting to give us this. They might read what you wrote and think, "Why bother? Maybe I should spend my weekends playing with my kids instead." I don't agree with the complaints about ordering ducts, but this is a ludicrous response. Appreciation of volunteers' work on Python isn't diminished by criticism of language features or changes. In fact, it enhances it: if people have opinions about the project you work on, it's a good sign that it's important, significant work. I've contributed to OSS projects before, and the ones in use by more than just my friends and I were naturally the ones that felt the most meaningful.
- shrimpx 7y ago
  As it’s widely known, more often than not “criticism” of open source software quickly devolves into hate and toxicity. Helpful criticism is great but be careful with defending the “criticism culture” around foss, it’s often angry unhappy people that want it all for free on a golden platter, and yesterday.
  - onlydeadheroes 7y ago
    That does not matter, nothing is above criticism.
    - cmendel 7y ago
      It does matter. If people get hate for doing work on open source they will stop doing work on open source. Yes you can be critical of everything, but the hate that some people spew toward certain open projects is not only nauseating but actively damaging to the community.
  - wutbrodo 7y ago
    Sure, hate and toxicity themselves should be called out. But that's not what's happening here, and not what GP comment was referring to. It doesn't make any sense to throw out the baby of legitimate criticism with the bathwater of toxicity.
alayne 7y ago
Other languages don't generally have special order guarantees about standard maps. This seems very idiosyncratic.
- dukoid 7y ago
  Java added LinkedHashMap a while ago. I tend to default to it except for small temporary use cases where order absolutely won't matter or have an outside impact...
  - masklinn 7y ago
    LinkedHashMap is not Java's "standard maps" though. If you tend to use it by default it's really a personal idiosyncrasy, especially as LLHM tend to be larger and slower than regular hashmaps due to having to maintain a doubly linked list. Python has had one such in the standard library for a decade or so.
    - oh-4-fucks-sake 7y ago
      Yep. Also, LinkedHashMap maintains ordering based on insertion order. So, iff you insert your data ordered how you want it, it's behaving as an ordered map. If you want true, self-reordering map, what you want is a TreeMap. Beauty with that one is that you have full control over defining the custom ordering function, because we're not always indexing a map by primitives or autoboxed type. I try to use it sparingly though as you're paying log(n) on pretty much all map operations. While we're here, another tidbit that's often overlooked in the Java collections: If you reallly care about iteration performance, your data is without nulls, your data is already ordered how you like it or you don't care about ordering, your qty items >= 10, and you don't need random access, then ArrayDeque is gonna be your horse because of how much better it co-locates its contents in memory and how much less overhead is required to maintain it during each operation compared to all the other List implementations, including ArrayList and LinkedList.
    - xxs 7y ago
      LHM is not slower for iteration (it's faster actually). LHM indeed pays 2 references per node but they are well worth as it has deterministic ordering/iteration and I have witnessed numerous cases with HashMap that show up in production only due to iteration order (esp after rehashing). The code is broken but the cases did not reproduce during testing... Now if the two extra references are an issue, consider that HashMap is quite space inefficient with having a dedicated node per each entry - that's the main price. The (simple) node memory footprint is 36bytes on heaps less than 32GB, i.e. compact pointers. The extra references add another 8 bytes for having an insert or access order. If the goal is getting a compact low memory footprint, HashMap is not fit for purpose. Overall it's the jack of all trades and even got reworked (java8) to support tri-based collision resolution for keys that implement Comparable. Couple years back I wrote CompactHashMap (under CC0) that has an average cost of 10bytes per entry and it's extremely compact for small sizes with having only 2 fields on its right own, so even small/empty maps are tiny. In microbenchmarks (same used in openjdk) it handily (2x) beats java.util.HashMap on "Traverse key or value", get/put have similar performance, and "Search Absent" is worse. The point is: LHM should be the go-to hashmap for java as being node based hashmap is justified (unlike java.util.HashMap)
lukeschlather 7y ago
Especially with the (lack of) change to sets I'm interested to benchmark some regular things before and after the change. set(dict.keys()) set(dict.values()) thing1 = dict(...) thing2 = copy.deepcopy(thing1) I feel like there could be some other testcases. I'm wholly in support of the change (regardless of benchmarks) but depending on the results I could see some arguments against.
- masklinn 7y ago
  1. sets don't use the new dict's implementation 2. IIRC the new dicts are no(t significantly) slower than the old dicts, however they use less memory, and iterate faster The iteration order is actually a side-effect of implementation details, the original goals were a more compact representation and a faster iteration: https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2012-December/1...
  - lukeschlather 7y ago
    > 1. sets don't use the new dict's implementation Yes, of course. That's why I'm wondering if turning dict keys into a set is slower now.
the_matrix 7y ago
I don't have any criticism if the performance of this new dict stays the same as earlier and that's a huge if. Considering python3 is already a crawling language compared to peers like java and php, I think devs should focus more on that before handing out features.
- masklinn 7y ago
  > I don't have any criticism if the performance of this new dict stays the same as earlier and that's a huge if. 1. it's been there since 2016, the new dict was released as part of Python 3.6 (though the ordering was only an implementation detail until 3.7) 2. the new dict is much more compact (20~25%) and should have much faster iteration speed, those were actually the original goals, ordered iteration was a side-effect 3. the new dict should not benchmark significantly slower across the board, it wouldn't have been merged if it were a regression
jonreem 7y ago
People who bother to complain are those who actually care about your thing. People who do not care simply leave without ever telling you why. Your complainers are often your most dedicated and invested users.
- mumblemumble 7y ago
  Though in this specific case, my sense has been that your complainers are often your most dedicated and invested users of a version of the language that was officially retired 38 days ago. Python is just bizarrely political. I suspect that it's now cursed for all time to have every future PEP become a battle in a proxy war over PEP 3000.
- gerikson 7y ago
  Sometimes they're just concern trolls though.
onei 7y ago
I was under the impression python dict ordering was deterministic, just not in an order recognisable by humans, i.e. ordered by hashes. Was this not the case?
- stestagg 7y ago
  From the docs: "CPython implementation detail: Keys and values are listed in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions." So I would claim that equates to non-deterministic
  - baq 7y ago
    I’d call undependable but maybe it’s the same thing in practice?
    - baddox 7y ago
      That's generally what people mean when they say "nondeterministic" in the context of computing. Yeah, in philosophy it generally means something like "the future is not completely determined by the past," but in computing it means something closer to "the programmer cannot reasonably determine the behavior and thus should not depend on specific behavior."
      - mypalmike 7y ago
        In computing it means a given set of inputs lead to a given set of outputs. It has nothing to do with how difficult it is for a programmer to reason about. Deterministic builds, deterministic tests, etc.
        Dylan16807 7y ago
        But what counts as "input" will vary based on who you ask and under what context.
  - 7y ago

bbrik 7y ago

As far as I know, it was implemented by Raymond Hettinger. This is a very interesting talk about the new tech: https://www.youtube.com/watch?v=p33CVV29OG8&t=1202s https://www.youtube.com/watch?v=p33CVV29OG8&t=1202s

brettcannon 7y ago
Raymond came up with the idea, PyPy implemented it, and then INADA Naoki implemented it for CPython.
- takeda 7y ago
  If I remember correctly his talk. He went to CPython first and it was rejected, but PyPy welcomed it.
lapink 7y ago
Indeed, here is the mail from Raymond Hettinger explaining the new design on the python-dev list: https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2012-December/1... Quite amazing that it is faster, more memory efficient and convenient for the end user.
masklinn 7y ago
It was proposed by Raymond Hettinger, but was actually first implemented by PHP and Pypy.

wutbrodo 7y ago

> As for people criticizing a change to what use to be a non-deterministic ordering of a dict iteration; I don't know what to say to them, other than, are you serious? There are people out there who are working for us, they work for free and they did some heavy lifting to give us this. They might read what you wrote and think, "Why bother? Maybe I should spend my weekends playing with my kids instead." I don't agree with the complaints about ordering ducts, but this is a ludicrous response. Appreciation of volunteers' work on Python isn't diminished by criticism of language features or changes. In fact, it enhances it: if people have opinions about the project you work on, it's a good sign that it's important, significant work. I've contributed to OSS projects before, and the ones in use by more than just my friends and I were naturally the ones that felt the most meaningful.

shrimpx 7y ago
As it’s widely known, more often than not “criticism” of open source software quickly devolves into hate and toxicity. Helpful criticism is great but be careful with defending the “criticism culture” around foss, it’s often angry unhappy people that want it all for free on a golden platter, and yesterday.
- onlydeadheroes 7y ago
  That does not matter, nothing is above criticism.
  - cmendel 7y ago
    It does matter. If people get hate for doing work on open source they will stop doing work on open source. Yes you can be critical of everything, but the hate that some people spew toward certain open projects is not only nauseating but actively damaging to the community.
- wutbrodo 7y ago
  Sure, hate and toxicity themselves should be called out. But that's not what's happening here, and not what GP comment was referring to. It doesn't make any sense to throw out the baby of legitimate criticism with the bathwater of toxicity.

alayne 7y ago

Other languages don't generally have special order guarantees about standard maps. This seems very idiosyncratic.

dukoid 7y ago
Java added LinkedHashMap a while ago. I tend to default to it except for small temporary use cases where order absolutely won't matter or have an outside impact...
- masklinn 7y ago
  LinkedHashMap is not Java's "standard maps" though. If you tend to use it by default it's really a personal idiosyncrasy, especially as LLHM tend to be larger and slower than regular hashmaps due to having to maintain a doubly linked list. Python has had one such in the standard library for a decade or so.
  - oh-4-fucks-sake 7y ago
    Yep. Also, LinkedHashMap maintains ordering based on insertion order. So, iff you insert your data ordered how you want it, it's behaving as an ordered map. If you want true, self-reordering map, what you want is a TreeMap. Beauty with that one is that you have full control over defining the custom ordering function, because we're not always indexing a map by primitives or autoboxed type. I try to use it sparingly though as you're paying log(n) on pretty much all map operations. While we're here, another tidbit that's often overlooked in the Java collections: If you reallly care about iteration performance, your data is without nulls, your data is already ordered how you like it or you don't care about ordering, your qty items >= 10, and you don't need random access, then ArrayDeque is gonna be your horse because of how much better it co-locates its contents in memory and how much less overhead is required to maintain it during each operation compared to all the other List implementations, including ArrayList and LinkedList.
  - xxs 7y ago
    LHM is not slower for iteration (it's faster actually). LHM indeed pays 2 references per node but they are well worth as it has deterministic ordering/iteration and I have witnessed numerous cases with HashMap that show up in production only due to iteration order (esp after rehashing). The code is broken but the cases did not reproduce during testing... Now if the two extra references are an issue, consider that HashMap is quite space inefficient with having a dedicated node per each entry - that's the main price. The (simple) node memory footprint is 36bytes on heaps less than 32GB, i.e. compact pointers. The extra references add another 8 bytes for having an insert or access order. If the goal is getting a compact low memory footprint, HashMap is not fit for purpose. Overall it's the jack of all trades and even got reworked (java8) to support tri-based collision resolution for keys that implement Comparable. Couple years back I wrote CompactHashMap (under CC0) that has an average cost of 10bytes per entry and it's extremely compact for small sizes with having only 2 fields on its right own, so even small/empty maps are tiny. In microbenchmarks (same used in openjdk) it handily (2x) beats java.util.HashMap on "Traverse key or value", get/put have similar performance, and "Search Absent" is worse. The point is: LHM should be the go-to hashmap for java as being node based hashmap is justified (unlike java.util.HashMap)

lukeschlather 7y ago

Especially with the (lack of) change to sets I'm interested to benchmark some regular things before and after the change. set(dict.keys()) set(dict.values()) thing1 = dict(...) thing2 = copy.deepcopy(thing1) I feel like there could be some other testcases. I'm wholly in support of the change (regardless of benchmarks) but depending on the results I could see some arguments against.

masklinn 7y ago
1. sets don't use the new dict's implementation 2. IIRC the new dicts are no(t significantly) slower than the old dicts, however they use less memory, and iterate faster The iteration order is actually a side-effect of implementation details, the original goals were a more compact representation and a faster iteration: https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2012-December/1...
- lukeschlather 7y ago
  > 1. sets don't use the new dict's implementation Yes, of course. That's why I'm wondering if turning dict keys into a set is slower now.

the_matrix 7y ago

I don't have any criticism if the performance of this new dict stays the same as earlier and that's a huge if. Considering python3 is already a crawling language compared to peers like java and php, I think devs should focus more on that before handing out features.

masklinn 7y ago
> I don't have any criticism if the performance of this new dict stays the same as earlier and that's a huge if. 1. it's been there since 2016, the new dict was released as part of Python 3.6 (though the ordering was only an implementation detail until 3.7) 2. the new dict is much more compact (20~25%) and should have much faster iteration speed, those were actually the original goals, ordered iteration was a side-effect 3. the new dict should not benchmark significantly slower across the board, it wouldn't have been merged if it were a regression

jonreem 7y ago

People who bother to complain are those who actually care about your thing. People who do not care simply leave without ever telling you why. Your complainers are often your most dedicated and invested users.

mumblemumble 7y ago
Though in this specific case, my sense has been that your complainers are often your most dedicated and invested users of a version of the language that was officially retired 38 days ago. Python is just bizarrely political. I suspect that it's now cursed for all time to have every future PEP become a battle in a proxy war over PEP 3000.
gerikson 7y ago
Sometimes they're just concern trolls though.

onei 7y ago

I was under the impression python dict ordering was deterministic, just not in an order recognisable by humans, i.e. ordered by hashes. Was this not the case?

stestagg 7y ago
From the docs: "CPython implementation detail: Keys and values are listed in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions." So I would claim that equates to non-deterministic
- baq 7y ago
  I’d call undependable but maybe it’s the same thing in practice?
  - baddox 7y ago
    That's generally what people mean when they say "nondeterministic" in the context of computing. Yeah, in philosophy it generally means something like "the future is not completely determined by the past," but in computing it means something closer to "the programmer cannot reasonably determine the behavior and thus should not depend on specific behavior."
    - mypalmike 7y ago
      In computing it means a given set of inputs lead to a given set of outputs. It has nothing to do with how difficult it is for a programmer to reason about. Deterministic builds, deterministic tests, etc.
      - Dylan16807 7y ago
        But what counts as "input" will vary based on who you ask and under what context.
- 7y ago