6 ms·
I've moved to python 3 over the past couple of months, after resisting for the better part of a decade. I like it. One surprising thing I learned from this doc
by AlexCoventry 9y ago
I've moved to python 3 over the past couple of months, after resisting for the better part of a decade. I like it.
One surprising thing I learned from this document is that dicts now iterate in assignment order, not hash order. That's going to break some code for people.
- StavrosK 9y agoDicts never had an iteration order, though. If you relied on dicts iterating in order, you were kind of asking for it.
- jstimpfle 9y agoI believe newer Python versions have at least deterministic iteration order (correct me if I'm wrong), while some previous ones had a non-deterministic one (non-deterministic between mutiple invocations of the interpreter). But there is also OrderedDict which iterates in assignment order.
- StavrosK 9y agoNo, you're right. I believe dict orders were made deterministic (by insertion order) in 3.6. Before that, it was undefined.
- rtpg 9y agoCPython interates in this fashion since 3.6, but it is not part of the Python spec. Might be a bit contentious but it's not supposed to be relied upon
- thijsvandien 9y agoIt was recently decided that as of 3.7 it will be. (Source: https://mail.python.org/pipermail/python-dev/2017-December/151283.html https://mail.python.org/pipermail/python-dev/2017-December/1...)
- dbcurtis 9y agoHmmmmmm.... that makes it hard for other implementations. CPython isn't the whole of the Python world. Requirements like that can be problematic for implementations like Micropython, for instance.
- jsmeaton 9y agoThe entire reason they decided to make it a standard rather than keep it as an implementation detail is to make things better for users. It’s not about just looking after cpython.
- sametmax 9y agoThey asked people from the Jython, pypy and uPython community if it was ok before doing so.
- lozenge 9y agoI suspect the reasoning was with CPython dominating, people would eventually make this assumption in their code, perhaps unintentionally, and reduce compatibility. Better to make it official.
- crdoconnor 9y agoThe headache that having non-deterministic code causes more than makes up for that. Among other things this makes writing tests much easier.
- ubernostrum 9y agoIn Python 2.7 prior to 2.7.3, or Python 3 prior to 3.3, iterating a dictionary would -- if it always contained the same set of keys -- be in a consistent order across multiple runs of the same program. This was an implementation detail not to be relied on. Starting in 2.7.3 and 3.3, the hashing algorithm applied a per-process random salt to certain built-in types, in order to mitigate a potential denial-of-service vector by crafting dictionary keys which cause hash collisions. Unless the PYTHONHASHSEED environment variable was set to a consistent value, iteration over a dictionary was no longer accidentally consistent across runs. In 3.6, the dictionary implementation was rewritten to drastically improve performance. As a side effect, dictionary iteration became consistent again, this time determined by insertion order. In 3.6, this consistency was an implementation detail and not to be relied on. Beginning with 3.7, dictionary iteration order is finally guaranteed by the language, independent of implementation, and goes by insertion order.
- sethgecko 9y agoIn 3.6 dicts are ordered (by insert order) as an implementation detail. From 3.7 on it will be a feature.
- fastball 9y agoPython dicts have always been touted as an unordered data structure though, so yeah, if you're depending on any type of order you shouldn't be.
- vonseel 9y agoThere’s a well known YouTube video which covers all this comprehensively. Raymond Hettinger, Modern Python Dictionaries... hopefully this is the correct one. https://youtu.be/npw4s1QTmPg https://youtu.be/npw4s1QTmPg
- kqr 9y agoOoh, I'm no longer involved in the Python world but I love presentations by Hettinger. Thanks!
- VHRanger 9y agoDicts are UNORDERED associative containers. If youre depending your app on implementation defined behavior, that's on your developpers shoulders. Stuff like that shouldn't pass code review
- jstimpfle 9y agoIt is surprising though that the order was non-deterministic, not only undefined. Determinism is something I intuitively expect from read-only operations like iteration.
- fyi1183 9y agoNon-determinism comes from randomness in the hash function. I don't know the details of the Python implementation, but generally you want randomness at the very least when processing potentially hostile user input. Otherwise your code is susceptible to a denial of service attack where the attacker sends data that was crafted to maximize hash collisions, to break the expected performance of hash tables. From a theoretical angle, the only way to build hash tables with some kind of provable runtime guarantees is to include randomness.
- brett40324 9y agoIt is simply that a python dict is not a list (read array).
- StavrosK 9y agoI think that's a habit you need to shake. If something is undefined, you should intuitively shy away from expecting things from it. If I found myself wondering why some undefined behavior was not consistent, I'd question why I even believed it should be consistent in the first place.
- ridiculous_fish 9y agoWould you expect that `str(some_dict) == str(some_dict)`?
- ATsch 9y agoEDIT: I was just informed it is as of 3.7 an official language feature, making everything below invalid Personally, I'm worried people will come to rely on the new behaviour in code instead. As core developers have repeatedly said, dict order is still an implementation detail, it should not be relied on as it is not officially part of the language. Other implementations (except pypy) will probably not have this behaviour. Yet, I feel like this will fall on deaf ears and become a de-facto part of the language. Blogs will state it as a new feature, Python books will teach it and new coders will rely on it, forever locking the dict internals in place for all python interpreters. (If you need to rely on ordering, use an OrderedDict instead.)
- sametmax 9y agoBefore 3.7, dict ordering is documented literally everywhere as "consider it implementation details, use OrderedDict if you need order". From 3.7, it's part of the spec. If you were relying on keys order before, you were not only doing it wrong, but you were doing so despite being told again and again.
- cup-of-tea 9y agoThis is why you read the documentation rather than relying on what a piece of library code appears to do. Maybe all programmers have to learn this the hard way.
- StreakyCobra 9y agoThis has been changed in Python 3.6 only, due to a change in dictionary implementation to make them more efficient. "Modern Dictionaries by Raymond Hettinger" [1] is a quite interesting and technical talk about these changes, explaining also the change in iteration order IIRC, worth a watch in my opinion! [1] https://www.youtube.com/watch?v=p33CVV29OG8 https://www.youtube.com/watch?v=p33CVV29OG8