7 ms·

This doesn't feel right.. it's not actually a nice side effect at all. 1. it's not in the spec 2. you shouldn't rely on it 3. python can't figure out if you'

by _hyn3 10y ago

This doesn't feel right.. it's not actually a nice side effect at all. 1. it's not in the spec 2. you shouldn't rely on it 3. python can't figure out if you're relying on it, so no error will be raised 4. subtle bugs are sure to be introduced by people who "know" this "feature" exists and use it. Regardless of the cool implementation details, this post shouldn't advertise that "keywords become ordered" and "A nice "side effect" of compact dict is that the dictionary now preserves the insertion order". Ergo... we built this awesome thing that you'd love to use but you can't. Don't use it or you'll be a bad programmer who doesn't read specs! (Just use addict or OrderedDict.)

tedmiston 10y ago
I tend to agree that this is something that could be confusing to rely on, especially if running tests against multiple version of Python (such as a previous version where this isn't the behavior). Explicit is better than implicit. There should be one-- and preferably only one --obvious way to do it.
- hayd 10y ago
  In python 2 the ordering was deterministic. Cue lots of weird bugs when upgrading to python 3. Of course, it should never have been relied on... but it had been. I'd prefer either a completely random order or a completely deterministic one - but please no take backs (people will rely on it).
  - Retra 10y ago
    I'm not sure anybody should be relying on code that relies upon the order of an unordered dict.
    - masklinn 10y ago
      These reliances usually aren't explicit, they just happen to work that way and it can be very hard to tease out the dependency unless you have a way to easily screw with iteration order. Pretty much any unspecified but (because of implementation) reliable ordering will eventually end up with code relying on it, usually implicitly, e.g. some Go code broke in 1.5 (and the release notes noted it explicitly) when scheduling order changed[0] despite it always having been undefined. In fact, specifically to catch this kind of hidden dependencies the Go developers added limited scheduling randomisation to the race detection mode. [0] from definition-ordering to last-definition-bias e.g. if you launched 3 goroutines 1, 2 and 3, in 1.4 the scheduler would just run them in that order, in 1.5 it would run 3 first followed by 1 and 2.
      - Retra 10y ago
        That's true, but to rely on these behaviors is to introduce bugs into your code, and we should be detecting and correcting them, not forgoing semantically clean optimizations because someone's buggy code will break.
- camel_Snake 10y ago
  And so we get a 3rd way to interpolate strings in 3.6.
notdonspaulding 10y ago
At least with respect to ordered kwargs, the spec is changing: https://www.python.org/dev/peps/pep-0468/ https://www.python.org/dev/peps/pep-0468/ I believe it's an implementation detail that in cpython 3.6, dict and odict are essentially equivalent now, but it's an implementation detail which is there to support an official feature of the language. If your project can sustain a minimum version of (the as yet unreleased) Python 3.6, I don't see why you shouldn't begin to rely on this feature straightaway.
- carapace 10y ago
  Ugh. This is all bad. This just isn't the sort of thing that benefits from fucking with. This is imposing a slight but not-insignificant open-ended technical debt on the whole of the Python ecosystem, for what seems to me to be very obscure benefits.
  - xapata 10y ago
    > obscure benefits 1. No need for metaclasses simply to make the class construction locals dict an ordered dict. ORM library devs will be happy. 2. No need to pass a list of tuples when you wanted to pass keyword arguments. That's a big win for usability. > technical debt The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt? > This just isn't the sort of thing that benefits from [change]. The core devs say that status quo is the sane default. They're not just mucking about for the amusement of code churn.
    - rocqua 10y ago
      > The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt? The debt will accrue in the wider ecosystem. The spec doesn't guarantee dicts are ordered, but people will rely on it. Then, when a new dict implementation comes along, all code written based on a (then) correct assumption (but not based on the spec) suddenly stops working.
munificent 10y ago
For what it's worth, the JS ecosystem has tried for years to get users to not rely on key iteration order. It was always unspecified but no matter what, users still expected them to iterate in insertion order. V8 fought against it for years. Look at this bug horror show: https://bugs.chromium.org/p/v8/issues/detail?id=164 https://bugs.chromium.org/p/v8/issues/detail?id=164 I think Python is doing the right thing. The only reason to not provide a deterministic iteration order for unsorted maps is to give more room for optimizers. But it seems like there is a clean implementation that is faster while also iterating over keys in insertion order. Sure, maybe someone will come up with an even better optimization in the future that would break this, but at some point, you have to say, "OK, what we have is good enough, and it gives users a more predictable system." If you are going to declare insertion order is non-deterministic, you really should make it non-deterministic by doing something like shuffling it. Otherwise, users will just inadvertently rely on whatever deterministic-but-unspecified-order the current implementation happens to provide.
- deleted 10y ago
  [deleted]
- xapata 10y ago
  > make it non-deterministic In Python v3.1 - v3.5 the order is deterministic within an execution, but non-deterministic across executions. This was implemented to avoid the possibility that a hacker could slow down dict key lookup/insertion by creating a large number of hash table collisions. https://mail.python.org/pipermail/python-announce-list/2012-March/009394.html https://mail.python.org/pipermail/python-announce-list/2012-... IMHO, that doesn't seem to be a big vulnerability.
  - deleted 10y ago
    [deleted]
  - wahern 10y ago
    It never seemed to catch on with the bad guys, but that doesn't mean it wouldn't have happened if major languages hadn't implemented counter measures. DDoS using DNS amplification was possible for decades but never happened. Then one day a bad guy got bored, wrote a proof-of-concept, and the rest is history.
duaneb 10y ago
I interpret it differently: the ordering is a side effect, and the positive aspect is that it's easier to debug conveniently, not a feature you should rely on version to version.
a3n 10y ago
Yeah, this is not a dict that has ordered keys. It's a dict that has that side effect. I can't think of any reason why a set's members, or a dict's keys (which are just set members with associated values, conceptually) should be unordered. Yes, set membership is traditionally unordered, but we have ordered sets and dicts, so they must be desirable, and they're certainly coherent. I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions.
- dragonwriter 10y ago
  > I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions. Unspecified-order minimizes constraints on future implementations. If someone finds a useful time or space optimization (for particular workloads, not necessarily in big-O asymptotic terms) that isn't consistent with the specified ordering, that might be a compelling reason to implement a version without the ordering constraint. (But, if its not the best choice for general use, that's true even if the default implementation isn't ordered, since its possible that the optimization will still be unsuitable as the default.) And, obviously, even if the base case has one ordering, there's all kinds of reasons you might need to implement a version with a different ordering.
- pekk 10y ago
  "Ordered sets" are coherent in the sense that you can make a data structure like a set and then define an order over its members, and call it an ordered set, but "ordered sets" are not sets. There are certain purposes sets are natural for. When you introduce ordering, you will naturally be asked to add methods for querying or perhaps modifying that ordering in various ways. Many methods later, it's worth asking why you are calling this data structure a set, and what its actual purpose is supposed to be.
rbanffy 10y ago
I am not very sure this is tat bad. If code breaks because dict is no longer ordered, people should have used an ordered one from the start. It's a happy accident it is faster and ordered and not bad because whoever relies on dicts bein unordered (when sometimes they aren't) deserves whatever breakage they get.
- takeda 10y ago
  This is very similar issue that made migration to Python 3 so painful. A lot of the code had broken unicode from the start. The difference was that it worked in python 2 most of the time and failed on special cases, while python 3 was stricter and mixing bytes with string always caused it to crash. Once dict() is ordered and people will use that property (even unknowingly) if the behavior changes again in the future, a lot of their code will be broken.
  - rbanffy 10y ago
    It may be similar in that it allows for relying on consistent behavior when one shouldn't, but fixing it seems a lot easier than decades of not thinking about Unicode.
- rocqua 10y ago
  A language that makes it easier to write shitty code is worse than one that doesn't
  - 1propionyl 10y ago
    I thought Lisp was the best language though, you're telling me it's the worst? :-p
pekk 10y ago
It isn't right, but popular personalities like the creator of Flask have been demanding it for years so here we are. Welcome to Open Source.

by _hyn3 10y ago

tedmiston 10y ago

I tend to agree that this is something that could be confusing to rely on, especially if running tests against multiple version of Python (such as a previous version where this isn't the behavior). Explicit is better than implicit. There should be one-- and preferably only one --obvious way to do it.

hayd 10y ago
In python 2 the ordering was deterministic. Cue lots of weird bugs when upgrading to python 3. Of course, it should never have been relied on... but it had been. I'd prefer either a completely random order or a completely deterministic one - but please no take backs (people will rely on it).
- Retra 10y ago
  I'm not sure anybody should be relying on code that relies upon the order of an unordered dict.
  - masklinn 10y ago
    These reliances usually aren't explicit, they just happen to work that way and it can be very hard to tease out the dependency unless you have a way to easily screw with iteration order. Pretty much any unspecified but (because of implementation) reliable ordering will eventually end up with code relying on it, usually implicitly, e.g. some Go code broke in 1.5 (and the release notes noted it explicitly) when scheduling order changed[0] despite it always having been undefined. In fact, specifically to catch this kind of hidden dependencies the Go developers added limited scheduling randomisation to the race detection mode. [0] from definition-ordering to last-definition-bias e.g. if you launched 3 goroutines 1, 2 and 3, in 1.4 the scheduler would just run them in that order, in 1.5 it would run 3 first followed by 1 and 2.
    - Retra 10y ago
      That's true, but to rely on these behaviors is to introduce bugs into your code, and we should be detecting and correcting them, not forgoing semantically clean optimizations because someone's buggy code will break.
camel_Snake 10y ago
And so we get a 3rd way to interpolate strings in 3.6.

notdonspaulding 10y ago

At least with respect to ordered kwargs, the spec is changing: https://www.python.org/dev/peps/pep-0468/ https://www.python.org/dev/peps/pep-0468/ I believe it's an implementation detail that in cpython 3.6, dict and odict are essentially equivalent now, but it's an implementation detail which is there to support an official feature of the language. If your project can sustain a minimum version of (the as yet unreleased) Python 3.6, I don't see why you shouldn't begin to rely on this feature straightaway.

carapace 10y ago
Ugh. This is all bad. This just isn't the sort of thing that benefits from fucking with. This is imposing a slight but not-insignificant open-ended technical debt on the whole of the Python ecosystem, for what seems to me to be very obscure benefits.
- xapata 10y ago
  > obscure benefits 1. No need for metaclasses simply to make the class construction locals dict an ordered dict. ORM library devs will be happy. 2. No need to pass a list of tuples when you wanted to pass keyword arguments. That's a big win for usability. > technical debt The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt? > This just isn't the sort of thing that benefits from [change]. The core devs say that status quo is the sane default. They're not just mucking about for the amusement of code churn.
  - rocqua 10y ago
    > The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt? The debt will accrue in the wider ecosystem. The spec doesn't guarantee dicts are ordered, but people will rely on it. Then, when a new dict implementation comes along, all code written based on a (then) correct assumption (but not based on the spec) suddenly stops working.

munificent 10y ago

For what it's worth, the JS ecosystem has tried for years to get users to not rely on key iteration order. It was always unspecified but no matter what, users still expected them to iterate in insertion order. V8 fought against it for years. Look at this bug horror show: https://bugs.chromium.org/p/v8/issues/detail?id=164 https://bugs.chromium.org/p/v8/issues/detail?id=164 I think Python is doing the right thing. The only reason to not provide a deterministic iteration order for unsorted maps is to give more room for optimizers. But it seems like there is a clean implementation that is faster while also iterating over keys in insertion order. Sure, maybe someone will come up with an even better optimization in the future that would break this, but at some point, you have to say, "OK, what we have is good enough, and it gives users a more predictable system." If you are going to declare insertion order is non-deterministic, you really should make it non-deterministic by doing something like shuffling it. Otherwise, users will just inadvertently rely on whatever deterministic-but-unspecified-order the current implementation happens to provide.

deleted 10y ago
[deleted]
xapata 10y ago
> make it non-deterministic In Python v3.1 - v3.5 the order is deterministic within an execution, but non-deterministic across executions. This was implemented to avoid the possibility that a hacker could slow down dict key lookup/insertion by creating a large number of hash table collisions. https://mail.python.org/pipermail/python-announce-list/2012-March/009394.html https://mail.python.org/pipermail/python-announce-list/2012-... IMHO, that doesn't seem to be a big vulnerability.
- deleted 10y ago
  [deleted]
- wahern 10y ago
  It never seemed to catch on with the bad guys, but that doesn't mean it wouldn't have happened if major languages hadn't implemented counter measures. DDoS using DNS amplification was possible for decades but never happened. Then one day a bad guy got bored, wrote a proof-of-concept, and the rest is history.

duaneb 10y ago

I interpret it differently: the ordering is a side effect, and the positive aspect is that it's easier to debug conveniently, not a feature you should rely on version to version.

a3n 10y ago

Yeah, this is not a dict that has ordered keys. It's a dict that has that side effect. I can't think of any reason why a set's members, or a dict's keys (which are just set members with associated values, conceptually) should be unordered. Yes, set membership is traditionally unordered, but we have ordered sets and dicts, so they must be desirable, and they're certainly coherent. I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions.

dragonwriter 10y ago
> I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions. Unspecified-order minimizes constraints on future implementations. If someone finds a useful time or space optimization (for particular workloads, not necessarily in big-O asymptotic terms) that isn't consistent with the specified ordering, that might be a compelling reason to implement a version without the ordering constraint. (But, if its not the best choice for general use, that's true even if the default implementation isn't ordered, since its possible that the optimization will still be unsuitable as the default.) And, obviously, even if the base case has one ordering, there's all kinds of reasons you might need to implement a version with a different ordering.
pekk 10y ago
"Ordered sets" are coherent in the sense that you can make a data structure like a set and then define an order over its members, and call it an ordered set, but "ordered sets" are not sets. There are certain purposes sets are natural for. When you introduce ordering, you will naturally be asked to add methods for querying or perhaps modifying that ordering in various ways. Many methods later, it's worth asking why you are calling this data structure a set, and what its actual purpose is supposed to be.

rbanffy 10y ago

I am not very sure this is tat bad. If code breaks because dict is no longer ordered, people should have used an ordered one from the start. It's a happy accident it is faster and ordered and not bad because whoever relies on dicts bein unordered (when sometimes they aren't) deserves whatever breakage they get.

takeda 10y ago
This is very similar issue that made migration to Python 3 so painful. A lot of the code had broken unicode from the start. The difference was that it worked in python 2 most of the time and failed on special cases, while python 3 was stricter and mixing bytes with string always caused it to crash. Once dict() is ordered and people will use that property (even unknowingly) if the behavior changes again in the future, a lot of their code will be broken.
- rbanffy 10y ago
  It may be similar in that it allows for relying on consistent behavior when one shouldn't, but fixing it seems a lot easier than decades of not thinking about Unicode.
rocqua 10y ago
A language that makes it easier to write shitty code is worse than one that doesn't
- 1propionyl 10y ago
  I thought Lisp was the best language though, you're telling me it's the worst? :-p

pekk 10y ago

It isn't right, but popular personalities like the creator of Flask have been demanding it for years so here we are. Welcome to Open Source.