21 ms·

Am I the only one that thinks this is a stupid decision? This will silently break code that starts to rely on this behaviour that gets executed on Python3.5 and

• kyrra 7y ago
  Go purposely "randomizes" order of a map when it's iterated over, which you can see running this demo: https://play.golang.org/p/DISpyv0Zuq_j https://play.golang.org/p/DISpyv0Zuq_j HN discussed this a little 6 years ago: https://news.ycombinator.com/item?id=7655948 https://news.ycombinator.com/item?id=7655948 As crawshaw pointed in that discussion, it helps catch people inadvertently relying on map order in tests or other places in their code.
  • throwaway8291 7y ago
    That surprised me, when I first saw it. If someone takes a minute to read the docs (e.g. Python), one would never build something knowing that order is not guaranteed (e.g. order in dictionaries). But seemingly people do.
    • UncleEntity 7y ago
      Had some code break while porting from 2 to 3 because it assumed dict order and it was part of the official Python build system, the old spark parser for the AST module.
  • ipsin 7y ago
    More important (in my opinion), adding a random seed to your hash prevents collision attacks, where an attacker tries to ruin your performance by sending many values (e.g. HTTP params) that would hash to the same bucket.
    • tomp 7y ago
      Many (most?) programming language runtimes now do this, especially the web-focused ones. But it's completely orthogonal to insertion-ordered iteration (which is implemented by a list).
• setr 7y ago
  I'm not clear on how this break existing code. Code that assumed it was arbitrary, would expect to handle any arbitrary order, including a happens-to-be sorted order. Code that assumed it was random, like actually inserted by random(), was already broken, because that simply isn't the case. Code that assumed the order would stay constant was relying on implementation-specific behavior, and could potentially break on any version update; as with any reliance on implementation-specific behavior, you'd break if the dictionary code ever got touched -- even if it were for a bugfix. Code that ordered the dictionary keys before iterating are now slightly innefficient due to extra work of sorting a sorted list.
  • deleted 7y ago
    [deleted]
  • biddlesby 7y ago
    It doesn’t break existing code. Code written for Python 3.7 might break on older versions of Python
    • setr 7y ago
      You're right, I read the gp too quickly. But in the case of downgrading, I'm fairly sure there's a number of other breaking changes that can't trivially downgrade minor versions. Like f-strings were only introduced in python3.6 as I recall. Async keyword only exists as of 3.4 as well I think?
      • ses1984 7y ago
        Introducing things is different than changing things.
        • deleted 7y ago
          [deleted]
        • setr 7y ago
          Sure, but you can't safely take everything from a higher version to a lower version in any case; if insertion order became gauranteed due to a bugfix, and wasn't backported, you'd be in the same boat. The only way to consistently code cross-version is to start with the lowest you plan to support (assuming the higher versions are actually backwards-compatible). Does any language gaurantee that code is both backwards and forwards compatible?
• pmart123 7y ago
  I agree, but more from the point of view on what if a new unordered implement if discovered 10 years from now that’s better? Would a new type have to be added?
• doubleunplussed 7y ago
  3.5 is EOL soon, and it's already uncommon to write code using a new interpreter that targets an older interpreter - there's more than just ordered dicts that you'll be missing if you do so. People will just do what they've always done - they'll be aware of the differences, or they'll test. There's nowhere I have seen that mentions dicts are ordered without mentioning in the same paragraph the version since which this has been the case, so anyone who knows they're ordered will be aware. Tldr its fine
• _bxg1 7y ago
  > I would consider changing how a builtin works to be a major breaking change. This one is unusual because it won't break old code being brought forward, only the other way around, and theoretically there are lots of things going the other direction that would break (though most of them are explicit, not silent).
  • manaskarekar 7y ago
    In other words, it breaks backwards compatibility, but not forward compatibility.
    • elcomet 7y ago
      That's not what people call backward compatibility. Backward compatibility is the ability to run old code with new interpreters. This is not broken here. What's broken is the ability to run new code on old interpreters. But this is already broken at every python update (new operators, methods, syntactic sugar..). We could call it reverse backward compatibility.
      • manaskarekar 7y ago
        I suppose the meaning of the term depends on what we consider to be the subject of the compatibility. - backwards compatible code: new code can run on an old interpreter - backwards compatible interpreter: old code can run on a new interpreter EDIT: After some thought, you're right. The second description is the reasonable interpretation. "python 3.7 is backwards compatible with python 3.6"
        • rovolo 7y ago
          This confusion reminds me of a scene from the 2002 film of Dave Barry's "Big Trouble". (Driving to an airport) "Okay, we gotta pick a road. Arrivals or departures? We're arriving, but then we're departing." https://en.wikiquote.org/wiki/Big_Trouble https://en.wikiquote.org/wiki/Big_Trouble https://www.youtube.com/watch?v=EjHHzZAUxbM&t=300 https://www.youtube.com/watch?v=EjHHzZAUxbM&t=300
• Waterluvian 7y ago
  You can't walk Python code backwards anyway without already considering the minor version differences. eg. f'strings'.
  • jkbbwr 7y ago
    At least f"strings" are an explicit failure case and are obvious when you run unit tests / coverage
  • elcomet 7y ago
    Not only f strings, but any change really.. New operators, new methods on standard library objects... Python never guaranteed compatibility of new code with old interpreters.
• mumblemumble 7y ago
  I'm of two minds. In principle, no, I don't love the idea of a change like this happening on a minor version. But, from a pragmatic angle, it strikes me as a genius move. As the article said, this was a natural by-product of a performance enhancement that was made in 3.6. In principle, that was fine, because there was officially no predictable ordering, so a change to how it was being ordered in practice shouldn't materially affect anyone. And nobody really paid much notice to it then. And that's where the danger lies - there's risk there, because, as you said, someone who's used to Python 3.6 might have problems switching to 3.5. And that's probably a greater risk if they left it out of the spec, because people would have continued to not pay it much notice. By making it official in 3.7, though, they've sent out a message that the Internet hate machine has ensured that everyone will hear. That probably, in practice, actually reduces the risk. And, while many understandably see this as violating the spirit of semver, it unambiguously does not break the letter: Minor changes are only supposed to avoid breaking the software's compatibility with code written against an old version. There was never any rule saying that old versions have to be compatible with code written against the new version.
  • frosted-flakes 7y ago
    Python (the language) doesn't follow semver anyway.
• Sean1708 7y ago
  My main issue with it is that if someone figures out an even better way to do dictionaries but it doesn't preserve insertion order, then that technique can't be used. But I certainly wouldn't call it a stupid decision, the problem is that once a behaviour is observable then people absolutely will start relying on it.
  • harikb 7y ago
    I think this is a key point. Since they advertised this, it became a commitment for all future versions. Sure they can break again in 4.0
  • mark-r 7y ago
    All they have to do is give it a different name, such as unordered_dict.
    • compiler-guy 7y ago
      Which means no one gets it for free when it is added to the language.
      • jmilloy 7y ago
        You could add an ordered dict by a different name and also have dict be the more efficient ordered version so that people get the efficiency gain for free but don't have to worry and forward compatibility.
    • jmilloy 7y ago
      I might have preferred odict be added and dict left alone, so that you didn't have to import OrderedDict when really you just want the new dict. If dict and odict mapped to the same thing for a while, fine, but dict could diverge again if desired.
      • masklinn 7y ago
        The problem is that when you behave a certain predictable and deterministic way users will leverage that even if it's not specified. That is actually why the Python maintainers decided to make this behaviour official: the ordering was the consequence of changes in implementation details, but over 3.6's lifecycle they feared it would cause compatibility issues as users would start relying on the ordering properties of CPython and that would be an issue for alternate implementations (though pypy had switched to the same implementation even earlier so was also insertion-ordered even when running in Python 2.7 mode). Providing stronger guarantees was considered useful and unlikely to be severely detrimental in the long run, so the project decided to make it part of the language spec.
• EGreg 7y ago
  PHP has had this undocumented feature since forever and so did all mainstream Javascript engines. Big whoop
  • dehrmann 7y ago
    > Javascript Javascript doesn't specify performance characteristics of objects and arrays, so even with implementations, one object could be a hashtable, another a balanced tree.
    • Arnavion 7y ago
      The implementation doesn't matter to EGreg's point, which is that ES standardized that iterating over object keys are generally required to return keys in insertion order. In the past (from before this standardization) Chrome had in fact changed the object iteration order due to an optimization, and had to revert it after lots of complaining on their bug tracker. (To be precise, the spec still requires array index properties to be returned ahead of other properties regardless of insertion order. The behavior that Chrome "reverted" to is this new one, so not exactly the same as the original behavior.)
• kayf 7y ago
  Actualy, dict are already ordered since a long time(since python 3 I think), but it wasn't certified. It was just the result of a new implementation of dict structure. I can't imagine how it can break something. In which case can you have an advantage to have an unordered list? Biggest downside can be about perf, but I don't think it's the case here.
  • pantsforbirds 7y ago
    python 3.6 is when they reworked the dictionary and had the side effect of being ordered.
• jokoon 7y ago
  I don't see exactly how one would rely on dicts not being sorted, as it was already answered by another comments. And beyond this, even if maintaining backward compatibility is important, I don't think it should prevent software from being improved, and dicts being ordered is pretty awesome. I'm pretty sure relying on unsorted dicts might involve pretty hacky code. I recently stopped writing a z-order test application because I remembered that dicts are not sorted, so it made my goal a little pointless. Meanwhile C++'s std::map is sorted. I think there was already sorted dict in python, but I don't remember. EDIT: I'm wrong, I'm mixing "ordered" and "sorted"
• riazrizvi 7y ago
  That is an original line of thinking in the world of software, “Let’s not add a feature to a new version because it wouldn’t work if used in an older version”.
  • Biganon 7y ago
    It would silently introduce bugs, that's the whole issue.
    • nkozyra 7y ago
      Right. In fact, if it explicitly broke something there'd be less of an issue. Instead, it will appear to work/interpret but you'll get unreliable results. That makes the bug more nebulous.
• carapace 7y ago
  No, I agree, it's stupid. To me it feels like a kind of conceptual backwash from JS. The whole P3 fiasco looks really silly from far enough away. It's like, you have a great marriage with everything working out and some fine children who are doing great too, but you get a divorce anyway to pursue your dreams of being a conceptual artist. (BTW, in case anyone is keeping track, I was going to maintain P2 but got caught up in Prolog, of all things, and now Python looks just as crude as everything else and I don't have the requisite love anymore to shoulder the work. I might make a Python 2 interpreter in Prolog though! That would be fun.)
• OJFord 7y ago
  It's not breaking, it was implementation defined, now it's guaranteed order. If a new guarantee or behaviour were a breaking change, every non-patch release of a semversioned tool (which python isn't) would be major.
  • nkozyra 7y ago
    > It's not breaking And that's sort of the problem. If it was broken, you'd know it and you'd fix/rearchitect it. Instead, it will appear to work.
    • dagmx 7y ago
      Sort of. You'd only know based on the interpreter you were using. You wouldn't know unless you tested other Python interpreters. For example, if you wrote something in pypy it would be ordered versus cpython.
  • masklinn 7y ago
    > It's not breaking, it was implementation defined And randomised at interpreter start, since Python 3.3.
• ptero 7y ago
  I think you are giving it bigger weight than it deserves. Writing Python code I would not want to depend on internal order of dict elements without a very good reason. If I am writing some low level code that could benefit from it, maybe I would use this feature, but I would then encapsulate it and insert a version check to complain. This is like many evaluation guarantees in latest C++-NN standards. Good to know, might be useful for performance tuning of automatic code generators, etc. , but irrelevant most of the time. My 2c.
• toyg 7y ago
  If you rely on this feature, surely you know it’s new to 3.7 and document your project accordingly...? It’s not like code magically appears, somebody has to write it.
• nwsm 7y ago
  By that logic, any change would be considered "breaking" because people will expect that the change also exists in older versions.
• jmilloy 7y ago
  I think if you are a library developer targeting different versions of Python, you generally know what features you can and can't use from new versions of Python (ie not many), and ideally you are also testing with each version in CI.
• boublepop 7y ago
  Code that assumes no order cannot be broken by a specific order unless it was setup to assume everything but that specific order in which case it was already broken, but only randomly showing it’s broken nature.
• pcj-github 7y ago
  This was my initial thought as well. Seems like people are celebrating, but this is going to be a source of difficult to figure out bugs.
• hinkley 7y ago
  For anyone who came to Unit Testing with the XP era and happened to use Java, Java 5 introduced changes to hash table ordering that somehow resulted in Junit 3.0 running some tests in a different order. It's important to discover coupling between your tests, this just isn't the way most people want to do it.
• wodenokoto 7y ago
  I think you bring up a good point. If someone hands me a python script and told me it was 3.7 I would only hear 3 and expect it to run or fail, not to run with differing outputs! Dot-upgrades do allow for breaking changes, but this one can break silently and I’m not sure where such breakages neither belongs in python nor where they should belong. Personally I’m happy to get ordered ducts sooner rather than later.