4 ms·

We are on the brink of completing the transition to python3 at my work. The end result of this is that I just spent a good chunk of last week reviewing a pull

We are on the brink of completing the transition to python3 at my work. The end result of this is that I just spent a good chunk of last week reviewing a pull request with 70,000 lines of changes, which was one of the final in a series of ~10k line pull requests that came in through the fall. All of this was the heroic effort of one of my coworkers who had the unenviable task of combing through our entire codebase to determine "This is unicode. This is bytes. Here is an api boundary where we need to encode / decode." etc. It was a nightmare of effort that I'm glad to have behind us.

vmchale 7y ago
> All of this was the heroic effort of one of my coworkers who had the unenviable task of combing through our entire codebase to determine "This is unicode. This is bytes. Dynamic typing!
- war1025 7y ago
  Not dynamic typing's fault. The issue is they changed the types out from underneath you. And then left it to each library to decide which type it was actually going to accept.
  - magicalhippo 7y ago
    But with static typing, the compiler can let you know when you're doing something wrong with the new Unicode-based string.
    - war1025 7y ago
      Well, it can, except you then need to go through and update all of your internal APIs to be correct. Really the string transition was just a poor choice in my opinion. Python2 already had unicode strings that were easy enough to specify (just prefix with a `u`). It would have been better to just delineate that barrier better from an API standpoint. I understand the appeal of having unicode for the default string literal type, but it was actively hostile to existing projects.
      - lmm 7y ago
        > Well, it can, except you then need to go through and update all of your internal APIs to be correct. You do, but it's easy: run a compile, fix the errors, repeat until no more errors. > It would have been better to just delineate that barrier better from an API standpoint. Isn't that exactly what the Python 3 transition was? i.e. stop accepting non-unicode "strings" (actually just arbitrary byte sequences) for APIs that semantically require a string, reserve them for APIs that actually want a byte sequence.
        war1025 7y ago
        > You do, but it's easy: run a compile, fix the errors, repeat until no more errors. The reason this doesn't work is that previously the double-quote literal was a "string" type. The string type was, yes, just a sequence of bytes, but in an ascii-centric world that also mean text. Python2 added unicode string literals that accepted unicode code points. Most APIs were happy to sloppily mix the two and generally work quite adequately. Python3 then made the hard distinction between byte-string and unicode-string. Not an unreasonable position to take on the face of it. The issue is many python2 APIs were written from the perspective of "accepts string literal types", where that could be either bytestring or unicode string. Now suppose you have a large codebase in python that spans the entire stack from database interaction, to webserver, to desktop application. All built on double quoted string literals. Accepting unicode strings in the places that needed that (user-facing places mainly, utf-8 bytestrings anywhere being stored on disk or sent over network) Then you go to switch to python3, and suddenly all of your string literals are interpreted as unicode instead of bytestring / ascii sequences. So now you need to go through every place in your codebase that accepts strings as an argument and determine, "is this a user-facing string, or a utf-8 bytestring", because they used to be basically the same thing, and now they aren't. It's not "difficult" really, it's just a pain in the neck.
        lmm 7y ago
        None of that would be a problem in a typed language. The ultimate destination of any string literal is some standard library function, whether that's write to network socket, display to user, or something else. So you just ripple backwards from that through your own functions that are calling those standard library functions, until you get to the point where you're passing in the literal, and then you know what kind of literal it needs to be.
d0mine 7y ago
Something is wrong if there is no third type: the "natural" string (bytes on Python 2, unicode on Python 3).
- war1025 7y ago
  I assume many of the strings were left untouched. But you still have to audit all of it to know which needs to be used where.
- eesmith 7y ago
  I believe that's included in the "etc."
- quietbritishjim 7y ago
  Surely any "natural" string would be better represented as unicode in Python 2? What is an example that wouldn't be?
  - war1025 7y ago
    I believe what they meant was that for many strings it really shouldn't matter if they were bytes or unicode. They would perform their function correctly either way. That's completely true, but you do still have to go through and find the cases where that doesn't work.
    - deleted 7y ago
      [deleted]
  - masklinn 7y ago
    > Surely any "natural" string would be better represented as unicode in Python 2? No because much of the stdlib works in terms of native strings and will choke (or worse silently fuck up) on the other. Yes also in python 2, the stdlib was absolutely not “unicode clean”. So a transitional / polyglot codebase has and needs not 2 but 3 string types: bytes, unicode, and native. And neither “unicode literals” nor “bytes literals” were good things to apply across the board.
  - 7y ago
magicalhippo 7y ago
Delphi also went through a similar transition from "strings are in whatever the local code page says" with one byte chars to Unicode strings (Windows-style). However the makers of Delphi spent many years preparing for this, so when the time came for us to switch we only had to spend half a day or so to migrate our half a million lines of code.

vmchale 7y ago

> All of this was the heroic effort of one of my coworkers who had the unenviable task of combing through our entire codebase to determine "This is unicode. This is bytes. Dynamic typing!

war1025 7y ago
Not dynamic typing's fault. The issue is they changed the types out from underneath you. And then left it to each library to decide which type it was actually going to accept.
- magicalhippo 7y ago
  But with static typing, the compiler can let you know when you're doing something wrong with the new Unicode-based string.
  - war1025 7y ago
    Well, it can, except you then need to go through and update all of your internal APIs to be correct. Really the string transition was just a poor choice in my opinion. Python2 already had unicode strings that were easy enough to specify (just prefix with a `u`). It would have been better to just delineate that barrier better from an API standpoint. I understand the appeal of having unicode for the default string literal type, but it was actively hostile to existing projects.
    - lmm 7y ago
      > Well, it can, except you then need to go through and update all of your internal APIs to be correct. You do, but it's easy: run a compile, fix the errors, repeat until no more errors. > It would have been better to just delineate that barrier better from an API standpoint. Isn't that exactly what the Python 3 transition was? i.e. stop accepting non-unicode "strings" (actually just arbitrary byte sequences) for APIs that semantically require a string, reserve them for APIs that actually want a byte sequence.
      - war1025 7y ago
        > You do, but it's easy: run a compile, fix the errors, repeat until no more errors. The reason this doesn't work is that previously the double-quote literal was a "string" type. The string type was, yes, just a sequence of bytes, but in an ascii-centric world that also mean text. Python2 added unicode string literals that accepted unicode code points. Most APIs were happy to sloppily mix the two and generally work quite adequately. Python3 then made the hard distinction between byte-string and unicode-string. Not an unreasonable position to take on the face of it. The issue is many python2 APIs were written from the perspective of "accepts string literal types", where that could be either bytestring or unicode string. Now suppose you have a large codebase in python that spans the entire stack from database interaction, to webserver, to desktop application. All built on double quoted string literals. Accepting unicode strings in the places that needed that (user-facing places mainly, utf-8 bytestrings anywhere being stored on disk or sent over network) Then you go to switch to python3, and suddenly all of your string literals are interpreted as unicode instead of bytestring / ascii sequences. So now you need to go through every place in your codebase that accepts strings as an argument and determine, "is this a user-facing string, or a utf-8 bytestring", because they used to be basically the same thing, and now they aren't. It's not "difficult" really, it's just a pain in the neck.
        lmm 7y ago
        None of that would be a problem in a typed language. The ultimate destination of any string literal is some standard library function, whether that's write to network socket, display to user, or something else. So you just ripple backwards from that through your own functions that are calling those standard library functions, until you get to the point where you're passing in the literal, and then you know what kind of literal it needs to be.

d0mine 7y ago

Something is wrong if there is no third type: the "natural" string (bytes on Python 2, unicode on Python 3).

war1025 7y ago
I assume many of the strings were left untouched. But you still have to audit all of it to know which needs to be used where.
eesmith 7y ago
I believe that's included in the "etc."
quietbritishjim 7y ago
Surely any "natural" string would be better represented as unicode in Python 2? What is an example that wouldn't be?
- war1025 7y ago
  I believe what they meant was that for many strings it really shouldn't matter if they were bytes or unicode. They would perform their function correctly either way. That's completely true, but you do still have to go through and find the cases where that doesn't work.
  - deleted 7y ago
    [deleted]
- masklinn 7y ago
  > Surely any "natural" string would be better represented as unicode in Python 2? No because much of the stdlib works in terms of native strings and will choke (or worse silently fuck up) on the other. Yes also in python 2, the stdlib was absolutely not “unicode clean”. So a transitional / polyglot codebase has and needs not 2 but 3 string types: bytes, unicode, and native. And neither “unicode literals” nor “bytes literals” were good things to apply across the board.
- 7y ago

magicalhippo 7y ago

Delphi also went through a similar transition from "strings are in whatever the local code page says" with one byte chars to Unicode strings (Windows-style). However the makers of Delphi spent many years preparing for this, so when the time came for us to switch we only had to spend half a day or so to migrate our half a million lines of code.