3 ms·

I agree that code quality is a big factor. But what is good code quality in Python? In our case, the oldest code is the most "pythonic" and is at the same time

I agree that code quality is a big factor. But what is good code quality in Python? In our case, the oldest code is the most "pythonic" and is at the same time the worst to maintain. The better code mitigates the drawbacks of dynamic typing and by that moves away from the pythonic standard you see in many libraries. But even if you nail the types to the board (e.g. assert isinstance(...)), use (the somewhat weak) Mypy wherever you can, and have good test coverage, you still have to grep your code base for usage of, e.g., .keys(), eyeball hundreds of modules for subtle Unicode madness or hunt for the odd division, replace every sort() that doesn't use key= yet, etc. The todos add up and someone has to go into the code and change those lines.

StavrosK 7y ago
Of the things you mentioned, only Unicode has been a problem, and that's exactly what 3 is fixing, so that's to be expected. The rest was automatically handled by 2to3 with only a cursory review.
- AnonymousPlanet 7y ago
  We tried 2to3 and it gave us poor results. But probably because we deviated too far from being pythonic. The str/unicode misery is one of the biggest gripes I have with Python. I'm glad this unpleasant knot has been mostly untied in Python 3. I came to the conclusion that the transition would have been much easier if Python 3 just concentrated on the separation of bytes and (unicode) strings. The other features could have been in Python 4. It's a bit like IPv6. If it would just solve the address space problem, most would have moved to it already. Instead it comes with a lot more baggage. And each additional feature has it's own uphill battle for acceptance. So nearly everyone is dragging their feet, citing their pet peeve with the technology.
  - StavrosK 7y ago
    I'm not sure that's true because, as I said, most of it is automatic and worked well with 2to3, leaving us to deal pretty much only with Unicode. I'd certainly prefer to only have to do this upgrade once.
    - AnonymousPlanet 7y ago
      How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper, but it makes the code more convoluted instead of transforming it to the key-paradigm. And if you want to sort by several keys, now you will have to call sort several times. How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator? Those are just off the top of my head. I don't remember all the cases where 2to3 tripped over and produced garbage. But there were too many cases to put actual faith in automatic conversion. It might work if your code is kind of new and homogenic. But looking at how much trouble Dropbox had, even with all the tooling and Guidos they could muster, I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.
      - StavrosK 7y ago
        > How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper Yes, there's a wrapper. You use it, add a comment "this is wrapped in the migration to 3" and move on. > How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator? If it's being iterated on first thing, it's an iterator. If list methods are called on to it, it's a list. This just hasn't been a problem for us, sure, it took some looking at, but it wasn't more than 30 seconds per case. > I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases. Maybe so, but the codebase was ten years old and hundreds of thousands of lines.
stult 7y ago
What specific “pythonic” habits have you found are more difficult to maintain? Asking out of genuine curiosity, not to challenge the premise. I work with a lot of data science people that really emphasize being pythonic, but coming from the software/static typing side of the house I always find their code style and architecture a little concerning, and I’m not sure if I’m just not getting it or if they really are writing spaghetti.
- AnonymousPlanet 7y ago
  One of the biggest footguns is method naming. Most Python libraries will gladly use generic method names like "add()" or "getName()". The moment you need to rename the method or change the signature, you will have a hard time telling it apart from all the other method calls by the same name. No type inference will save you here because type inference is incomplete and will never let you find all the callers. What you should do is use unique names. But that will give you ugly code like myFoobar.foobar_addBar(). The kind of code that makes the pythonic crowd cringe. Another problem is making code too generic with regards to what types it consumes, instead of nailing it down to the few types you're ever going to use here. This makes it hard to reason about your code months and years down the line. How is this method used in the rest of the code base? Do all callers expect an int? What if my method now happens to return a float? And there's also abuse of duck typing. Throw around a lot of objects, sprinkling methods and other members on to them as you go. Then when you consume the object, just look if it has the method you want to call. This makes any kind of type checking and static type inference useless. And then there's a whole lot of Python 2 libraries where you get the feeling that the authors didn't give too much thought about whether they are dealing with str or unicode. The method might just call .encode(...) on one of its arguments without being too sure what it is. And every one of the mistakes that result from the above practices might only pop up when your code has already been shipped to the customer site.

StavrosK 7y ago

Of the things you mentioned, only Unicode has been a problem, and that's exactly what 3 is fixing, so that's to be expected. The rest was automatically handled by 2to3 with only a cursory review.

AnonymousPlanet 7y ago
We tried 2to3 and it gave us poor results. But probably because we deviated too far from being pythonic. The str/unicode misery is one of the biggest gripes I have with Python. I'm glad this unpleasant knot has been mostly untied in Python 3. I came to the conclusion that the transition would have been much easier if Python 3 just concentrated on the separation of bytes and (unicode) strings. The other features could have been in Python 4. It's a bit like IPv6. If it would just solve the address space problem, most would have moved to it already. Instead it comes with a lot more baggage. And each additional feature has it's own uphill battle for acceptance. So nearly everyone is dragging their feet, citing their pet peeve with the technology.
- StavrosK 7y ago
  I'm not sure that's true because, as I said, most of it is automatic and worked well with 2to3, leaving us to deal pretty much only with Unicode. I'd certainly prefer to only have to do this upgrade once.
  - AnonymousPlanet 7y ago
    How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper, but it makes the code more convoluted instead of transforming it to the key-paradigm. And if you want to sort by several keys, now you will have to call sort several times. How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator? Those are just off the top of my head. I don't remember all the cases where 2to3 tripped over and produced garbage. But there were too many cases to put actual faith in automatic conversion. It might work if your code is kind of new and homogenic. But looking at how much trouble Dropbox had, even with all the tooling and Guidos they could muster, I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.
    - StavrosK 7y ago
      > How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper Yes, there's a wrapper. You use it, add a comment "this is wrapped in the migration to 3" and move on. > How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator? If it's being iterated on first thing, it's an iterator. If list methods are called on to it, it's a list. This just hasn't been a problem for us, sure, it took some looking at, but it wasn't more than 30 seconds per case. > I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases. Maybe so, but the codebase was ten years old and hundreds of thousands of lines.

stult 7y ago

What specific “pythonic” habits have you found are more difficult to maintain? Asking out of genuine curiosity, not to challenge the premise. I work with a lot of data science people that really emphasize being pythonic, but coming from the software/static typing side of the house I always find their code style and architecture a little concerning, and I’m not sure if I’m just not getting it or if they really are writing spaghetti.

AnonymousPlanet 7y ago
One of the biggest footguns is method naming. Most Python libraries will gladly use generic method names like "add()" or "getName()". The moment you need to rename the method or change the signature, you will have a hard time telling it apart from all the other method calls by the same name. No type inference will save you here because type inference is incomplete and will never let you find all the callers. What you should do is use unique names. But that will give you ugly code like myFoobar.foobar_addBar(). The kind of code that makes the pythonic crowd cringe. Another problem is making code too generic with regards to what types it consumes, instead of nailing it down to the few types you're ever going to use here. This makes it hard to reason about your code months and years down the line. How is this method used in the rest of the code base? Do all callers expect an int? What if my method now happens to return a float? And there's also abuse of duck typing. Throw around a lot of objects, sprinkling methods and other members on to them as you go. Then when you consume the object, just look if it has the method you want to call. This makes any kind of type checking and static type inference useless. And then there's a whole lot of Python 2 libraries where you get the feeling that the authors didn't give too much thought about whether they are dealing with str or unicode. The method might just call .encode(...) on one of its arguments without being too sure what it is. And every one of the mistakes that result from the above practices might only pop up when your code has already been shipped to the customer site.