6 ms·

The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble th

The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. My personal experience with compile time type checking is that the benefits kick in for a single developer project within a week, and for a multi-developer project immediately.

tus88 7y ago
How are you sure typing would have made a difference to that piece of Python? Are you sure there is no troublesome code written anywhere in Java or C++?
- swiley 7y ago
  IMO type checking is to python what bounds checking is to C.
  - speedplane 7y ago
    > IMO type checking is to python what bounds checking is to C. Interesting analogy, but in my experience it doesn't hold up. If you have a Python typing bug, it will crash your program almost immediately and you can quickly catch it and fix it. In contrast, C boundary checking issues often come up in relatively rare corner cases, and when they do occur, they often do not result in an exception / error especially if the memory is already paged and set to zero. In production code, boundary checking errors in C occur all the time, but Python type bugs are pretty rare.
- justwalt 7y ago
  I think he's talking specifically about the Java and C++ code that accompanied that Python file. I'm sure he's seen bad Java and C++ :)
  - jdm2212 7y ago
    I've seen some disgusting Java and C++, but that company had pretty high engineering standards so the Java and C++ was always good. The Python was _supposed_ to be good, too, but 20+ devs making changes to this one config generator script meant people were constantly making their own (and breaking each other's) assumptions about how the code should behave.
    - proverbialbunny 7y ago
      At that point the first person to write good unit tests gets to specify how that part of the program is going to run. Though, really it sounds like you guys need a design pattern to address everyone's needs that everyone can agree to. People can change unit tests. The benefit of this, is it's a kind of documentation. People can see the tests and know how it _should_ work, or they can not read them and if they create errors, they'll find out the hard way. Unit tests don't make a specification static, but they do reduce chaos.
      - jdm2212 7y ago
        This was a few years ago. The chaos was partly because of a lack of ownership: one team owned inputs, the other owned outputs, and there wasn't a clear spec. The whole pipeline was pretty experimental and people made up new semantics as needed, and were happy to change failing tests (or just added untested codepaths behind boolean flags in existing functions). Compile-time type checking would've enforced a greater degree of rigor about modeling what was happening and made clear all the ways existing code could be affected.
dbcurtis 7y ago
I am not going to dispute the value of compile-time type checking. And I admit to only having had time to skim the link at this point but it looks like a great read so I bookmarked it for later. That said... Python can be as strongly typed as you like it to be. But of course, with Python, instead of insisting on an incoming parameter being of a particular type, the thing to do is Pythonically ask nicely if it can be the type that you want. If you need parameter x to be a float, call float(x), which will raise TypeError or give you a float. And of course, it gives the implementor of the class the option to implement __float__(). With a small amount of effort, you can design all of your __init__() constructors to do that style of validation. Yes, it requires some elbow grease. But you will not be able to pass bad stuff to my __init__ constructors if I don't want you to. And if you want some parameter y to be an instance of Foo, just call Foo(y) and if it is already a Foo, you get the Foo, and if it can be made a Foo, you get a Foo, and if it can't, you get TypeError. So of course you actually need to exercise the code also, which is where the hypothesis package comes in. Anyway, I think one of the reasons Python is popular is that it is very consistent about types. Haskel and Rust are also very consistent about types -- most other languages, not so much. Give me one end of the spectrum or the other. The murky middle is suboptimal.
- joshuamorton 7y ago
  This doesn't work as well as you'd hope. It requires among other things, a unique and toilsome coding style, constructor overloading (which isn't a first class citizen of python), and you only get late runtime validation (which means, for example, you can easily miss type validation on only one side of a conditional). Something like pytype detects such errors without any code modifications.
  - dbcurtis 7y ago
    Most of what you say I agree with. But it is actually very easy to construct a __init__ that handles a very flexible parameter list. That style is well represented in the standard library. And I find that writing a beefy constructor simplifies parameter validation in methods. But certainly, it doesn’t remove the need for good test coverage. Now, if you want to argue for less testing, that is a different pitch.
realusername 7y ago
> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. Another problem with Python is that the culture of testing is definitely not as strong as in Ruby so you won't even get that to compensate the untyped nature of the language.
- btown 7y ago
  And yet another problem is that while Ruby classes require you to whitelist attributes (e.g. with attr_accessor) that will receive publicly-accessible getters/setters, Python classes by default allow the caller to "tack on" any instance variables they desire. (I've rarely seen __slots__ used in the wild, for instance.) So the type of almost any class is really more like Struct{WhatYouThinkTheClassIs; ArbitraryDict;} ... and the temptation to misuse this, say for instance if you don't want to add arguments to function_used_lots_of_places(some_object), can be huge. And more generally, this openness means that there may be discrepancies between what's unit tested for a class, and how that class is used in the wild.
lawn 7y ago
At our job we use a homebrew C++-like language. Now some people are planning to rewrite a core part of the problem to be completely dynamic, only using key-value to control the behavior of a large number of things. There is no namespaces and the value is an Object (base class for all classes), so you always have to cast. The absolute horror it will be to debug this mess when people starts to use it, my god.
siempreb 7y ago
> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. That cannot be because of missing types. If you make just 500 lines python unmaintainable you're just a terrible programmer! In it's core it cannot be a type problem. And that is my main point here; there is a hype going on in the dynamic languages to add types, as if it solves everything. IMHO it's not that simple. To avoid unnecessary explanation about the value of static typing; I am not against, I'm doing c++ and it serves me well.
- jdm2212 7y ago
  The code in question was somewhat experimental, didn't have a clear owner and was modified by a number of people fairly frequently. There would've been much less room for sloppiness and shortcuts (let me just stick this extra field on this object) in Java, C++ or Go.
wutbrodo 7y ago
I'm still astonished that large companies run large systems on Python. I don't doubt that they're making the right decision, but it's a testament to how powerful inertia in large engineering systems is,given how incredibly horrible Python is for writing maintainable, sustainable multi-developer code. The most unfortunate part is that every engineer on a Python system has to be much more talented and disciplined to keep a workable system than they have to for statically typed languages (esp memory-safe ones), and yet one of the main attractions of choosing Python is chosen is that any rando can be nominally productive in it, which is helpful for talent-starved small companies.
- soulnothing 7y ago
  The early portion of my career was python. Coming from OCaml/Haskell. I couldn't really deal with J2EE. But I've seen a frustrating trend recently. Where companies on languages like C#/Java. With a massive code base are moving to Python. I often get recruiter calls to lead a new python team, transitioning away from (Java/C#/etc). The sole reason, is it takes to long to develop a feature in the legacy language. Which is resultant of outdated software and practices. I generally try and steer them towards CI/CD and newer lighter frame works. But someone way up heard python was the hot new thing. Now all critical back-end services must be in python. To add to this I pushed for type hinting. But type hinting will "slow down developers". Part of the problem to me, at least. Is the person dictating development frameworks, languages, etc. Isn't really a coder, and heard about thing X at some conference. I like python, but it's an infrastructure quick scripting language. Additionally a lot of the good python devs I know shifted over to tensor flow and machine learning.
segmondy 7y ago
Why didn't you just rewrite it? If I have 500 lines that is a mess. It's getting rewritten. If you don't understand it enough to rewrite it, then you probably don't understand it enough to even maintain or change it. In my opinion, first thing I will do is dissect it till I understand it, rewrite it and move on. 500 lines of code ain't shit. I will kill the monster before it grows.
ken 7y ago
Counterpoint: I've worked on a production >100-KLOC >10-programmer Python system, and lack of types was not an issue at all. For the past week I've been updating a (much shorter) solo app, written in a statically-typed language, and having a bear of a time getting the compiler to stop throwing false type errors at me. Yes, it's the stable release version of the compiler. That has never stopped any compiler from shipping with bugs. These type inference systems can be pretty complex. Then again, I seem to run into more compiler bugs in general than anyone I know. How do you all do it?

tus88 7y ago

How are you sure typing would have made a difference to that piece of Python? Are you sure there is no troublesome code written anywhere in Java or C++?

swiley 7y ago
IMO type checking is to python what bounds checking is to C.
- speedplane 7y ago
  > IMO type checking is to python what bounds checking is to C. Interesting analogy, but in my experience it doesn't hold up. If you have a Python typing bug, it will crash your program almost immediately and you can quickly catch it and fix it. In contrast, C boundary checking issues often come up in relatively rare corner cases, and when they do occur, they often do not result in an exception / error especially if the memory is already paged and set to zero. In production code, boundary checking errors in C occur all the time, but Python type bugs are pretty rare.
justwalt 7y ago
I think he's talking specifically about the Java and C++ code that accompanied that Python file. I'm sure he's seen bad Java and C++ :)
- jdm2212 7y ago
  I've seen some disgusting Java and C++, but that company had pretty high engineering standards so the Java and C++ was always good. The Python was _supposed_ to be good, too, but 20+ devs making changes to this one config generator script meant people were constantly making their own (and breaking each other's) assumptions about how the code should behave.
  - proverbialbunny 7y ago
    At that point the first person to write good unit tests gets to specify how that part of the program is going to run. Though, really it sounds like you guys need a design pattern to address everyone's needs that everyone can agree to. People can change unit tests. The benefit of this, is it's a kind of documentation. People can see the tests and know how it _should_ work, or they can not read them and if they create errors, they'll find out the hard way. Unit tests don't make a specification static, but they do reduce chaos.
    - jdm2212 7y ago
      This was a few years ago. The chaos was partly because of a lack of ownership: one team owned inputs, the other owned outputs, and there wasn't a clear spec. The whole pipeline was pretty experimental and people made up new semantics as needed, and were happy to change failing tests (or just added untested codepaths behind boolean flags in existing functions). Compile-time type checking would've enforced a greater degree of rigor about modeling what was happening and made clear all the ways existing code could be affected.

dbcurtis 7y ago

I am not going to dispute the value of compile-time type checking. And I admit to only having had time to skim the link at this point but it looks like a great read so I bookmarked it for later. That said... Python can be as strongly typed as you like it to be. But of course, with Python, instead of insisting on an incoming parameter being of a particular type, the thing to do is Pythonically ask nicely if it can be the type that you want. If you need parameter x to be a float, call float(x), which will raise TypeError or give you a float. And of course, it gives the implementor of the class the option to implement __float__(). With a small amount of effort, you can design all of your __init__() constructors to do that style of validation. Yes, it requires some elbow grease. But you will not be able to pass bad stuff to my __init__ constructors if I don't want you to. And if you want some parameter y to be an instance of Foo, just call Foo(y) and if it is already a Foo, you get the Foo, and if it can be made a Foo, you get a Foo, and if it can't, you get TypeError. So of course you actually need to exercise the code also, which is where the hypothesis package comes in. Anyway, I think one of the reasons Python is popular is that it is very consistent about types. Haskel and Rust are also very consistent about types -- most other languages, not so much. Give me one end of the spectrum or the other. The murky middle is suboptimal.

joshuamorton 7y ago
This doesn't work as well as you'd hope. It requires among other things, a unique and toilsome coding style, constructor overloading (which isn't a first class citizen of python), and you only get late runtime validation (which means, for example, you can easily miss type validation on only one side of a conditional). Something like pytype detects such errors without any code modifications.
- dbcurtis 7y ago
  Most of what you say I agree with. But it is actually very easy to construct a __init__ that handles a very flexible parameter list. That style is well represented in the standard library. And I find that writing a beefy constructor simplifies parameter validation in methods. But certainly, it doesn’t remove the need for good test coverage. Now, if you want to argue for less testing, that is a different pitch.

realusername 7y ago

> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. Another problem with Python is that the culture of testing is definitely not as strong as in Ruby so you won't even get that to compensate the untyped nature of the language.

btown 7y ago
And yet another problem is that while Ruby classes require you to whitelist attributes (e.g. with attr_accessor) that will receive publicly-accessible getters/setters, Python classes by default allow the caller to "tack on" any instance variables they desire. (I've rarely seen __slots__ used in the wild, for instance.) So the type of almost any class is really more like Struct{WhatYouThinkTheClassIs; ArbitraryDict;} ... and the temptation to misuse this, say for instance if you don't want to add arguments to function_used_lots_of_places(some_object), can be huge. And more generally, this openness means that there may be discrepancies between what's unit tested for a class, and how that class is used in the wild.

lawn 7y ago

At our job we use a homebrew C++-like language. Now some people are planning to rewrite a core part of the problem to be completely dynamic, only using key-value to control the behavior of a large number of things. There is no namespaces and the value is an Object (base class for all classes), so you always have to cast. The absolute horror it will be to debug this mess when people starts to use it, my god.

siempreb 7y ago

> The worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. That cannot be because of missing types. If you make just 500 lines python unmaintainable you're just a terrible programmer! In it's core it cannot be a type problem. And that is my main point here; there is a hype going on in the dynamic languages to add types, as if it solves everything. IMHO it's not that simple. To avoid unnecessary explanation about the value of static typing; I am not against, I'm doing c++ and it serves me well.

jdm2212 7y ago
The code in question was somewhat experimental, didn't have a clear owner and was modified by a number of people fairly frequently. There would've been much less room for sloppiness and shortcuts (let me just stick this extra field on this object) in Java, C++ or Go.

wutbrodo 7y ago

I'm still astonished that large companies run large systems on Python. I don't doubt that they're making the right decision, but it's a testament to how powerful inertia in large engineering systems is,given how incredibly horrible Python is for writing maintainable, sustainable multi-developer code. The most unfortunate part is that every engineer on a Python system has to be much more talented and disciplined to keep a workable system than they have to for statically typed languages (esp memory-safe ones), and yet one of the main attractions of choosing Python is chosen is that any rando can be nominally productive in it, which is helpful for talent-starved small companies.

soulnothing 7y ago
The early portion of my career was python. Coming from OCaml/Haskell. I couldn't really deal with J2EE. But I've seen a frustrating trend recently. Where companies on languages like C#/Java. With a massive code base are moving to Python. I often get recruiter calls to lead a new python team, transitioning away from (Java/C#/etc). The sole reason, is it takes to long to develop a feature in the legacy language. Which is resultant of outdated software and practices. I generally try and steer them towards CI/CD and newer lighter frame works. But someone way up heard python was the hot new thing. Now all critical back-end services must be in python. To add to this I pushed for type hinting. But type hinting will "slow down developers". Part of the problem to me, at least. Is the person dictating development frameworks, languages, etc. Isn't really a coder, and heard about thing X at some conference. I like python, but it's an infrastructure quick scripting language. Additionally a lot of the good python devs I know shifted over to tensor flow and machine learning.

segmondy 7y ago

Why didn't you just rewrite it? If I have 500 lines that is a mess. It's getting rewritten. If you don't understand it enough to rewrite it, then you probably don't understand it enough to even maintain or change it. In my opinion, first thing I will do is dissect it till I understand it, rewrite it and move on. 500 lines of code ain't shit. I will kill the monster before it grows.

ken 7y ago

Counterpoint: I've worked on a production >100-KLOC >10-programmer Python system, and lack of types was not an issue at all. For the past week I've been updating a (much shorter) solo app, written in a statically-typed language, and having a bear of a time getting the compiler to stop throwing false type errors at me. Yes, it's the stable release version of the compiler. That has never stopped any compiler from shipping with bugs. These type inference systems can be pretty complex. Then again, I seem to run into more compiler bugs in general than anyone I know. How do you all do it?