7 ms·

My current role is at a company with a few hundred thousand lines of un-typed, undocumented, mostly un-tested Python. I get a small allowance of "tech debt bust

My current role is at a company with a few hundred thousand lines of un-typed, undocumented, mostly un-tested Python. I get a small allowance of "tech debt busting" time, and it's absurd how many bugs I find just by trying to suss out the types on a given code path. It's maddening work.

l0b0 3y ago
Bringing a non-trivial code base up to even half-decent standards can be a monumental task, and because such code bases are inherently overly complex and risky to change there's only so much you can do without breaking production constantly. It's a lose-lose proposition, which is why I'd never want to be "the programmer" at some place like that, no matter the pay.
synergy20 3y ago
django has 500K lines of python and it's considered well written and stable, so it's possible to do python with large code base I think. oddly, django does not seem to be even using type hints, new features such as dataclasses was not found from the source base either, still it just works well somehow.
- dsissitka 3y ago
  I was curious: SLOC Directory SLOC-by-Language (Sorted) 232099 tests python=232099 102470 django python=102470 424 docs python=424 158 scripts python=142,sh=16 21 top_dir python=21 0 extras (none) 0 js_tests (none) Totals grouped by language (dominant language first): python: 335156 (100.00%) sh: 16 (0.00%) Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."
  - deepsun 3y ago
    Wow, 2.3x for tests/src ratio. I remember some recommendation for Java to have around 0.9x ratio, otherwise too much tests maintenance. Depends on the project of course, e.g. SQLite has an insane ratio.
    - cies 3y ago
      Could also be that Java's typing already accounts for some part of what would need to be tests in Python code...
      - deepsun 3y ago
        Exactly, that's what I meant. E.g. if very_rare_condition: raise MyRareException Java wouldn't compile (no exception instantiation), but Python would work fine in production until the rare condition happened (but not captured above, as it's a class, not instance). Cases like that justify 100% tests coverage in Python, but not Java.
- simonw 3y ago
  That's because a great deal of Django was written before type hints or even dataclasses existed.
  - Hamuko 3y ago
    And type hints aren't really that mature yet. If you look at the `typing` module's documentation in Python 3.11, there's 23 occurrences of the string "Changed in version". And the Python 3.12 version of the same document adds 9 more. Type hinting in Python is constantly evolving.
    - walthamstow 3y ago
      That's the main reason we haven't adopted it at my place. We do have types of args and returns in docstrings though and it is enough for IDEs to do checks and suggestions from those types.
jeroenhd 3y ago
My experience of porting Javascript code to Typescript (or calling into Javascript from Typescript and figuring out the correct types to pass) is very similar. Thousands of files all become filled with very obvious bugs once you enable a basic type checker, which then helps brand Typescript as "too much effort" by those who normally write Javascript because now they need to deal with their buggy code. Luckily you can just explicitly cast all of your types without checking and hope that nobody notices! When you ask people to get their typing right, someone will use buzzwords like "rapid prototyping" and remind you that features are more important than fixing the existing product as long as nobody complains about the bugs, and that's that. Also, Typescript is hard, apparently. Untyped legacy code sucks. It's one of the reasons I detest working with languages like Javascript, Python, and PHP. At least Python and PHP have types these days, making Javascript objectively inferior.
- valzam 3y ago
  "having type annotations" and "having types" are two very different things. The people who say TS is too hard/not worth are most definitely not adding type annotations to Python/PHP code.
- wirrbel 3y ago
  The other way around you have the "it compiles, so I don't need tests" crowd in statically-typed languages.
  - erik_seaberg 3y ago
    This might sort of work if everything from the domain gets a disjoint type, but stringly-typed code is a lot more common and the type system doesn’t know what would make sense.
  - HdS84 3y ago
    I am currently firefighting for a customer who has a medium Sized application in c#. Started around 2019 so not ancient, but every programming, architecture and technology choice was wrong. Sure it compiles. But often it crashes without rhyme or reason and for some reason, customers hate it. First thing I did was to get it from .net 4.8 to .net 7 and enable all available errors checkers and style linters. The other howled, but their builds stopped because they could no longer check-in crap. That helped. But fixing this is more work than what this would have needed if they had done it right the first time. Funny examples: reinvanted rabbitmw badly using raw TCP and XML. Better hope your network is good, If the messages arrive at the same time, they are merged and the app crashes. So there are lots of time.sleep(50) calls to avoid that. Why is that app slow? The iot devices can only address 254 states. The UI for that allows ulong devices. It's never checked, but if the vast fails, the app just crashes.
Turskarama 3y ago
This is why I can't help but mentally roll my eyes when people say that static typing makes programming slower. It makes writing the happy path slower, but is writing the happy path _programming?_ By the time you have a program which is complete and (at least mostly) bug free, the static typing has saved you hundreds of hours.
- l0b0 3y ago
  > It makes writing the happy path slower, but is writing the happy path _programming?_ Most managers and even some programmers seem to think so. Build the thing with no quality control, ship it, deal with the fallout, repeat. Bad programmers think this is normal, good programmers ensure that any fallout is rare.
  - smif 3y ago
    I don't know if it's necessarily bad programmers. Think about this analogy, there's a blacksmith that can make swords but if you want it done well, it's gonna take more time/money. That same blacksmith can choose to take on customers that insist that he make the sword quicker and cheaper (and probably shittier). He's not gonna be the one using the sword and he can make it clear up front about the tradeoffs. If it shatters in use and the customer gets maimed, oh well, they were warned. Sometimes the market is flooded with people who just want cheap swords made fast and the blacksmith needs to pay the bills too right?
    - MereInterest 3y ago
      One problem with this mentality is that customers need to search more and more to find blacksmiths that provide anything other than cheap swords. They may reasonably conclude that this is just the nature of swords, and that there’s no such thing as a sword that withstands more than one or two blows. Eventually, even people who want good swords accept that they won’t be able to find a blacksmith who makes them, and settle for cheap swords instead. In effect, https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons
s6ro 3y ago
Ah memories... I once had 100K Python 2.x codebase to maintain. No docs, no tests and then being required to add features. Had to use the debugger[1] to make sense of the codebase and types, formats etc. Sheer madness and such a waste of time... To this day, I'm baffled by the dynamic language folks who cannot get their head around how strictness/rigor (via a good expressive type system) actually makes maintenance easier and more importantly: cheaper. [1] https://github.com/inducer/pudb https://github.com/inducer/pudb
john61 3y ago
It is no problem to write buggy unmaintainable code in a strongly typed language. It is also perfectly possible to write clean and maintainable code in a duck type language.
bronxpockfabz 3y ago
https://github.com/beartype/beartype https://github.com/beartype/beartype I wish more people started using Beartype, it makes Python bearable
BiteCode_dev 3y ago
That's more of a cultural thing. Because Python is easy to get into, and you can be productive very fast, a lot of people with little experience in programming start to build things that are out of their reach. They just don't know it, and the people hiring them don't either. It's not a new phenomenon, we got the same when Java arrived, then PHP, etc. Every time something makes things easier, you get a wave of beginners set to create things they are not qualified to do. And because you now need software for everything, leaders also get into positions where are not qualify to make some decisions about it. You pair unqualified leaders with unqualified devs, and you get this. And it's now easier than ever to be in this position, also you are more pressured than ever to get into this position. I've worked on plenty of big Python systems myself, it's not the tech per se (it has the usual pros and cons game like every language), but Python is the language used "when you don't know programming". Which is ironic, because when I started to use it in Python 2.4, only passionate devs used it because it was niche, and we had the opposite effect.
- pjmlp 3y ago
  It is BASIC all over again.
- st4lz 3y ago
  > Every time something makes things easier, you get a wave of beginners set to create things they are not qualified to do. And because you now need software for everything, leaders also get into positions where are not qualify to make some decisions about it. You pair unqualified leaders with unqualified devs, and you get this. How easy it is to call other people not qualified because they have different opinions. I've seen more projects failed because the only thing the devs cared about was an abstract code correctness, which made even basic PRs getting polished for weeks. I don't think you need to worry too much about types or tests if you are just doing a prototoype to validate ideas or your user base is 0.
BoppreH 3y ago
I've always wondered if we shouldn't be pushing more for type inference with flow analysis. I consider static type checking a must, but the voices opposing type annotations have some valid points: it pollutes the source code, slows down prototyping, and only catches a limited class of errors. A static checker with global type inference could be a middle ground. Pylance/Pyright already does that for most VS Code users. A more advanced form of this could track types that are too tedious for humans, like list emptiness, bounds, string alphabet, functions that can raise exceptions, idempotence, side effects, etc. Those types can then be shown in the IDE, according to user preference, like docs or source control metadata. I have a feeling that with enough caching this could be done with ok performance, but I hear that useful error reporting is an unsolved problem.
classified 3y ago
So much for all those "you don't need static typing" / "dynamic typing is better" arguments.
flohofwoe 3y ago
Are the 'bugs' actually triggered in the real-world usage of the code, or are they just 'potential bugs' which would be triggered given specific inputs that don't happen in the current usage of the code? Might seem like nitpicking, but the 'probability of a bug being triggered in the future' is an important metric for figuring out if large scale refactorings are actually worth the risk of introducing new bugs. Disclaimer: I'm coming from the static typing hemisphere, but adding type information to a codebase that was originally written with runtime-duck-typing in mind sounds like it might actually make maintainability of the code worse - that's at least my experience after trying to do exactly this in a medium-sized Python codebase - some of the 'typings' ended up so complex and weird that they completely dominated the code and turned what was very simple code at the beginning into an unreadable mess (looking almost like C++ template code). I eventually rolled back the changes. Basically, if I wanted a statically typed language, I wouldn't have chosen Python in the first place for this type of project.
- eyko 3y ago
  Can't speak for the parent commenter but their situation felt familiar. I'm in a new role where I'm contributing to a very large Python codebase (first commit from over a decade ago) and whilst there's a lot of more recent code written with type hints, they're not enforced and they're not always reliable or consistent. On my second contribution, I introduced a bug that pyright immediately spotted after I upped the checks from basic to strict. Our Sentry reports a lot of non-critical bugs that are being triggered in production that in many cases are _already_ highlighted by the type checker. It's just a big enough codebase that it's not practical to attempt to fix them all. If I were starting a project today in Python, I would definitely try setting up my tooling and processes to enforce type checking to at least some extent. Fortunately, we have a lot of guardrails that recover from the typical exceptions that occur in those cases, but as I experienced first hand, it's very easy to introduce new bugs when relying only on my intuition for those difficult-to-spot cases. Tooling definitely helps there. My previous role had me writing TypeScript and Rust so there's always the temptation, or bias, to advocate for strict typing everywhere, but I'm conscious that it's neither practical nor feasible in large codebases with hundreds of contributors (most of whom may be more accustomed to dynamically typed languages).
crabbone 3y ago
Just imagine how much more bugs you add... Also, imagine how much more productive you would've been if you instead of doing what you are doing you'd do something worthwhile, like, design the program or organize the testing better. I've heard so many nonsense claims like yours both in proprietary projects and in open-source ones, and not a single time did the code base became better. In most cases a small(-er) pile of garbage became a bigger pile of garbage.
keithalewis 3y ago
Are you implying people writing code should have some training in how computers actually work? Heretic!
bilsbie 3y ago
> trying to suss out the types on a given code path. Are there any tools for that? If not could one be made? Sounds almost like an advanced linter.
bilsbie 3y ago
That sounds impressive but I feel like this might be a misunderstanding of Python. You can break almost any python code by passing in the wrong type. The code throwing runtime type-errors IS the type checking. The danger is only if the code happily operates on the wrong types and returns incorrect results.
cmrdporcupine 3y ago
It's funny how the pendulum swings. 10-15 years ago sticking your neck our and advocating for static types was like painting a target on your back. Everything was about dynamic languages, Ruby was the coolest of cool, and people who waved the placard of high level types, type inference, etc. from the ML/Haskell cool were seen on the whole as impractical academic ivory tower types, cloistered around Lambda The Ultimate... When I advocated for the importance of static typing for software robustness the response I got from many other industry people I knew was that this was the job of intensive unit testing. That the supposed loss of rapid turnaround that came with static types wasn't worth it, and that test-first-design solved this problem. Now Typescript and Rust are common place, and we have Pythonistas annotating their programs with types. I'm happy, but maybe a bit bitter about the past :-)
roflyear 3y ago
I work at a company that has a fucking nightmare of a C# project. It's total hell to work on. I don't think the language makes much of a difference.

l0b0 3y ago

Bringing a non-trivial code base up to even half-decent standards can be a monumental task, and because such code bases are inherently overly complex and risky to change there's only so much you can do without breaking production constantly. It's a lose-lose proposition, which is why I'd never want to be "the programmer" at some place like that, no matter the pay.

synergy20 3y ago

django has 500K lines of python and it's considered well written and stable, so it's possible to do python with large code base I think. oddly, django does not seem to be even using type hints, new features such as dataclasses was not found from the source base either, still it just works well somehow.

dsissitka 3y ago
I was curious: SLOC Directory SLOC-by-Language (Sorted) 232099 tests python=232099 102470 django python=102470 424 docs python=424 158 scripts python=142,sh=16 21 top_dir python=21 0 extras (none) 0 js_tests (none) Totals grouped by language (dominant language first): python: 335156 (100.00%) sh: 16 (0.00%) Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."
- deepsun 3y ago
  Wow, 2.3x for tests/src ratio. I remember some recommendation for Java to have around 0.9x ratio, otherwise too much tests maintenance. Depends on the project of course, e.g. SQLite has an insane ratio.
  - cies 3y ago
    Could also be that Java's typing already accounts for some part of what would need to be tests in Python code...
    - deepsun 3y ago
      Exactly, that's what I meant. E.g. if very_rare_condition: raise MyRareException Java wouldn't compile (no exception instantiation), but Python would work fine in production until the rare condition happened (but not captured above, as it's a class, not instance). Cases like that justify 100% tests coverage in Python, but not Java.
simonw 3y ago
That's because a great deal of Django was written before type hints or even dataclasses existed.
- Hamuko 3y ago
  And type hints aren't really that mature yet. If you look at the `typing` module's documentation in Python 3.11, there's 23 occurrences of the string "Changed in version". And the Python 3.12 version of the same document adds 9 more. Type hinting in Python is constantly evolving.
  - walthamstow 3y ago
    That's the main reason we haven't adopted it at my place. We do have types of args and returns in docstrings though and it is enough for IDEs to do checks and suggestions from those types.

jeroenhd 3y ago

My experience of porting Javascript code to Typescript (or calling into Javascript from Typescript and figuring out the correct types to pass) is very similar. Thousands of files all become filled with very obvious bugs once you enable a basic type checker, which then helps brand Typescript as "too much effort" by those who normally write Javascript because now they need to deal with their buggy code. Luckily you can just explicitly cast all of your types without checking and hope that nobody notices! When you ask people to get their typing right, someone will use buzzwords like "rapid prototyping" and remind you that features are more important than fixing the existing product as long as nobody complains about the bugs, and that's that. Also, Typescript is hard, apparently. Untyped legacy code sucks. It's one of the reasons I detest working with languages like Javascript, Python, and PHP. At least Python and PHP have types these days, making Javascript objectively inferior.

valzam 3y ago
"having type annotations" and "having types" are two very different things. The people who say TS is too hard/not worth are most definitely not adding type annotations to Python/PHP code.
wirrbel 3y ago
The other way around you have the "it compiles, so I don't need tests" crowd in statically-typed languages.
- erik_seaberg 3y ago
  This might sort of work if everything from the domain gets a disjoint type, but stringly-typed code is a lot more common and the type system doesn’t know what would make sense.
- HdS84 3y ago
  I am currently firefighting for a customer who has a medium Sized application in c#. Started around 2019 so not ancient, but every programming, architecture and technology choice was wrong. Sure it compiles. But often it crashes without rhyme or reason and for some reason, customers hate it. First thing I did was to get it from .net 4.8 to .net 7 and enable all available errors checkers and style linters. The other howled, but their builds stopped because they could no longer check-in crap. That helped. But fixing this is more work than what this would have needed if they had done it right the first time. Funny examples: reinvanted rabbitmw badly using raw TCP and XML. Better hope your network is good, If the messages arrive at the same time, they are merged and the app crashes. So there are lots of time.sleep(50) calls to avoid that. Why is that app slow? The iot devices can only address 254 states. The UI for that allows ulong devices. It's never checked, but if the vast fails, the app just crashes.

Turskarama 3y ago

This is why I can't help but mentally roll my eyes when people say that static typing makes programming slower. It makes writing the happy path slower, but is writing the happy path _programming?_ By the time you have a program which is complete and (at least mostly) bug free, the static typing has saved you hundreds of hours.

l0b0 3y ago
> It makes writing the happy path slower, but is writing the happy path _programming?_ Most managers and even some programmers seem to think so. Build the thing with no quality control, ship it, deal with the fallout, repeat. Bad programmers think this is normal, good programmers ensure that any fallout is rare.
- smif 3y ago
  I don't know if it's necessarily bad programmers. Think about this analogy, there's a blacksmith that can make swords but if you want it done well, it's gonna take more time/money. That same blacksmith can choose to take on customers that insist that he make the sword quicker and cheaper (and probably shittier). He's not gonna be the one using the sword and he can make it clear up front about the tradeoffs. If it shatters in use and the customer gets maimed, oh well, they were warned. Sometimes the market is flooded with people who just want cheap swords made fast and the blacksmith needs to pay the bills too right?
  - MereInterest 3y ago
    One problem with this mentality is that customers need to search more and more to find blacksmiths that provide anything other than cheap swords. They may reasonably conclude that this is just the nature of swords, and that there’s no such thing as a sword that withstands more than one or two blows. Eventually, even people who want good swords accept that they won’t be able to find a blacksmith who makes them, and settle for cheap swords instead. In effect, https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons

s6ro 3y ago

Ah memories... I once had 100K Python 2.x codebase to maintain. No docs, no tests and then being required to add features. Had to use the debugger[1] to make sense of the codebase and types, formats etc. Sheer madness and such a waste of time... To this day, I'm baffled by the dynamic language folks who cannot get their head around how strictness/rigor (via a good expressive type system) actually makes maintenance easier and more importantly: cheaper. [1] https://github.com/inducer/pudb https://github.com/inducer/pudb

john61 3y ago

It is no problem to write buggy unmaintainable code in a strongly typed language. It is also perfectly possible to write clean and maintainable code in a duck type language.

bronxpockfabz 3y ago

https://github.com/beartype/beartype https://github.com/beartype/beartype I wish more people started using Beartype, it makes Python bearable

BiteCode_dev 3y ago

That's more of a cultural thing. Because Python is easy to get into, and you can be productive very fast, a lot of people with little experience in programming start to build things that are out of their reach. They just don't know it, and the people hiring them don't either. It's not a new phenomenon, we got the same when Java arrived, then PHP, etc. Every time something makes things easier, you get a wave of beginners set to create things they are not qualified to do. And because you now need software for everything, leaders also get into positions where are not qualify to make some decisions about it. You pair unqualified leaders with unqualified devs, and you get this. And it's now easier than ever to be in this position, also you are more pressured than ever to get into this position. I've worked on plenty of big Python systems myself, it's not the tech per se (it has the usual pros and cons game like every language), but Python is the language used "when you don't know programming". Which is ironic, because when I started to use it in Python 2.4, only passionate devs used it because it was niche, and we had the opposite effect.

pjmlp 3y ago
It is BASIC all over again.
st4lz 3y ago
> Every time something makes things easier, you get a wave of beginners set to create things they are not qualified to do. And because you now need software for everything, leaders also get into positions where are not qualify to make some decisions about it. You pair unqualified leaders with unqualified devs, and you get this. How easy it is to call other people not qualified because they have different opinions. I've seen more projects failed because the only thing the devs cared about was an abstract code correctness, which made even basic PRs getting polished for weeks. I don't think you need to worry too much about types or tests if you are just doing a prototoype to validate ideas or your user base is 0.

BoppreH 3y ago

I've always wondered if we shouldn't be pushing more for type inference with flow analysis. I consider static type checking a must, but the voices opposing type annotations have some valid points: it pollutes the source code, slows down prototyping, and only catches a limited class of errors. A static checker with global type inference could be a middle ground. Pylance/Pyright already does that for most VS Code users. A more advanced form of this could track types that are too tedious for humans, like list emptiness, bounds, string alphabet, functions that can raise exceptions, idempotence, side effects, etc. Those types can then be shown in the IDE, according to user preference, like docs or source control metadata. I have a feeling that with enough caching this could be done with ok performance, but I hear that useful error reporting is an unsolved problem.

classified 3y ago

So much for all those "you don't need static typing" / "dynamic typing is better" arguments.

flohofwoe 3y ago

Are the 'bugs' actually triggered in the real-world usage of the code, or are they just 'potential bugs' which would be triggered given specific inputs that don't happen in the current usage of the code? Might seem like nitpicking, but the 'probability of a bug being triggered in the future' is an important metric for figuring out if large scale refactorings are actually worth the risk of introducing new bugs. Disclaimer: I'm coming from the static typing hemisphere, but adding type information to a codebase that was originally written with runtime-duck-typing in mind sounds like it might actually make maintainability of the code worse - that's at least my experience after trying to do exactly this in a medium-sized Python codebase - some of the 'typings' ended up so complex and weird that they completely dominated the code and turned what was very simple code at the beginning into an unreadable mess (looking almost like C++ template code). I eventually rolled back the changes. Basically, if I wanted a statically typed language, I wouldn't have chosen Python in the first place for this type of project.

eyko 3y ago
Can't speak for the parent commenter but their situation felt familiar. I'm in a new role where I'm contributing to a very large Python codebase (first commit from over a decade ago) and whilst there's a lot of more recent code written with type hints, they're not enforced and they're not always reliable or consistent. On my second contribution, I introduced a bug that pyright immediately spotted after I upped the checks from basic to strict. Our Sentry reports a lot of non-critical bugs that are being triggered in production that in many cases are _already_ highlighted by the type checker. It's just a big enough codebase that it's not practical to attempt to fix them all. If I were starting a project today in Python, I would definitely try setting up my tooling and processes to enforce type checking to at least some extent. Fortunately, we have a lot of guardrails that recover from the typical exceptions that occur in those cases, but as I experienced first hand, it's very easy to introduce new bugs when relying only on my intuition for those difficult-to-spot cases. Tooling definitely helps there. My previous role had me writing TypeScript and Rust so there's always the temptation, or bias, to advocate for strict typing everywhere, but I'm conscious that it's neither practical nor feasible in large codebases with hundreds of contributors (most of whom may be more accustomed to dynamically typed languages).

crabbone 3y ago

Just imagine how much more bugs you add... Also, imagine how much more productive you would've been if you instead of doing what you are doing you'd do something worthwhile, like, design the program or organize the testing better. I've heard so many nonsense claims like yours both in proprietary projects and in open-source ones, and not a single time did the code base became better. In most cases a small(-er) pile of garbage became a bigger pile of garbage.

keithalewis 3y ago

Are you implying people writing code should have some training in how computers actually work? Heretic!

bilsbie 3y ago

> trying to suss out the types on a given code path. Are there any tools for that? If not could one be made? Sounds almost like an advanced linter.

bilsbie 3y ago

That sounds impressive but I feel like this might be a misunderstanding of Python. You can break almost any python code by passing in the wrong type. The code throwing runtime type-errors IS the type checking. The danger is only if the code happily operates on the wrong types and returns incorrect results.

cmrdporcupine 3y ago

It's funny how the pendulum swings. 10-15 years ago sticking your neck our and advocating for static types was like painting a target on your back. Everything was about dynamic languages, Ruby was the coolest of cool, and people who waved the placard of high level types, type inference, etc. from the ML/Haskell cool were seen on the whole as impractical academic ivory tower types, cloistered around Lambda The Ultimate... When I advocated for the importance of static typing for software robustness the response I got from many other industry people I knew was that this was the job of intensive unit testing. That the supposed loss of rapid turnaround that came with static types wasn't worth it, and that test-first-design solved this problem. Now Typescript and Rust are common place, and we have Pythonistas annotating their programs with types. I'm happy, but maybe a bit bitter about the past :-)

roflyear 3y ago

I work at a company that has a fucking nightmare of a C# project. It's total hell to work on. I don't think the language makes much of a difference.