6 ms·

On the one side, yes Python is incredibly versatile. On the other side, I have worked on many Python projects, some of them fairly high profile, and I have see

by Yoric 3y ago

On the one side, yes Python is incredibly versatile. On the other side, I have worked on many Python projects, some of them fairly high profile, and I have seen exactly two kinds of Python codebases: 1. a few were written by extreme professionals, plugging at every single hole, with ~100% coverage, plus considerable maintenance because every dependency upgrade tends to break something; 2. many that feel cobbled together, with undocumented (and often inconsistent) invariants "because something will eventually throw a TypeError if I make a mistake", undocumented metaprogramming, and heroic levels of maintenance because this stuff works until it doesn't. Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety. Option 2. feels like "experiments running out of control". That is the unfortunate price of this incredible versatility. I don't have a clear conclusion to this, except perhaps that Python is really good for the initial experimentation (when versatility actually gets you to initial results much faster), but really bad for... well, let's call it industralization.

BiteCode_dev 3y ago
To be fair, a division between hell and heaven will happen with any language. The question is: is this particular hell worth the result? There is no generic answer to that of course, it just happens is has been the case for me during those 20 years. First, you have to get to the industrialization phase. And of course, you have to get there, with the constraint of time, budgets, and talent. Second, Python does have less benefits for that phase than rust or haskel, but it's not impotent. Industrialization is just hard, no matter the language. Because we are an industry where a lot of self-taught people get a career, we tend to forget this last point. Creating a serious computing system is engineering, and this includes the project management part, and being rigorous about a lot of thing. Granted, the rigor needs to be higher with Python once your reach that scale, and at a certain point (which I wish everybody would reach), you may want to ditch it. It's a good problem to have though. Yet we have to remember not all industrialization attempts are equal. Most are really tame. I'd argue you could replace half the website code base out there with a bunch of bash scripts on a VPS and they would still make money. So even in this context, Python can deliver a lot.
- mumblemumble 3y ago
  This is roughly my experience, too. Static type safety has interesting YAGNI characteristics. For the bits of the code that must work, and where defects and regressions due to type errors may be subtle and difficult to detect, it's indispensable. But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst. Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment, and be selective in where I spend my industrialization tokens. Here's a list of the languages where I've found selectively rewriting the important bits in Rust to work well: C, C++, Python. I know which of those I'd rather use for prototyping, scripting, and high level control. Relevant Fred Brooks quote: "Plan to throw one away. You will, anyway."
  - giantrobot 3y ago
    > Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment This is one of my favorite aspects of Python. I can start with every module in prototype form and industrialize each module as its design firms up. I can spend my early development getting an idea fleshed out with minimal overhead.
    - Yoric 3y ago
      > I can start with every module in prototype form and industrialize each module as its design firms up. That is definitely the theory. And this flexibility is indeed very precious for some types of work. However... does it actually happen as you describe? I can count on half of the fingers of one hand the number of Python codebases that I've seen that actually feel like they've properly been reworked into something of industrial quality. All the other codebases I've seen are of the "I guess it works, maybe?" persuasion, quite possibly because there is always something higher priority than quality.
mnbant 3y ago
This is my experience as well. There are very few code bases with 100% coverage and strict development practices. Moreover, if the average developers get their hands on one of them, they'll degrade soon. The problem comes from the very "top", since CPython developers are in camp (2) and celebrate their development practices. Other languages also have this divide of course but not to the extent that Python has. As for maintenance: This is also true. I have rewritten a Python code base in C++. Even in C++ it is easier to maintain, because the C++ compiler does not break between versions and there are no new bugs. And modern C++ can look quite like Python.
tored 3y ago
I think you have to grade code in your project, i.e not every piece of code has the same value, high value => more strict, low value => less strict. Less strict code is something that you can throw away and rewrite without too much effort. This has the implication that you have bulkheads (like in a ship) to separate high value and low value code, thus you have to avoid code that is toxic (i.e magic) that spreads between them, e.g. ORM entities. When using the type system correctly you don't need 100% coverage, now I'm not a Python programmer so I don't how feasible that is with that type system, but in PHP I use the type system extensible and I don't write that many tests anymore, except for legacy code or when I need to test something specific, like an algorithm. By using the type system you can catch most programmer errors, and combine that with routinely throwing exceptions for any unwanted state, you will detect most bugs. The way unit testing has been interpreted and implemented it has become more of a nuisance than a help in fast moving projects. Usually what happens is that the team refactors some unit and then multiple test breaks and then time needs to be spent rewriting these test, but what that means is that the tests were actually useless after they where committed if you can change them as you like, when you change a test you are no longer testing the same thing. Thus tests can also can be graded, from high value tests, i.e stable, and low value tests. Depending how you organize your code, high value tests will usually test a larger set of functionality in one run, like a system (or integration) test from call to finish, these should never change regardless of much you refactor. High level tests should only change when your business logic changes. Low value test typically involves a lot of mocking and stubbing, those are pretty much pointless to commit after you implemented your unit. Too much mocking and stubbing is code smell.
rowanG077 3y ago
I would like to see a codebase of the first kind. All larger python projects I have seen are of kind 2...
- Yoric 3y ago
  The Firefox build system feels to me like kind 1. The Synapse Matrix server also feels pretty close (although I don't understand why it doesn't use Pydantic or something like Pydantic).
dagmx 3y ago
How are any of these points exclusive to Python among the commonly used languages? Everything you said applies equally to many C++ projects I work with as well
- Yoric 3y ago
  You are right, this is probably not specific to Python. It just happens that: 1. we're currently discussing Python; 2. anecdotally, the C++ projects on which I have worked were all case 1. ("code it like it's Rust") – I make no claim that this is representative, though.
martopix 3y ago
I think this is a form of selection bias. Certain other languages wouldn't even allow a project like your point 2. to get started. In Python, the bar for something to start kind of working is quite low, even if you're a beginner, a scientist of another discipline who needs a little script to manage some data, a teen who's learning to code, etc. So of course you see a lot of half-baked stuff. It's important we distinguish between 1. and 2., but 2s might sometimes be better than nothing.
- Yoric 3y ago
  Absolutely. In some domains, the choice is between 2. and nothing – and 2. is better. That's true for Python or JavaScript, for instance (in each their domain). I believe that we agree that there are actually many domains in which you really want 1., and if you can't have it, then "nothing" is generally preferable to 2.
fastasucan 3y ago
>Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety. ...but with the benefits of Python. You might not see them/need them but they are there :)
- Yoric 3y ago
  Fair enough.
berkes 3y ago
I see exactly this with Ruby. In that regard, Ruby and Python too, are very alike.

by Yoric 3y ago

BiteCode_dev 3y ago

To be fair, a division between hell and heaven will happen with any language. The question is: is this particular hell worth the result? There is no generic answer to that of course, it just happens is has been the case for me during those 20 years. First, you have to get to the industrialization phase. And of course, you have to get there, with the constraint of time, budgets, and talent. Second, Python does have less benefits for that phase than rust or haskel, but it's not impotent. Industrialization is just hard, no matter the language. Because we are an industry where a lot of self-taught people get a career, we tend to forget this last point. Creating a serious computing system is engineering, and this includes the project management part, and being rigorous about a lot of thing. Granted, the rigor needs to be higher with Python once your reach that scale, and at a certain point (which I wish everybody would reach), you may want to ditch it. It's a good problem to have though. Yet we have to remember not all industrialization attempts are equal. Most are really tame. I'd argue you could replace half the website code base out there with a bunch of bash scripts on a VPS and they would still make money. So even in this context, Python can deliver a lot.

mumblemumble 3y ago
This is roughly my experience, too. Static type safety has interesting YAGNI characteristics. For the bits of the code that must work, and where defects and regressions due to type errors may be subtle and difficult to detect, it's indispensable. But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst. Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment, and be selective in where I spend my industrialization tokens. Here's a list of the languages where I've found selectively rewriting the important bits in Rust to work well: C, C++, Python. I know which of those I'd rather use for prototyping, scripting, and high level control. Relevant Fred Brooks quote: "Plan to throw one away. You will, anyway."
- giantrobot 3y ago
  > Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment This is one of my favorite aspects of Python. I can start with every module in prototype form and industrialize each module as its design firms up. I can spend my early development getting an idea fleshed out with minimal overhead.
  - Yoric 3y ago
    > I can start with every module in prototype form and industrialize each module as its design firms up. That is definitely the theory. And this flexibility is indeed very precious for some types of work. However... does it actually happen as you describe? I can count on half of the fingers of one hand the number of Python codebases that I've seen that actually feel like they've properly been reworked into something of industrial quality. All the other codebases I've seen are of the "I guess it works, maybe?" persuasion, quite possibly because there is always something higher priority than quality.

mnbant 3y ago

This is my experience as well. There are very few code bases with 100% coverage and strict development practices. Moreover, if the average developers get their hands on one of them, they'll degrade soon. The problem comes from the very "top", since CPython developers are in camp (2) and celebrate their development practices. Other languages also have this divide of course but not to the extent that Python has. As for maintenance: This is also true. I have rewritten a Python code base in C++. Even in C++ it is easier to maintain, because the C++ compiler does not break between versions and there are no new bugs. And modern C++ can look quite like Python.

tored 3y ago

I think you have to grade code in your project, i.e not every piece of code has the same value, high value => more strict, low value => less strict. Less strict code is something that you can throw away and rewrite without too much effort. This has the implication that you have bulkheads (like in a ship) to separate high value and low value code, thus you have to avoid code that is toxic (i.e magic) that spreads between them, e.g. ORM entities. When using the type system correctly you don't need 100% coverage, now I'm not a Python programmer so I don't how feasible that is with that type system, but in PHP I use the type system extensible and I don't write that many tests anymore, except for legacy code or when I need to test something specific, like an algorithm. By using the type system you can catch most programmer errors, and combine that with routinely throwing exceptions for any unwanted state, you will detect most bugs. The way unit testing has been interpreted and implemented it has become more of a nuisance than a help in fast moving projects. Usually what happens is that the team refactors some unit and then multiple test breaks and then time needs to be spent rewriting these test, but what that means is that the tests were actually useless after they where committed if you can change them as you like, when you change a test you are no longer testing the same thing. Thus tests can also can be graded, from high value tests, i.e stable, and low value tests. Depending how you organize your code, high value tests will usually test a larger set of functionality in one run, like a system (or integration) test from call to finish, these should never change regardless of much you refactor. High level tests should only change when your business logic changes. Low value test typically involves a lot of mocking and stubbing, those are pretty much pointless to commit after you implemented your unit. Too much mocking and stubbing is code smell.

rowanG077 3y ago

I would like to see a codebase of the first kind. All larger python projects I have seen are of kind 2...

Yoric 3y ago
The Firefox build system feels to me like kind 1. The Synapse Matrix server also feels pretty close (although I don't understand why it doesn't use Pydantic or something like Pydantic).

dagmx 3y ago

How are any of these points exclusive to Python among the commonly used languages? Everything you said applies equally to many C++ projects I work with as well

Yoric 3y ago
You are right, this is probably not specific to Python. It just happens that: 1. we're currently discussing Python; 2. anecdotally, the C++ projects on which I have worked were all case 1. ("code it like it's Rust") – I make no claim that this is representative, though.

martopix 3y ago

I think this is a form of selection bias. Certain other languages wouldn't even allow a project like your point 2. to get started. In Python, the bar for something to start kind of working is quite low, even if you're a beginner, a scientist of another discipline who needs a little script to manage some data, a teen who's learning to code, etc. So of course you see a lot of half-baked stuff. It's important we distinguish between 1. and 2., but 2s might sometimes be better than nothing.

Yoric 3y ago
Absolutely. In some domains, the choice is between 2. and nothing – and 2. is better. That's true for Python or JavaScript, for instance (in each their domain). I believe that we agree that there are actually many domains in which you really want 1., and if you can't have it, then "nothing" is generally preferable to 2.

fastasucan 3y ago

>Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety. ...but with the benefits of Python. You might not see them/need them but they are there :)

Yoric 3y ago
Fair enough.

berkes 3y ago

I see exactly this with Ruby. In that regard, Ruby and Python too, are very alike.