13 ms·

I think in general Python's biggest challenge is that it doesn't scale well. This is an agglomeration of issues around the same theme: bad packaging when there'

I think in general Python's biggest challenge is that it doesn't scale well. This is an agglomeration of issues around the same theme: bad packaging when there's a lot of cross-cutting dependencies, slow performance, no concurrency, typing as second-class citizens, etc. All of that is barely noticeable when you're just getting started on a small experimental project, but incredibly painful in large production systems. I strongly suspect that devs' satisfaction with Python is strongly correlated with the size of the codebase they're working on. Generally people using Python for one-off projects or self-contained tools tend to be pretty happy. People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language. What I've observed a lot is that many startups or greenfield projects start with Python to get an MVP out the door as fast as possible. Then as the scope of the software expands they feel increasingly bogged down and trapped in the language.

d0mine 6y ago
Have you ever seen O(million) lines enterprise codebase that didn't suck?
- MR4D 6y ago
  LOL!! Or even worse - could you imagine how many lines that would be in C++ ? Yowza!
  - oarabbus_ 6y ago
    n!
- lrem 6y ago
  I've worked on several separate projects of that size in C++ and Go. None of them seemed to achieve a similar mess as Python codebases with one or two dozen thousand lines seem to. OTOH, all the typing developments in Python should have helped? I don't have that much experience with them in enterprise setting.
- zo1 6y ago
  I have - and it's not that bad. The key is you have to have someone coordinating and driving a shared vision for the codebase and patterns. But it's hard to find people with that sort of passion and drive to follow-through as it's a multi-year endeavor with politics all over. Otherwise its a thousand implementations of the same 100-line piece of code interspersed everywhere.
  - ineedasername 6y ago
    It seems like quality code management gets passed over by (bad) management because it looks like it doesn't directly move the project forward. Which is strange because those same managers may be full adherents to micro tasking projects in a project management system whose purpose is basically to do for the project what code management does for the code itself. In my workplace, we've recently had leadership that appreciates these things, and the difference is night & day. Simple requests from "stakeholders" (I hate that term) are often filled in days, or same day, instead of weeks. I think it helps tremendously that the primary manager is also a coder herself, and still codes ~25% of her job.
  - JAlexoid 6y ago
    That's the problem with some languages - they lack a visionary, that drives the overall understanding of how things should be structured. I believe it's Guido that basically said - if you don't like how Python does it, then implement it in C. And that's how you end up with great C based libraries bound to python... and python is often used as a messy orchestrating language.
- gordaco 6y ago
  This is surely anecdotic and very subjective, but I have (in Java and in C++; IIRC the exact versions were Java 7 and C++03), and the level of pain was lower than with a Python code base that was about one order of magnitude smaller. In the case of C++, the pain was mostly asociated with an ancient build system we used; the code itself was relatively manageable. There was almost zero template code, and maybe that helped (although in other occasions I've worked with smaller C++03 codebases that relied heavily on templates and I didn't find them that bad). Not all codebases are equal and maybe I was lucky, but in my experience, using dynamic languages (or, to be exact, any language where the compiler doesn't nag you when there is a potential problem) doesn't scale well.
- commonturtle 6y ago
  I've worked with an O(100k) line code base in Python that was pure torture. Honestly, I was so desperate for static-typing by the end that I would have preferred if it was all written in C++. Large codebases are really hard reason about without types. I'm glad we now have projects like Pyre that are trying to bring typing to Python.
  - ineedasername 6y ago
    > desperate for static-typing Can you explain why? I honestly don't know, because my experience with C++ was during school ~20 years ago, and since then professionally I've used mostly python in relatively small codebases where it's all my own code (mostly for data processing/analysis). Thanks! (Although I did have to write some C code to glue together data in a very old legacy system that didn't support C++, much less python. It took a lot more effort to do something simple, but it was also strangely a really rewarding experience. Kind of similar to feeling when I work with assembly on hobby projects)
    - johntb86 6y ago
      The main problem with duck-typing like python has is the lack of consistency between different objects that code has to work on. Different callers may pass objects with different sets of methods into a function and expect it to work. You run into the case where the object that was passed in is one with subtly-mismatched behavior from what your method expects, but you don't know who created it - it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on. Static typing prevents that by telling you early where the mismatch is happening - some method calls into another with a variable of the wrong type, and that's where the bug is. It also allows tooling to look up the types of variables and quickly get information about their properties.
      - ineedasername 6y ago
        Got it, that makes sense. It also makes sense why I've not much been bothered by it in python since my relative small code bases don't have that many layers of abstraction laid on top of each other. I'm generally not working with more than 2,000-3,000 lines, and I can just about keep the basic structure in my head. (Unless it's been a while since I've had to revisit it... then I often hate my past self for getting "clever" in some way)
- JAlexoid 6y ago
  I am literally working on two projects that are roughly 100kLOC each. The Scala Spark project I can navigate, understand, test and consider to be average complexity... with some failures, unique to Scala. The Python Spark project is barely readable. People who built the Python Spark codebase are "experienced Python devs". While Scala codebase was built by people who used Scala for the first time. (take this anecdote, as evidence for the poor tooling and guidance present in python community.... and BDFL's own failures)
Guthur 6y ago
And how much of those problems are an artefact of moving fast and getting things down. I've seen the exact same scenario with other languages. The problem is that in a start up environment you are likely adding amd retiring more "features" at a speed that layers so much complexity that you can no longer reason about what business rules are actually valid any more.
- EsotericAlgo 6y ago
  I think that's part of it. There is a convention over configuration issue as well. A language like Go forces some patterns like package management and formatting unless you actively try to subvert it. It wouldn't surprise me if many of these issues are self-selecting in the language communities as well.
aserafini 6y ago
No concurrency? asyncio is great for I/O bound network stuff!
- colonwqbang 6y ago
  "No parallelism" is probably what was meant.
  - dragonwriter 6y ago
    > "No parallelism" is probably what was meant. Which is still wrong, of course, but "no in-process (or in-single-runtime-instance) parallelism" would be correct, as would "forking inconvenient parallelism".
    - colonwqbang 6y ago
      Posix fork() doesn't really count, if that's what you mean...
      - dragonwriter 6y ago
        Why doesn't Python's multiprocessing module (which uses fork by default on Unix) count? It literally exists for parallelism.
        int_19h 6y ago
        It doesn't use fork() on macOS anymore, because some of Apple's own APIs get broken by its use. Pretty much any app that uses both fork and threads, has to jump through many hoops to make the two work together well. And this applies to all the libraries that it uses, directly or indirectly - if any library spawns a thread and does some locking in it, you get all kinds of hard-to-debug deadlocks if you try to fork. So unless you have very good perf reasons to need fork, I would strongly recommend multiprocessing.set_start_method("spawn") on all platforms. No obscure bugs, and it'll also behave the same everywhere, so things will be more portable. Code using multiprocessing that's written to rely on fork semantics can be very difficult to port later.
deleted 6y ago
[deleted]
jnwatson 6y ago
I work on Python every day on a reasonably large code base and have none of the issues you’re talking about. I’m 10x more productive than similar C or Java projects. Dependency management is about as easy as it is going to get. We have problems with our dependencies breaking stuff, but who doesn’t? People talk as if packaging is a solved problem. It isn’t in any language. And then they complain that Python packaging changes too much. That’s because folks are iterating on a hard problem.
- disgruntledphd2 6y ago
  Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.
  - erjiang 6y ago
    Curious to know what issues you have with deploying Python codebases. Out of all of the minor and major gripes I have with Python, deployment is not one of them.
    - tetha 6y ago
      To me, python deployments are painless, as long as you can stick to pure dependencies and possibly wheels. Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting. I've done that too much to like python anymore.
      - disgruntledphd2 6y ago
        Yup, yup. I deploy statistical models with Python, and these always have C dependencies. Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy. I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.
talolard 6y ago
This described is perfectly
adamc 6y ago
At O(million), the problem wrangling it has more to do with how well its architected and written than it being Python. Python is at least easy to read. Its major deficiency is the lack of annotation of parameters, and that's something that could now be fixed... but it isn't going to be fixed in that much historical code. It you are trying to get performance out of it (which doesn't really hinge on whether it's a million lines of code), then Python might be the wrong choice. But you can always write it in Rust or C and give Python an API to the functionality. I agree that packaging is a mess. Fixing that mess with modularization in Java took a long time, and most other languages have that problem, too.
- JAlexoid 6y ago
  I disagree. Python is not inherently easy to read. Explicitness and naming standards screw up the clarity of any code... Not to mention the complexity when you get into OOP.
  - andi999 6y ago
    Also lack of switch-case statements dont help. (Workaround is either if statements or dict of functions to be called)
KaiserPro 6y ago
"devop" here > doesn't scale well. Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points. > bad packaging when there's a lot of cross-cutting dependencies Much as I hate it, docker solves this. Failing that poetry or if you must venv. (if you're being "clever" statically compile everything and ship the whole environment, including the interpreter) its packaging is a joy compared to node. Even better, enforce standard environments, which stops all of this. One version of everything. you want to change it? best upgrade it for everyone else. > slow performance Meh, again depends on your use case. If you're really into performance then dump out to C/C++ and pybind it. fronting performance critical code in python is a fairly decent way to allow non specialists handle and interface performance critical code. Its far cheaper to staff it that way too. standard python programmers are cheaper than performance experts. If we are being realistic, most of the time 80% of python programs are spend waiting on network. Granted, python is not overly fast, but then most of the time your bottleneck is the developer not the language. > no concurrency Yes, this is a pain. I would really like some non GIL based threading. However its not really been that much of a problem. multiprocessing Queues are useful here, if limited. Failing that, make more processes and use an rpc system. > typing as second-class citizens The annotation is under developed. being reliant on dataclass libraries to enforce typing is a bit poop. > People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language. I work with a _massive_ monorepo. Python isn't the problem, its programmer being "clever" or making needless abstractions of abstractions. None of that is python's issues, its egotistical programmer not wanting to read other people's (un documented) code. And not wanting to spend time make other people's code better.
- rovr138 6y ago
  >Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points. This is very important. A lot of people think that just using go or rust or whatever other language is new fixes all of this. But with a big enough project, you'll find all the issues. It's just a matter of time.
  - 6y ago
ineedasername 6y ago
>People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language. This seems to be the case with most languages, especially if good code control isn't practiced, and unfortunately that's not uncommon.
hobby-coder-guy 6y ago
> O(million) What does this mean?
zmmmmm 6y ago
Very much agree. I oversee a relatively small python codebase, but getting good quality, safe code out of the developers in a controlled way is really hard - there are so many ways in which the language just doesn't have enough power to serve the needs of more complex apps. We have massive amounts of linting, type hinting, code reviews spotting obvious errors that would be just invalid code in other languages. It's like getting on a roller coaster without a seat belt or a guard rail. It's fun at first, and you will make it around the first few bends OK ... then get ready ... Of course, with enormous discipline, skill and effort you can overcome all this. But it just leaves the question - really, is this the best tool for the job in the end? Especially when you are paying for it with horrifically bad performance and other limitations.
reissbaker 6y ago
I work at Instagram, which is a O(millions) LOC Python monorepo with massive throughput and a large engineering team. It's actually quite nice — but our code is heavily, heavily typed. It would be miserable without the type system. Some of the older parts of the codebase are more loosely typed (although they're shrinking reasonably quickly), and those sections are indeed a huge pain to ramp up on. Part of the success of IG's large Python codebase is FB's investment into developer tooling; for example, FB wrote (and open-sourced, FWIW) our own PEP-484 compliant type checker, Pyre [1], because mypy was too slow for a codebase of our size. 1: https://github.com/facebook/pyre-check https://github.com/facebook/pyre-check
- david-gpu 6y ago
  Have you guys published any whitepaper on this subject? These last few years working on moderately large python codebases with dynamic typing have been less than idilic.
  - wilsonthewhale 6y ago
    Here's a small collection https://pyre-check.org/docs/pysa-additional-resources https://pyre-check.org/docs/pysa-additional-resources
- anonymouse008 6y ago
  Yeah, you could use my hairline to tell when I’m working with Python... I still haven’t figured out a good typed framework for it yet.
  - kevinarpe 6y ago
    I also struggle with weak(er) typing in Python, compared to strong(er) typing in C++, Java, or C# -- or even VBA(!). Frustrated like you, I wrote my own open source type checking library for Python. You might be interested to read about it here: https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master/rambutan3/check_args/RCheckArgs.py#L149 https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master... I have used that library on multiple projects for my job. It makes the code run about 50% slower, on average, because all the type checking is done at run-time. I am OK with the slow down because I don't use Python when I need speed. My "developer speed" was greatly improved with stricter types. Finally, this isn't the first time I wrote a type checking library/framework. I did the same for Perl more than 10yrs ago. Unfortunately, that code is proprietary, so not open source. :( The abstract ideas were very similar. Our team was so frustrated with legacy Perl code, so I wrote a type checking library, and we slowly applied it to the code base. About two years later, it was much less painful!
kbumsik 6y ago
Could I ask you what language you use instead then?
andi999 6y ago
Is concurrency really an issue? Yes, you do not have threads, but you can launch multiple processes. Do you really need to habe shared memory for your concurrent needs (i think it is muuuch easier to introduce subtle bug into a shared memory concurrency (threads))

d0mine 6y ago

Have you ever seen O(million) lines enterprise codebase that didn't suck?

MR4D 6y ago
LOL!! Or even worse - could you imagine how many lines that would be in C++ ? Yowza!
- oarabbus_ 6y ago
  n!
lrem 6y ago
I've worked on several separate projects of that size in C++ and Go. None of them seemed to achieve a similar mess as Python codebases with one or two dozen thousand lines seem to. OTOH, all the typing developments in Python should have helped? I don't have that much experience with them in enterprise setting.
zo1 6y ago
I have - and it's not that bad. The key is you have to have someone coordinating and driving a shared vision for the codebase and patterns. But it's hard to find people with that sort of passion and drive to follow-through as it's a multi-year endeavor with politics all over. Otherwise its a thousand implementations of the same 100-line piece of code interspersed everywhere.
- ineedasername 6y ago
  It seems like quality code management gets passed over by (bad) management because it looks like it doesn't directly move the project forward. Which is strange because those same managers may be full adherents to micro tasking projects in a project management system whose purpose is basically to do for the project what code management does for the code itself. In my workplace, we've recently had leadership that appreciates these things, and the difference is night & day. Simple requests from "stakeholders" (I hate that term) are often filled in days, or same day, instead of weeks. I think it helps tremendously that the primary manager is also a coder herself, and still codes ~25% of her job.
- JAlexoid 6y ago
  That's the problem with some languages - they lack a visionary, that drives the overall understanding of how things should be structured. I believe it's Guido that basically said - if you don't like how Python does it, then implement it in C. And that's how you end up with great C based libraries bound to python... and python is often used as a messy orchestrating language.
gordaco 6y ago
This is surely anecdotic and very subjective, but I have (in Java and in C++; IIRC the exact versions were Java 7 and C++03), and the level of pain was lower than with a Python code base that was about one order of magnitude smaller. In the case of C++, the pain was mostly asociated with an ancient build system we used; the code itself was relatively manageable. There was almost zero template code, and maybe that helped (although in other occasions I've worked with smaller C++03 codebases that relied heavily on templates and I didn't find them that bad). Not all codebases are equal and maybe I was lucky, but in my experience, using dynamic languages (or, to be exact, any language where the compiler doesn't nag you when there is a potential problem) doesn't scale well.
commonturtle 6y ago
I've worked with an O(100k) line code base in Python that was pure torture. Honestly, I was so desperate for static-typing by the end that I would have preferred if it was all written in C++. Large codebases are really hard reason about without types. I'm glad we now have projects like Pyre that are trying to bring typing to Python.
- ineedasername 6y ago
  > desperate for static-typing Can you explain why? I honestly don't know, because my experience with C++ was during school ~20 years ago, and since then professionally I've used mostly python in relatively small codebases where it's all my own code (mostly for data processing/analysis). Thanks! (Although I did have to write some C code to glue together data in a very old legacy system that didn't support C++, much less python. It took a lot more effort to do something simple, but it was also strangely a really rewarding experience. Kind of similar to feeling when I work with assembly on hobby projects)
  - johntb86 6y ago
    The main problem with duck-typing like python has is the lack of consistency between different objects that code has to work on. Different callers may pass objects with different sets of methods into a function and expect it to work. You run into the case where the object that was passed in is one with subtly-mismatched behavior from what your method expects, but you don't know who created it - it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on. Static typing prevents that by telling you early where the mismatch is happening - some method calls into another with a variable of the wrong type, and that's where the bug is. It also allows tooling to look up the types of variables and quickly get information about their properties.
    - ineedasername 6y ago
      Got it, that makes sense. It also makes sense why I've not much been bothered by it in python since my relative small code bases don't have that many layers of abstraction laid on top of each other. I'm generally not working with more than 2,000-3,000 lines, and I can just about keep the basic structure in my head. (Unless it's been a while since I've had to revisit it... then I often hate my past self for getting "clever" in some way)
JAlexoid 6y ago
I am literally working on two projects that are roughly 100kLOC each. The Scala Spark project I can navigate, understand, test and consider to be average complexity... with some failures, unique to Scala. The Python Spark project is barely readable. People who built the Python Spark codebase are "experienced Python devs". While Scala codebase was built by people who used Scala for the first time. (take this anecdote, as evidence for the poor tooling and guidance present in python community.... and BDFL's own failures)

Guthur 6y ago

And how much of those problems are an artefact of moving fast and getting things down. I've seen the exact same scenario with other languages. The problem is that in a start up environment you are likely adding amd retiring more "features" at a speed that layers so much complexity that you can no longer reason about what business rules are actually valid any more.

EsotericAlgo 6y ago
I think that's part of it. There is a convention over configuration issue as well. A language like Go forces some patterns like package management and formatting unless you actively try to subvert it. It wouldn't surprise me if many of these issues are self-selecting in the language communities as well.

aserafini 6y ago

No concurrency? asyncio is great for I/O bound network stuff!

colonwqbang 6y ago
"No parallelism" is probably what was meant.
- dragonwriter 6y ago
  > "No parallelism" is probably what was meant. Which is still wrong, of course, but "no in-process (or in-single-runtime-instance) parallelism" would be correct, as would "forking inconvenient parallelism".
  - colonwqbang 6y ago
    Posix fork() doesn't really count, if that's what you mean...
    - dragonwriter 6y ago
      Why doesn't Python's multiprocessing module (which uses fork by default on Unix) count? It literally exists for parallelism.
      - int_19h 6y ago
        It doesn't use fork() on macOS anymore, because some of Apple's own APIs get broken by its use. Pretty much any app that uses both fork and threads, has to jump through many hoops to make the two work together well. And this applies to all the libraries that it uses, directly or indirectly - if any library spawns a thread and does some locking in it, you get all kinds of hard-to-debug deadlocks if you try to fork. So unless you have very good perf reasons to need fork, I would strongly recommend multiprocessing.set_start_method("spawn") on all platforms. No obscure bugs, and it'll also behave the same everywhere, so things will be more portable. Code using multiprocessing that's written to rely on fork semantics can be very difficult to port later.

deleted 6y ago

[deleted]

jnwatson 6y ago

I work on Python every day on a reasonably large code base and have none of the issues you’re talking about. I’m 10x more productive than similar C or Java projects. Dependency management is about as easy as it is going to get. We have problems with our dependencies breaking stuff, but who doesn’t? People talk as if packaging is a solved problem. It isn’t in any language. And then they complain that Python packaging changes too much. That’s because folks are iterating on a hard problem.

disgruntledphd2 6y ago
Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.
- erjiang 6y ago
  Curious to know what issues you have with deploying Python codebases. Out of all of the minor and major gripes I have with Python, deployment is not one of them.
  - tetha 6y ago
    To me, python deployments are painless, as long as you can stick to pure dependencies and possibly wheels. Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting. I've done that too much to like python anymore.
    - disgruntledphd2 6y ago
      Yup, yup. I deploy statistical models with Python, and these always have C dependencies. Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy. I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.

talolard 6y ago

This described is perfectly

adamc 6y ago

At O(million), the problem wrangling it has more to do with how well its architected and written than it being Python. Python is at least easy to read. Its major deficiency is the lack of annotation of parameters, and that's something that could now be fixed... but it isn't going to be fixed in that much historical code. It you are trying to get performance out of it (which doesn't really hinge on whether it's a million lines of code), then Python might be the wrong choice. But you can always write it in Rust or C and give Python an API to the functionality. I agree that packaging is a mess. Fixing that mess with modularization in Java took a long time, and most other languages have that problem, too.

JAlexoid 6y ago
I disagree. Python is not inherently easy to read. Explicitness and naming standards screw up the clarity of any code... Not to mention the complexity when you get into OOP.
- andi999 6y ago
  Also lack of switch-case statements dont help. (Workaround is either if statements or dict of functions to be called)

KaiserPro 6y ago

"devop" here > doesn't scale well. Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points. > bad packaging when there's a lot of cross-cutting dependencies Much as I hate it, docker solves this. Failing that poetry or if you must venv. (if you're being "clever" statically compile everything and ship the whole environment, including the interpreter) its packaging is a joy compared to node. Even better, enforce standard environments, which stops all of this. One version of everything. you want to change it? best upgrade it for everyone else. > slow performance Meh, again depends on your use case. If you're really into performance then dump out to C/C++ and pybind it. fronting performance critical code in python is a fairly decent way to allow non specialists handle and interface performance critical code. Its far cheaper to staff it that way too. standard python programmers are cheaper than performance experts. If we are being realistic, most of the time 80% of python programs are spend waiting on network. Granted, python is not overly fast, but then most of the time your bottleneck is the developer not the language. > no concurrency Yes, this is a pain. I would really like some non GIL based threading. However its not really been that much of a problem. multiprocessing Queues are useful here, if limited. Failing that, make more processes and use an rpc system. > typing as second-class citizens The annotation is under developed. being reliant on dataclass libraries to enforce typing is a bit poop. > People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language. I work with a _massive_ monorepo. Python isn't the problem, its programmer being "clever" or making needless abstractions of abstractions. None of that is python's issues, its egotistical programmer not wanting to read other people's (un documented) code. And not wanting to spend time make other people's code better.

rovr138 6y ago
>Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points. This is very important. A lot of people think that just using go or rust or whatever other language is new fixes all of this. But with a big enough project, you'll find all the issues. It's just a matter of time.
- 6y ago

ineedasername 6y ago

>People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language. This seems to be the case with most languages, especially if good code control isn't practiced, and unfortunately that's not uncommon.

hobby-coder-guy 6y ago

> O(million) What does this mean?

zmmmmm 6y ago

Very much agree. I oversee a relatively small python codebase, but getting good quality, safe code out of the developers in a controlled way is really hard - there are so many ways in which the language just doesn't have enough power to serve the needs of more complex apps. We have massive amounts of linting, type hinting, code reviews spotting obvious errors that would be just invalid code in other languages. It's like getting on a roller coaster without a seat belt or a guard rail. It's fun at first, and you will make it around the first few bends OK ... then get ready ... Of course, with enormous discipline, skill and effort you can overcome all this. But it just leaves the question - really, is this the best tool for the job in the end? Especially when you are paying for it with horrifically bad performance and other limitations.

reissbaker 6y ago

I work at Instagram, which is a O(millions) LOC Python monorepo with massive throughput and a large engineering team. It's actually quite nice — but our code is heavily, heavily typed. It would be miserable without the type system. Some of the older parts of the codebase are more loosely typed (although they're shrinking reasonably quickly), and those sections are indeed a huge pain to ramp up on. Part of the success of IG's large Python codebase is FB's investment into developer tooling; for example, FB wrote (and open-sourced, FWIW) our own PEP-484 compliant type checker, Pyre [1], because mypy was too slow for a codebase of our size. 1: https://github.com/facebook/pyre-check https://github.com/facebook/pyre-check

david-gpu 6y ago
Have you guys published any whitepaper on this subject? These last few years working on moderately large python codebases with dynamic typing have been less than idilic.
- wilsonthewhale 6y ago
  Here's a small collection https://pyre-check.org/docs/pysa-additional-resources https://pyre-check.org/docs/pysa-additional-resources
anonymouse008 6y ago
Yeah, you could use my hairline to tell when I’m working with Python... I still haven’t figured out a good typed framework for it yet.
- kevinarpe 6y ago
  I also struggle with weak(er) typing in Python, compared to strong(er) typing in C++, Java, or C# -- or even VBA(!). Frustrated like you, I wrote my own open source type checking library for Python. You might be interested to read about it here: https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master/rambutan3/check_args/RCheckArgs.py#L149 https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master... I have used that library on multiple projects for my job. It makes the code run about 50% slower, on average, because all the type checking is done at run-time. I am OK with the slow down because I don't use Python when I need speed. My "developer speed" was greatly improved with stricter types. Finally, this isn't the first time I wrote a type checking library/framework. I did the same for Perl more than 10yrs ago. Unfortunately, that code is proprietary, so not open source. :( The abstract ideas were very similar. Our team was so frustrated with legacy Perl code, so I wrote a type checking library, and we slowly applied it to the code base. About two years later, it was much less painful!

kbumsik 6y ago

Could I ask you what language you use instead then?

andi999 6y ago

Is concurrency really an issue? Yes, you do not have threads, but you can launch multiple processes. Do you really need to habe shared memory for your concurrent needs (i think it is muuuch easier to introduce subtle bug into a shared memory concurrency (threads))