5 ms·

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale I assume you're saying this out of experience, so can you gi

by stinos 5y ago

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale I assume you're saying this out of experience, so can you give a practical example of what you call 'notable scale'? And are you talking about desktop or web or mobile, or just all of them? Just so that we know what you are talking about in a concrete way. Not the 'large software with large number of developers is always a problem' way. edit I also see others here using 'scale' in a 'performance / number of request per second' etc meaning - I thought we were talking codebase size though. I mean if you know on beforehand this type of performace is a requirement it seems weird to turn to Python, of all things?

ck_one 5y ago
+1 Would be cool if some experienced dev could share some estimates when the Python scaling issues start. When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues. When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.
- ferdowsi 5y ago
  I worked for a startup that built their services using Python and Twisted. The codebase was a monolith broken up into several Twisted services, by the time I left it was probably 200k LOC of Python. For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task. The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale. In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.
  - deleted 5y ago
    [deleted]
gjulianm 5y ago
I am seeing what the GP comment says in a codebase I manage, around 47k LOC. Not massive at all, but enough where the problems mentioned above start to pop up. This is an application that runs in servers to analyze traffic data, so it has both analysis code and also a lot of code for the analysis framework. It gets hard to manage. I have unit tests and also integration tests that cover some, but not all of the code paths (it's very hard in this case to have everything covered). Most of the times, when they fail is due to something that would have been caught by static typing. The codebase is being slowly migrated to static typing. On one hand, as the parent says, the typing module is still immature and there are still some Python constructs (not too weird ones, see [1] for an example) that you can't type-check correctly. On the other hand, I like the fact that you can include typing slowly and not all at once, it makes the effort much easier to tackle. And, if typing works, it works well. Regarding performance, well. Parallelism is pretty hard to do well, and the language itself is not the fastest thing. Some parts are migrated to a C extension but that's costly in terms of development and debugging time. Despite all of that, I do think that Python was the best choice for our situation, and still is. Maybe from this point onwards another language would make things easier, but without Python's library support, ease and speed of development and expressiveness the cost of just getting to market would have been far higher, and probably we wouldn't have reached this point with other languages. And migrating the codebase to another language is just not worth it at all, as there are still a lot of areas we can improve quite a lot with far less effort than a full rewrite. 1: https://github.com/python/mypy/issues/2756#issuecomment-877255741 https://github.com/python/mypy/issues/2756#issuecomment-8772...

by stinos 5y ago

ck_one 5y ago

+1 Would be cool if some experienced dev could share some estimates when the Python scaling issues start. When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues. When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.

ferdowsi 5y ago
I worked for a startup that built their services using Python and Twisted. The codebase was a monolith broken up into several Twisted services, by the time I left it was probably 200k LOC of Python. For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task. The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale. In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.
- deleted 5y ago
  [deleted]

gjulianm 5y ago

I am seeing what the GP comment says in a codebase I manage, around 47k LOC. Not massive at all, but enough where the problems mentioned above start to pop up. This is an application that runs in servers to analyze traffic data, so it has both analysis code and also a lot of code for the analysis framework. It gets hard to manage. I have unit tests and also integration tests that cover some, but not all of the code paths (it's very hard in this case to have everything covered). Most of the times, when they fail is due to something that would have been caught by static typing. The codebase is being slowly migrated to static typing. On one hand, as the parent says, the typing module is still immature and there are still some Python constructs (not too weird ones, see [1] for an example) that you can't type-check correctly. On the other hand, I like the fact that you can include typing slowly and not all at once, it makes the effort much easier to tackle. And, if typing works, it works well. Regarding performance, well. Parallelism is pretty hard to do well, and the language itself is not the fastest thing. Some parts are migrated to a C extension but that's costly in terms of development and debugging time. Despite all of that, I do think that Python was the best choice for our situation, and still is. Maybe from this point onwards another language would make things easier, but without Python's library support, ease and speed of development and expressiveness the cost of just getting to market would have been far higher, and probably we wouldn't have reached this point with other languages. And migrating the codebase to another language is just not worth it at all, as there are still a lot of areas we can improve quite a lot with far less effort than a full rewrite. 1: https://github.com/python/mypy/issues/2756#issuecomment-877255741 https://github.com/python/mypy/issues/2756#issuecomment-8772...