6 ms·

Hi - yes loved your blogpost! Also very tired of the "async magic performance fairy dust" :) It's a difficult myth to dispel and I think the situation in term

Hi - yes loved your blogpost! Also very tired of the "async magic performance fairy dust" :) It's a difficult myth to dispel and I think the situation in terms of public mindshare is much worse now than it was in 2015. Some very silly claims from the async crowd now have basically widespread credence. I think one of the root causes is that people are sometimes very woolly about how multi-processing works. One of the others is that I think it's easy to make the conceptual mistake of 1 sync workers = 1 async worker and do a comparison that way One of my worries is that right now it feels like everything in Python is being rewritten in asyncio and the balkanisation of the community could well be more problematic than 2 vs 3.

throwaway894345 6y ago
For me it's worth the effort to deal with async if it means not having to deal with uwsgi or other frontends. But in general I think Python has too many problems (packaging, performance, distribution, etc) that it doesn't make sense IMO to invest in new Python projects.
- 1337shadow 6y ago
  uWSGI is a lot of joy for me, really, I've never been happier with my deployments since I have discovered uWSGI back in 2008 or something, and nowadays it supports plenty of languages so there's just nothing I don't deploy on uWSGI anymore. Python packaging is something that I have fully automated (maintaining over 50 packages here) and that I'm pretty happy with. I fail to see the problem with Python packaging, maybe because I have an aggressive continuous integration practice ? (always integrate upstream changes, contribute to dependencies that I need, and when I'm not doing TDD it's only because I have not yet proof that the code I'm writing is not actually going to be useful) That's not something everybody wants to do (I don't understand their reasoning though). People would rather freeze their dependencies and then cry because upgrading is a lot of work, instead of upgrading at the rhythm of upstream releases. If other packages managers or other languages have packaging features that encourages what I consider to be non-continuous integration then good for them, but that's not how a hacker like me wants to work, being able to "ignore upstream releases" is not a good feature, it made me a sad developer really, "ignoring non-latest releases" have made me a really happy developer. Most performance issues are not imputable to the language. If they are, it's probably not affecting all your features, you can still rewrite the feature that Python is not well performing for into a compiled language. I need most of my code to be easy to manipulate, and very little of it to actually outperform Python. I've recently re-assessed if I should keep going with Python for another 10 years, tried a bunch of languages, frameworks, at the end of the month I still wanted a language that easy to manipulate with basic text tools, that's sufficiently easy so that I can onboard junior collegues on my tools, that provides sufficiently advanced OOP because I find it efficient to structure and reuse code. Python does what it claims, it solves a basic human-computer problem, let's face it: it's here to stay and shine, and its wide ecosystem seems like a solid proof. Wether it makes sense to invest in a project or not should not depend in the language anyway.
  - throwaway894345 6y ago
    > uWSGI is a lot of joy for me, there's nothing I don't deploy on uWSGI, even PHP code. Oh man, we moved away from uwsgi to async a couple of years ago and that's been one of the best decisions we've made. Async is no walk in the park, but not having to deal with uwsgi configuration, etc has been well worth it. > Python packaging is something that I have fully automated (maintaining over 50 packages here) and that I'm pretty happy with. Yeah, I don't doubt this. Many people have found a happy path that works for them, but I've found that those tend to be people who don't have significant constraints (e.g., they don't need fast builds, or they don't care about reproducibility, or they don't have to deal with a large number of regular contributors, or etc). > Most performance issues are not imputable to the language. This isn't true in a meaningful sense. For the most part, if you're doing anything more complicated than a CRUD app, you will run into performance problems with Python almost immediately upon leaving the prototype phase, and your main options for improving performance are horizontal scaling (multiprocess/multihost parallelism) or rewriting the hot path in a faster language. As previously discussed, these options only work for certain use cases where the ratio of de/serialization to real work is low, so you often find yourself without options. Further, horizontal scaling is expensive (compute is expensive) and rewriting in a different language is differently expensive (you now have to integrate a separate build system and employ developers who are not only well-versed in the new language, but also in implementing Python extensions specifically). On the other hand, if you chose a language like Go, you would be in the same ballpark of maintainability, onboarding, etc (many would argue Go is easier to write and maintain due to simplicity and static typing) but you would be in a much better place with respect to packaging and performance. You likely wouldn't need to optimize anything since naive Go tends to be 10-100X faster than naive Python, and if you needed to optimize, you can do so in-language without paying any sort of de/serialization overhead (parallelism, memory management, etc), allowing you to eek out another magnitude of performance. There are other options besides Go that also give performance gains, but they often involve trading off simplicity/packaging/deployment/tooling/ecosystem/etc. > If they are, it's probably not affecting all your features, you can still rewrite the feature that Python is not well performing for into a compiled language. This is true, but "rewriting features" is usually prohibitively expensive, and it's often non-trivial to figure out up-front which features will have performance problems in the future such that you could otherwise avoid a rewrite. > Python does what it claims that's a basic human problem, let's face it: it's here to stay and shine. Yes, Python is here to stay, but that's more attributable to network effects and misinformation than merit in my experience.
1337shadow 6y ago
> I think the situation in terms of public mindshare is much worse now than it was in 2015 Ok in 2015 it was a pain but with Python 3.8 it's actually a only joy & fun in my opinion. > the balkanisation of the community could well be more problematic than 2 vs 3 If you could call Python2 code from Python3 or vice-versa as easily as you can do with async then it would be comparable.
zzzeek 6y ago
> One of my worries is that right now it feels like everything in Python is being rewritten in asyncio and the balkanisation of the community could well be more problematic than 2 vs 3. this is exactly why the issue is so concerning for me as well.

throwaway894345 6y ago

For me it's worth the effort to deal with async if it means not having to deal with uwsgi or other frontends. But in general I think Python has too many problems (packaging, performance, distribution, etc) that it doesn't make sense IMO to invest in new Python projects.

1337shadow 6y ago
uWSGI is a lot of joy for me, really, I've never been happier with my deployments since I have discovered uWSGI back in 2008 or something, and nowadays it supports plenty of languages so there's just nothing I don't deploy on uWSGI anymore. Python packaging is something that I have fully automated (maintaining over 50 packages here) and that I'm pretty happy with. I fail to see the problem with Python packaging, maybe because I have an aggressive continuous integration practice ? (always integrate upstream changes, contribute to dependencies that I need, and when I'm not doing TDD it's only because I have not yet proof that the code I'm writing is not actually going to be useful) That's not something everybody wants to do (I don't understand their reasoning though). People would rather freeze their dependencies and then cry because upgrading is a lot of work, instead of upgrading at the rhythm of upstream releases. If other packages managers or other languages have packaging features that encourages what I consider to be non-continuous integration then good for them, but that's not how a hacker like me wants to work, being able to "ignore upstream releases" is not a good feature, it made me a sad developer really, "ignoring non-latest releases" have made me a really happy developer. Most performance issues are not imputable to the language. If they are, it's probably not affecting all your features, you can still rewrite the feature that Python is not well performing for into a compiled language. I need most of my code to be easy to manipulate, and very little of it to actually outperform Python. I've recently re-assessed if I should keep going with Python for another 10 years, tried a bunch of languages, frameworks, at the end of the month I still wanted a language that easy to manipulate with basic text tools, that's sufficiently easy so that I can onboard junior collegues on my tools, that provides sufficiently advanced OOP because I find it efficient to structure and reuse code. Python does what it claims, it solves a basic human-computer problem, let's face it: it's here to stay and shine, and its wide ecosystem seems like a solid proof. Wether it makes sense to invest in a project or not should not depend in the language anyway.
- throwaway894345 6y ago
  > uWSGI is a lot of joy for me, there's nothing I don't deploy on uWSGI, even PHP code. Oh man, we moved away from uwsgi to async a couple of years ago and that's been one of the best decisions we've made. Async is no walk in the park, but not having to deal with uwsgi configuration, etc has been well worth it. > Python packaging is something that I have fully automated (maintaining over 50 packages here) and that I'm pretty happy with. Yeah, I don't doubt this. Many people have found a happy path that works for them, but I've found that those tend to be people who don't have significant constraints (e.g., they don't need fast builds, or they don't care about reproducibility, or they don't have to deal with a large number of regular contributors, or etc). > Most performance issues are not imputable to the language. This isn't true in a meaningful sense. For the most part, if you're doing anything more complicated than a CRUD app, you will run into performance problems with Python almost immediately upon leaving the prototype phase, and your main options for improving performance are horizontal scaling (multiprocess/multihost parallelism) or rewriting the hot path in a faster language. As previously discussed, these options only work for certain use cases where the ratio of de/serialization to real work is low, so you often find yourself without options. Further, horizontal scaling is expensive (compute is expensive) and rewriting in a different language is differently expensive (you now have to integrate a separate build system and employ developers who are not only well-versed in the new language, but also in implementing Python extensions specifically). On the other hand, if you chose a language like Go, you would be in the same ballpark of maintainability, onboarding, etc (many would argue Go is easier to write and maintain due to simplicity and static typing) but you would be in a much better place with respect to packaging and performance. You likely wouldn't need to optimize anything since naive Go tends to be 10-100X faster than naive Python, and if you needed to optimize, you can do so in-language without paying any sort of de/serialization overhead (parallelism, memory management, etc), allowing you to eek out another magnitude of performance. There are other options besides Go that also give performance gains, but they often involve trading off simplicity/packaging/deployment/tooling/ecosystem/etc. > If they are, it's probably not affecting all your features, you can still rewrite the feature that Python is not well performing for into a compiled language. This is true, but "rewriting features" is usually prohibitively expensive, and it's often non-trivial to figure out up-front which features will have performance problems in the future such that you could otherwise avoid a rewrite. > Python does what it claims that's a basic human problem, let's face it: it's here to stay and shine. Yes, Python is here to stay, but that's more attributable to network effects and misinformation than merit in my experience.

1337shadow 6y ago

> I think the situation in terms of public mindshare is much worse now than it was in 2015 Ok in 2015 it was a pain but with Python 3.8 it's actually a only joy & fun in my opinion. > the balkanisation of the community could well be more problematic than 2 vs 3 If you could call Python2 code from Python3 or vice-versa as easily as you can do with async then it would be comparable.

zzzeek 6y ago

> One of my worries is that right now it feels like everything in Python is being rewritten in asyncio and the balkanisation of the community could well be more problematic than 2 vs 3. this is exactly why the issue is so concerning for me as well.