5 ms·

My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far. I wonder how Meta (and Go

by Yoric 3y ago

My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far. I wonder how Meta (and Google, and all the others who write serious Python code) manage to keep things running – and I'm not talking (only) of performance, but largely of bugs. My personal suspicion is that the secret sauce is simply to have a sufficient budget to hire an army of code quality engineers. I could be wrong, though.

ambivalence 3y ago
In short: static analysis, unit tests, staging environments, effective log aggregation. The first one is a big productivity booster as it shows you bugs before you commit them. Here's a post about this, not exactly new but still describing the general principles very well: https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c https://instagram-engineering.com/static-analysis-at-scale-a...
- VWWHFSfQ 3y ago
  I would say that those things don't really contribute to the actual runtime scalability of the application. Those things ensure that the code is at least correct for what it's supposed to do, not necessarily whether it's performant. Even the most perfectly-written Python code will still be dog-slow and require an enormous budget to actually run at scale. It's obviously fine to do that and there are many success stories, but all of them required massive infrastructure investment.
  - ambivalence 3y ago
    > Even the most perfectly-written Python code will still be dog-slow and require an enormous budget to actually run at scale. Nah, asyncio on uvloop is plenty fast.
  - Yoric 3y ago
    For context, I was asking about scaling code complexity, not scaling performance. That being said, my personal experience suggests that both are really hard in Python, and not independent.
- deleted 3y ago
  [deleted]
- Yoric 3y ago
  Thanks. For context, I happen to have a PhD in static analysis, so I'm not entirely stranger to the general idea :) Sadly, all the attempts I've had with static analysis in Python screamed that while the language and tools make a valiant effort at supporting a reasonable set of annotations, 8-9 years after PEP 484, the libraries are simply not yet ready for it (not even, in many cases, the standard library). Unit tests, staging environment, effective log aggregation are all important tools, of course.
code_runner 3y ago
Tests is likely a large part of it. There are some great tools for code formatting, quality, and type checking. It’s a very flexible language… When people are hyper critical of Python I feel it’s typically they haven’t seen real professional Python before and just throw scripts around… if you work with some hardened Python pros you pick up the tricks really quickly and it’s very enlightening. I’ve always struggled to find all of those tricks in one online resource personally.
- epgui 3y ago
  Every language will have its gurus and its own peculiar with-the-grain approach, but the python language has a ton of non-essential complexity baked into its very design/core (especially for a language that people seem to think is "simple"-- it's not, not at all; it's a language full of "pragmatic compromises", apparently by intentional philosophy), which places fundamental limits on how safe your application can be.
o1y32 3y ago
I cannot find any substance in your comment. At minimum it should provide some rough examples instead of simply "underwhelming".
- Yoric 3y ago
  Well, I wasn't planning to spend time criticizing Python. I think that we all know roughly its pros and cons, no need to add fire. My questions remain, though.
marcinzm 3y ago
You can write crap code in any language. To avoid it you need to enforce some decent set of best practices for that specific language. That also means not using best practices from language X in language Y and then complaining that language Y is crap.
- Yoric 3y ago
  Yes. And one way to rephrase my question is: which best practices work for these Python mega-projects?
hospitalJail 3y ago
>My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far You have years of experience but couldn't point to anything in particular? Heck, I like python and I can complain about dynamic typing issues in for loops, or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the zen of python. (But I still think python is great)
- Yoric 3y ago
  Why would I need to point to anything in particular? I think that all Python developers know them already.
- antod 3y ago
  > or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the zen of python. Those features are 20ish years old which made the word "adding" seem a little weird. Interestingly the Zen of Python itself is not much older than generators.
smrtinsert 3y ago
The secret sauce is Intellij and some community knowledge. There's a library for everything and a great IDE makes it pretty painless and fun. Not sure I'd always jump for it thanks to the GIL but I wouldn't mind a job where it's my primary instead of my secondary or third language. The biggest pain with it is of course refactoring. It's tedious, but I think still comes out ahead in terms of productivity for high level stuff.

by Yoric 3y ago

ambivalence 3y ago

In short: static analysis, unit tests, staging environments, effective log aggregation. The first one is a big productivity booster as it shows you bugs before you commit them. Here's a post about this, not exactly new but still describing the general principles very well: https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c https://instagram-engineering.com/static-analysis-at-scale-a...

VWWHFSfQ 3y ago
I would say that those things don't really contribute to the actual runtime scalability of the application. Those things ensure that the code is at least correct for what it's supposed to do, not necessarily whether it's performant. Even the most perfectly-written Python code will still be dog-slow and require an enormous budget to actually run at scale. It's obviously fine to do that and there are many success stories, but all of them required massive infrastructure investment.
- ambivalence 3y ago
  > Even the most perfectly-written Python code will still be dog-slow and require an enormous budget to actually run at scale. Nah, asyncio on uvloop is plenty fast.
- Yoric 3y ago
  For context, I was asking about scaling code complexity, not scaling performance. That being said, my personal experience suggests that both are really hard in Python, and not independent.
deleted 3y ago
[deleted]
Yoric 3y ago
Thanks. For context, I happen to have a PhD in static analysis, so I'm not entirely stranger to the general idea :) Sadly, all the attempts I've had with static analysis in Python screamed that while the language and tools make a valiant effort at supporting a reasonable set of annotations, 8-9 years after PEP 484, the libraries are simply not yet ready for it (not even, in many cases, the standard library). Unit tests, staging environment, effective log aggregation are all important tools, of course.

code_runner 3y ago

Tests is likely a large part of it. There are some great tools for code formatting, quality, and type checking. It’s a very flexible language… When people are hyper critical of Python I feel it’s typically they haven’t seen real professional Python before and just throw scripts around… if you work with some hardened Python pros you pick up the tricks really quickly and it’s very enlightening. I’ve always struggled to find all of those tricks in one online resource personally.

epgui 3y ago
Every language will have its gurus and its own peculiar with-the-grain approach, but the python language has a ton of non-essential complexity baked into its very design/core (especially for a language that people seem to think is "simple"-- it's not, not at all; it's a language full of "pragmatic compromises", apparently by intentional philosophy), which places fundamental limits on how safe your application can be.

o1y32 3y ago

I cannot find any substance in your comment. At minimum it should provide some rough examples instead of simply "underwhelming".

Yoric 3y ago
Well, I wasn't planning to spend time criticizing Python. I think that we all know roughly its pros and cons, no need to add fire. My questions remain, though.

marcinzm 3y ago

You can write crap code in any language. To avoid it you need to enforce some decent set of best practices for that specific language. That also means not using best practices from language X in language Y and then complaining that language Y is crap.

Yoric 3y ago
Yes. And one way to rephrase my question is: which best practices work for these Python mega-projects?

hospitalJail 3y ago

>My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far You have years of experience but couldn't point to anything in particular? Heck, I like python and I can complain about dynamic typing issues in for loops, or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the zen of python. (But I still think python is great)

Yoric 3y ago
Why would I need to point to anything in particular? I think that all Python developers know them already.
antod 3y ago
> or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the zen of python. Those features are 20ish years old which made the word "adding" seem a little weird. Interestingly the Zen of Python itself is not much older than generators.

smrtinsert 3y ago

The secret sauce is Intellij and some community knowledge. There's a library for everything and a great IDE makes it pretty painless and fun. Not sure I'd always jump for it thanks to the GIL but I wouldn't mind a job where it's my primary instead of my secondary or third language. The biggest pain with it is of course refactoring. It's tedious, but I think still comes out ahead in terms of productivity for high level stuff.