5 ms·

I work at Instagram, which is a O(millions) LOC Python monorepo with massive throughput and a large engineering team. It's actually quite nice — but our code is

I work at Instagram, which is a O(millions) LOC Python monorepo with massive throughput and a large engineering team. It's actually quite nice — but our code is heavily, heavily typed. It would be miserable without the type system. Some of the older parts of the codebase are more loosely typed (although they're shrinking reasonably quickly), and those sections are indeed a huge pain to ramp up on. Part of the success of IG's large Python codebase is FB's investment into developer tooling; for example, FB wrote (and open-sourced, FWIW) our own PEP-484 compliant type checker, Pyre [1], because mypy was too slow for a codebase of our size. 1: https://github.com/facebook/pyre-check https://github.com/facebook/pyre-check

david-gpu 6y ago
Have you guys published any whitepaper on this subject? These last few years working on moderately large python codebases with dynamic typing have been less than idilic.
- wilsonthewhale 6y ago
  Here's a small collection https://pyre-check.org/docs/pysa-additional-resources https://pyre-check.org/docs/pysa-additional-resources
anonymouse008 6y ago
Yeah, you could use my hairline to tell when I’m working with Python... I still haven’t figured out a good typed framework for it yet.
- kevinarpe 6y ago
  I also struggle with weak(er) typing in Python, compared to strong(er) typing in C++, Java, or C# -- or even VBA(!). Frustrated like you, I wrote my own open source type checking library for Python. You might be interested to read about it here: https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master/rambutan3/check_args/RCheckArgs.py#L149 https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master... I have used that library on multiple projects for my job. It makes the code run about 50% slower, on average, because all the type checking is done at run-time. I am OK with the slow down because I don't use Python when I need speed. My "developer speed" was greatly improved with stricter types. Finally, this isn't the first time I wrote a type checking library/framework. I did the same for Perl more than 10yrs ago. Unfortunately, that code is proprietary, so not open source. :( The abstract ideas were very similar. Our team was so frustrated with legacy Perl code, so I wrote a type checking library, and we slowly applied it to the code base. About two years later, it was much less painful!
- reissbaker 6y ago
  Try using Pyre! It's open-source. I use it daily at IG.
JAlexoid 6y ago
That's my major complaint about Python... For it's age and popularity - the tooling is abysmal. I have to rewrite too many things, that I expected to just be there for the age of the project. And some things were only fixed now! dict + dict only started to work with 3.9!
- nomdep 6y ago
  dict1 + dict2 is syntactic sugar, you had been able to do `dict1.update(dict2)` since forever.
  - HelloNurse 6y ago
    No, it's completely different: dict1+dict2 creates a new object and leaves inputs unchanged, dict1.update(dict2) modifies dict1.
    - slaman 6y ago
      you've been able to do splat for forever! https://dpaste.org/yU7s https://dpaste.org/yU7s

david-gpu 6y ago

Have you guys published any whitepaper on this subject? These last few years working on moderately large python codebases with dynamic typing have been less than idilic.

wilsonthewhale 6y ago
Here's a small collection https://pyre-check.org/docs/pysa-additional-resources https://pyre-check.org/docs/pysa-additional-resources

anonymouse008 6y ago

Yeah, you could use my hairline to tell when I’m working with Python... I still haven’t figured out a good typed framework for it yet.

kevinarpe 6y ago
I also struggle with weak(er) typing in Python, compared to strong(er) typing in C++, Java, or C# -- or even VBA(!). Frustrated like you, I wrote my own open source type checking library for Python. You might be interested to read about it here: https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master/rambutan3/check_args/RCheckArgs.py#L149 https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master... I have used that library on multiple projects for my job. It makes the code run about 50% slower, on average, because all the type checking is done at run-time. I am OK with the slow down because I don't use Python when I need speed. My "developer speed" was greatly improved with stricter types. Finally, this isn't the first time I wrote a type checking library/framework. I did the same for Perl more than 10yrs ago. Unfortunately, that code is proprietary, so not open source. :( The abstract ideas were very similar. Our team was so frustrated with legacy Perl code, so I wrote a type checking library, and we slowly applied it to the code base. About two years later, it was much less painful!
reissbaker 6y ago
Try using Pyre! It's open-source. I use it daily at IG.

JAlexoid 6y ago

That's my major complaint about Python... For it's age and popularity - the tooling is abysmal. I have to rewrite too many things, that I expected to just be there for the age of the project. And some things were only fixed now! dict + dict only started to work with 3.9!

nomdep 6y ago
dict1 + dict2 is syntactic sugar, you had been able to do `dict1.update(dict2)` since forever.
- HelloNurse 6y ago
  No, it's completely different: dict1+dict2 creates a new object and leaves inputs unchanged, dict1.update(dict2) modifies dict1.
  - slaman 6y ago
    you've been able to do splat for forever! https://dpaste.org/yU7s https://dpaste.org/yU7s