7 ms·

> try to maintain > 10k LOC codebases in Python to understand what I mean Lots of companies successfully manage far larger Python codebases. We have a 35M LOC

> try to maintain > 10k LOC codebases in Python to understand what I mean Lots of companies successfully manage far larger Python codebases. We have a 35M LOC Python codebase with 2500 active developers making 25000 commits per week and continuous deployment globally of new code. (I gave a talk on it at PyData London in April: "Python at Massive Scale")

mcintyre1994 8y ago
If anyone else is looking: https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.
- stevesimmons 8y ago
  The most concise answer is 'tooling'. We have a DevTools team that builds the SDLC and automated testing infra, all tightly integrated into MS Visual Studio. That lets the thousands of developers focus on productive coding.
  - mino 8y ago
    Just watched it. Thanks, it was indeed a great talk, and indeed a lot of tooling. Do you have any extra writeup to suggest?
    - stevesimmons 8y ago
      There isn't a ton of detailed public information on this kind of stuff. My talk focused on the general principles of our approach, rather than the myriad of features and optimizations necessary at such a large scale. A good next step would be reading the 2016 ACM paper "Why Google Stores Billions of Lines of Code in a Single Repository" [1] then reading a recent HN discussion [2]. That has a ton of good links. [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371
collyw 8y ago
Is there a quick way to find out your LOC in a python project? Does it include whitespace and comments? What about HTML and JS code?
- rusk 8y ago
  The typical measure is “Source Lines of Code” (SLOC), which would be calculated on a per-language basis and doesn’t include comments or blank lines. You’d sum these together for your codebase overall. Not sure how you’d do it for HTML or mixed code. Presumably the same since it’s such a rough measure that that level of accuracy really doesn’t matter.
- sacado2 8y ago
  I'm using tokei for that. EDIT: it gives you the number of lines of code, blank lines and comment lines, for each language in your project.
- shermanyo 8y ago
  cloc is another commonly used package. It handles multiple languages and has a nice, readable output format.
simonh 8y ago
I used to work on the subsequent brain child of Kirat Singh et al at another big financial institution, and ditto. Admittedly we used a custom built source control system and release tooling, but also developed in Python.
jernfrost 8y ago
Is this made as a cluster of independent programs or is this actually a monolith, where you can access all functionality from some python method or class?
- stevesimmons 8y ago
  To grossly oversimply: - Yes, it is a monolithic application architecture. There are distinct frameworks (e.g. GUIs, DBs, distributed compute, pubsub, lazily evaluated graph, reactive graphs, etc) designed to be used together in a unified way based on a global object store. - A 150 person platform dev team writes these core frameworks and dev tools and runs the SDLC infra. - Then the several thousand developers in business teams build applications for their particular needs. - When the platform team sees businesses building their own solutions to a common need, it will productionize that as a new core framework for all business to use. - In Python terms, there is a relative small number of core classes, and very many application-specific subclasses.
patagonia 8y ago
Perhaps paradoxically it is easier to maintain a gigantic Python code base than it is to maintain one that is small or medium sized. The reasoning, a gigantic code base means infrastructure and tooling investment will obviously pay off. It probably even means you get one or more dedicated resources of this sort. Whereas a small to medium sized codebase might bump into some of the same issues as a larger code base but not justify such investments.

mcintyre1994 8y ago

If anyone else is looking: https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.

stevesimmons 8y ago
The most concise answer is 'tooling'. We have a DevTools team that builds the SDLC and automated testing infra, all tightly integrated into MS Visual Studio. That lets the thousands of developers focus on productive coding.
- mino 8y ago
  Just watched it. Thanks, it was indeed a great talk, and indeed a lot of tooling. Do you have any extra writeup to suggest?
  - stevesimmons 8y ago
    There isn't a ton of detailed public information on this kind of stuff. My talk focused on the general principles of our approach, rather than the myriad of features and optimizations necessary at such a large scale. A good next step would be reading the 2016 ACM paper "Why Google Stores Billions of Lines of Code in a Single Repository" [1] then reading a recent HN discussion [2]. That has a ton of good links. [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371

collyw 8y ago

Is there a quick way to find out your LOC in a python project? Does it include whitespace and comments? What about HTML and JS code?

rusk 8y ago
The typical measure is “Source Lines of Code” (SLOC), which would be calculated on a per-language basis and doesn’t include comments or blank lines. You’d sum these together for your codebase overall. Not sure how you’d do it for HTML or mixed code. Presumably the same since it’s such a rough measure that that level of accuracy really doesn’t matter.
sacado2 8y ago
I'm using tokei for that. EDIT: it gives you the number of lines of code, blank lines and comment lines, for each language in your project.
shermanyo 8y ago
cloc is another commonly used package. It handles multiple languages and has a nice, readable output format.

simonh 8y ago

I used to work on the subsequent brain child of Kirat Singh et al at another big financial institution, and ditto. Admittedly we used a custom built source control system and release tooling, but also developed in Python.

jernfrost 8y ago

Is this made as a cluster of independent programs or is this actually a monolith, where you can access all functionality from some python method or class?

stevesimmons 8y ago
To grossly oversimply: - Yes, it is a monolithic application architecture. There are distinct frameworks (e.g. GUIs, DBs, distributed compute, pubsub, lazily evaluated graph, reactive graphs, etc) designed to be used together in a unified way based on a global object store. - A 150 person platform dev team writes these core frameworks and dev tools and runs the SDLC infra. - Then the several thousand developers in business teams build applications for their particular needs. - When the platform team sees businesses building their own solutions to a common need, it will productionize that as a new core framework for all business to use. - In Python terms, there is a relative small number of core classes, and very many application-specific subclasses.

patagonia 8y ago

Perhaps paradoxically it is easier to maintain a gigantic Python code base than it is to maintain one that is small or medium sized. The reasoning, a gigantic code base means infrastructure and tooling investment will obviously pay off. It probably even means you get one or more dedicated resources of this sort. Whereas a small to medium sized codebase might bump into some of the same issues as a larger code base but not justify such investments.