4 ms·

I disagree with the idea that you can't maintain a robust system/codebase in python, mainly because nearly our entire distributed computing stack is written in

I disagree with the idea that you can't maintain a robust system/codebase in python, mainly because nearly our entire distributed computing stack is written in python. Granted it isn't 10s of thousands of lines, but it isn't small either. It does a lot of things ranging from webapis to data pipelining with ingestion and egestion of large datasets. One batch job can result in 10s of GBs in output files. It has to be performant too, 10s of millions of "operations" in one batch job is common. Admittedly the process/threading model can be painful to work around at times, but totally doable, and is honestly somewhat friendly to distributed systems to an extent because it somewhat forces you into an actor model paradigm.

mlazos 7y ago
This is a little off topic but is the performance mainly coming from the computation being distributed? I wrote this text parser in python a few years ago, rewrote the same thing in Java and got like a 10x speedup. It’s possible I’m a bad python programmer.
- iudqnolq 7y ago
  Also there's a general effect that any rewrite can improve the code quality. Same thing might have happened if you rewrote in Python.
iamaelephant 7y ago
> Granted it isn't 10s of thousands of lines, but it isn't small either. I'm not trying to belittle what you do, but if your good base is in the 4 figures in size then it's not large (in fact yes, it is small). When people talk about large code bases they are generally talking 100-1000x larger than yours, or more.
- jimbokun 7y ago
  True, but maybe that's partly because it's an order of magnitude smaller than the same program written in Java?
  - eropple 7y ago
    You're missing one or two zeroes from the post to which you are replying.
- TheFiend7 7y ago
  It sits on top of our own IP database which is 6 figures in lines and is written in C++. Java has fancy bells and whistles when it comes to IDE's, but that's all they are. I just think it's foolish to think Java is the only language that can handle large codebases. I mean just look at the linux kernel. EDIT: Removed comment about processing capabilities because it's irrelevant to the discussion. I left my original comments about lines and the IDE though.
  - mlazos 7y ago
    I don’t think anyone is saying Java is the only one they’re saying because it has good IDE support it is better than many other languages, e.g. python. I tend to agree. Once a python program gets over 20k lines (I touched a program like that before) it’s an absolute nightmare to maintain and without type annotations can be near impossible to figure out what is going on.
  - nikanj 7y ago
    The size of data processed, or the number of nodes running, has almost no effect on the quality of the IDEs.
- LandR 7y ago
  This is true. At my last job our primary codebase was around 5 million Loc split over 400+ assemblies. Current job I've seen single files that are over 30k lines.

mlazos 7y ago

This is a little off topic but is the performance mainly coming from the computation being distributed? I wrote this text parser in python a few years ago, rewrote the same thing in Java and got like a 10x speedup. It’s possible I’m a bad python programmer.

iudqnolq 7y ago
Also there's a general effect that any rewrite can improve the code quality. Same thing might have happened if you rewrote in Python.

iamaelephant 7y ago

> Granted it isn't 10s of thousands of lines, but it isn't small either. I'm not trying to belittle what you do, but if your good base is in the 4 figures in size then it's not large (in fact yes, it is small). When people talk about large code bases they are generally talking 100-1000x larger than yours, or more.

jimbokun 7y ago
True, but maybe that's partly because it's an order of magnitude smaller than the same program written in Java?
- eropple 7y ago
  You're missing one or two zeroes from the post to which you are replying.
TheFiend7 7y ago
It sits on top of our own IP database which is 6 figures in lines and is written in C++. Java has fancy bells and whistles when it comes to IDE's, but that's all they are. I just think it's foolish to think Java is the only language that can handle large codebases. I mean just look at the linux kernel. EDIT: Removed comment about processing capabilities because it's irrelevant to the discussion. I left my original comments about lines and the IDE though.
- mlazos 7y ago
  I don’t think anyone is saying Java is the only one they’re saying because it has good IDE support it is better than many other languages, e.g. python. I tend to agree. Once a python program gets over 20k lines (I touched a program like that before) it’s an absolute nightmare to maintain and without type annotations can be near impossible to figure out what is going on.
- nikanj 7y ago
  The size of data processed, or the number of nodes running, has almost no effect on the quality of the IDEs.
LandR 7y ago
This is true. At my last job our primary codebase was around 5 million Loc split over 400+ assemblies. Current job I've seen single files that are over 30k lines.