8 ms·

Have you ever seen O(million) lines enterprise codebase that didn't suck?

by d0mine 6y ago

MR4D 6y ago
LOL!! Or even worse - could you imagine how many lines that would be in C++ ? Yowza!
- oarabbus_ 6y ago
  n!
lrem 6y ago
I've worked on several separate projects of that size in C++ and Go. None of them seemed to achieve a similar mess as Python codebases with one or two dozen thousand lines seem to. OTOH, all the typing developments in Python should have helped? I don't have that much experience with them in enterprise setting.
zo1 6y ago
I have - and it's not that bad. The key is you have to have someone coordinating and driving a shared vision for the codebase and patterns. But it's hard to find people with that sort of passion and drive to follow-through as it's a multi-year endeavor with politics all over. Otherwise its a thousand implementations of the same 100-line piece of code interspersed everywhere.
- ineedasername 6y ago
  It seems like quality code management gets passed over by (bad) management because it looks like it doesn't directly move the project forward. Which is strange because those same managers may be full adherents to micro tasking projects in a project management system whose purpose is basically to do for the project what code management does for the code itself. In my workplace, we've recently had leadership that appreciates these things, and the difference is night & day. Simple requests from "stakeholders" (I hate that term) are often filled in days, or same day, instead of weeks. I think it helps tremendously that the primary manager is also a coder herself, and still codes ~25% of her job.
- JAlexoid 6y ago
  That's the problem with some languages - they lack a visionary, that drives the overall understanding of how things should be structured. I believe it's Guido that basically said - if you don't like how Python does it, then implement it in C. And that's how you end up with great C based libraries bound to python... and python is often used as a messy orchestrating language.
gordaco 6y ago
This is surely anecdotic and very subjective, but I have (in Java and in C++; IIRC the exact versions were Java 7 and C++03), and the level of pain was lower than with a Python code base that was about one order of magnitude smaller. In the case of C++, the pain was mostly asociated with an ancient build system we used; the code itself was relatively manageable. There was almost zero template code, and maybe that helped (although in other occasions I've worked with smaller C++03 codebases that relied heavily on templates and I didn't find them that bad). Not all codebases are equal and maybe I was lucky, but in my experience, using dynamic languages (or, to be exact, any language where the compiler doesn't nag you when there is a potential problem) doesn't scale well.
commonturtle 6y ago
I've worked with an O(100k) line code base in Python that was pure torture. Honestly, I was so desperate for static-typing by the end that I would have preferred if it was all written in C++. Large codebases are really hard reason about without types. I'm glad we now have projects like Pyre that are trying to bring typing to Python.
- ineedasername 6y ago
  > desperate for static-typing Can you explain why? I honestly don't know, because my experience with C++ was during school ~20 years ago, and since then professionally I've used mostly python in relatively small codebases where it's all my own code (mostly for data processing/analysis). Thanks! (Although I did have to write some C code to glue together data in a very old legacy system that didn't support C++, much less python. It took a lot more effort to do something simple, but it was also strangely a really rewarding experience. Kind of similar to feeling when I work with assembly on hobby projects)
  - johntb86 6y ago
    The main problem with duck-typing like python has is the lack of consistency between different objects that code has to work on. Different callers may pass objects with different sets of methods into a function and expect it to work. You run into the case where the object that was passed in is one with subtly-mismatched behavior from what your method expects, but you don't know who created it - it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on. Static typing prevents that by telling you early where the mismatch is happening - some method calls into another with a variable of the wrong type, and that's where the bug is. It also allows tooling to look up the types of variables and quickly get information about their properties.
    - ineedasername 6y ago
      Got it, that makes sense. It also makes sense why I've not much been bothered by it in python since my relative small code bases don't have that many layers of abstraction laid on top of each other. I'm generally not working with more than 2,000-3,000 lines, and I can just about keep the basic structure in my head. (Unless it's been a while since I've had to revisit it... then I often hate my past self for getting "clever" in some way)
JAlexoid 6y ago
I am literally working on two projects that are roughly 100kLOC each. The Scala Spark project I can navigate, understand, test and consider to be average complexity... with some failures, unique to Scala. The Python Spark project is barely readable. People who built the Python Spark codebase are "experienced Python devs". While Scala codebase was built by people who used Scala for the first time. (take this anecdote, as evidence for the poor tooling and guidance present in python community.... and BDFL's own failures)

Have you ever seen O(million) lines enterprise codebase that didn't suck?

by d0mine 6y ago

Have you ever seen O(million) lines enterprise codebase that didn't suck?

MR4D 6y ago

LOL!! Or even worse - could you imagine how many lines that would be in C++ ? Yowza!

oarabbus_ 6y ago
n!

lrem 6y ago

I've worked on several separate projects of that size in C++ and Go. None of them seemed to achieve a similar mess as Python codebases with one or two dozen thousand lines seem to. OTOH, all the typing developments in Python should have helped? I don't have that much experience with them in enterprise setting.

zo1 6y ago

I have - and it's not that bad. The key is you have to have someone coordinating and driving a shared vision for the codebase and patterns. But it's hard to find people with that sort of passion and drive to follow-through as it's a multi-year endeavor with politics all over. Otherwise its a thousand implementations of the same 100-line piece of code interspersed everywhere.

ineedasername 6y ago
It seems like quality code management gets passed over by (bad) management because it looks like it doesn't directly move the project forward. Which is strange because those same managers may be full adherents to micro tasking projects in a project management system whose purpose is basically to do for the project what code management does for the code itself. In my workplace, we've recently had leadership that appreciates these things, and the difference is night & day. Simple requests from "stakeholders" (I hate that term) are often filled in days, or same day, instead of weeks. I think it helps tremendously that the primary manager is also a coder herself, and still codes ~25% of her job.
JAlexoid 6y ago
That's the problem with some languages - they lack a visionary, that drives the overall understanding of how things should be structured. I believe it's Guido that basically said - if you don't like how Python does it, then implement it in C. And that's how you end up with great C based libraries bound to python... and python is often used as a messy orchestrating language.

gordaco 6y ago

This is surely anecdotic and very subjective, but I have (in Java and in C++; IIRC the exact versions were Java 7 and C++03), and the level of pain was lower than with a Python code base that was about one order of magnitude smaller. In the case of C++, the pain was mostly asociated with an ancient build system we used; the code itself was relatively manageable. There was almost zero template code, and maybe that helped (although in other occasions I've worked with smaller C++03 codebases that relied heavily on templates and I didn't find them that bad). Not all codebases are equal and maybe I was lucky, but in my experience, using dynamic languages (or, to be exact, any language where the compiler doesn't nag you when there is a potential problem) doesn't scale well.

commonturtle 6y ago

I've worked with an O(100k) line code base in Python that was pure torture. Honestly, I was so desperate for static-typing by the end that I would have preferred if it was all written in C++. Large codebases are really hard reason about without types. I'm glad we now have projects like Pyre that are trying to bring typing to Python.

ineedasername 6y ago
> desperate for static-typing Can you explain why? I honestly don't know, because my experience with C++ was during school ~20 years ago, and since then professionally I've used mostly python in relatively small codebases where it's all my own code (mostly for data processing/analysis). Thanks! (Although I did have to write some C code to glue together data in a very old legacy system that didn't support C++, much less python. It took a lot more effort to do something simple, but it was also strangely a really rewarding experience. Kind of similar to feeling when I work with assembly on hobby projects)
- johntb86 6y ago
  The main problem with duck-typing like python has is the lack of consistency between different objects that code has to work on. Different callers may pass objects with different sets of methods into a function and expect it to work. You run into the case where the object that was passed in is one with subtly-mismatched behavior from what your method expects, but you don't know who created it - it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on. Static typing prevents that by telling you early where the mismatch is happening - some method calls into another with a variable of the wrong type, and that's where the bug is. It also allows tooling to look up the types of variables and quickly get information about their properties.
  - ineedasername 6y ago
    Got it, that makes sense. It also makes sense why I've not much been bothered by it in python since my relative small code bases don't have that many layers of abstraction laid on top of each other. I'm generally not working with more than 2,000-3,000 lines, and I can just about keep the basic structure in my head. (Unless it's been a while since I've had to revisit it... then I often hate my past self for getting "clever" in some way)

JAlexoid 6y ago

I am literally working on two projects that are roughly 100kLOC each. The Scala Spark project I can navigate, understand, test and consider to be average complexity... with some failures, unique to Scala. The Python Spark project is barely readable. People who built the Python Spark codebase are "experienced Python devs". While Scala codebase was built by people who used Scala for the first time. (take this anecdote, as evidence for the poor tooling and guidance present in python community.... and BDFL's own failures)