3 ms·

Have you ever worked on a 1M line code base? Or with dozens (or hundreds!) of developers. Once the team gets big, or the code gets so big you can't hold it all

by dons 15y ago

Have you ever worked on a 1M line code base? Or with dozens (or hundreds!) of developers. Once the team gets big, or the code gets so big you can't hold it all in your head, any machine-enforced QC can have a major impact. (I'm working on a very large Haskell code base now -- the parts that tend to cause trouble are the dynamically typed bits -- they can just go wrong in so many unanticipated ways, and the complexity means its almost unavoidable that devs miss things). I'm increasingly convinced that to wrangle massive code bases, you need both heavy abstraction abilities (higher order functions, DSLs, algebraic data types, polymorphism), and very strong typing, to keep the complexity under control

rbanffy 15y ago
A 1M line codebase can mean many things. It can mean code that belongs to many systems mashed together as if it were a single thing. It can mean that the code that should belong to different systems is tightly coupled into one monolithic entity that should have been several little entities. It can mean lots of boilerplate code too. It can indicate a lack of timely refactoring on an aging codebase so interconnected nobody has the courage to separate in more manageable pieces. None of those can be solved by clever language choices alone. You also mention things going wrong in unanticipated ways - this may signal that the problematic code where errors bubble up is not problematic at all - it is called by code written by people who don't really understand what the functions do and who probably didn't write adequate tests for that - because the tests should catch the unanticipated parts. The problematic code is the one calling the parts where errors bubble up. The canary is not responsible for the gases in the mine. While you may be right that, in order to deal with multi-million-line codebases you need static typing, I'd much rather split that codebase into smaller units that could be more easily managed. Wearing a straitjacket is often a consequence of an underlying condition that can, sometimes, be corrected.
- route66 15y ago
  I would expect that dons is talking about a 1M codebase which already consists out of manageable pieces. It's only, that the pieces have to work together, talk to each other, know about each other (but not too much). Sometimes software solves problems which provoke incidental complexity because of sheer size and my experience (albeit not above 500k) tells me that indeed, all bits help, also compiler enforced type checks. I would never bet my life on tests. As you write: "because the tests should catch the unanticipated parts", that's the point: tests never catch unanticipated parts by their very nature. Sometimes, by sheer luck, yes.
  - rbanffy 15y ago
    > tests never catch unanticipated parts No, but calling a function with arguments of the wrong type is something tests would catch.
    - maximilianburke 15y ago
      > No, but calling a function with arguments of the wrong type is something tests would catch. Or the compiler.
      - rbanffy 15y ago
        A compiler can only go so far. It'll happily compile: #include <stdio.h> int main(int argc, char *argv[]) { FILE *fp = fopen("outfile", "w"); fclose(fp); do_something(fp); return 0; } int do_something(FILE *f) { fprintf(f, "This should work if you know what you're doing"); } And it won't work.
        maximilianburke 15y ago
        In C++, proper scoping with RAII (similar to CL's with-blah idioms) would alleviate that issue as do_something in the code below won't compile. struct WithOpenFile { FILE *fp; WithOpenFile(const char *fileName) : fp(fopen(fileName, "w")) {} ~WithOpenFile() { fclose(fp); } }; int do_something(FILE *f) { fprintf(f, "This should work if you know what you're doing"); } int main(int argc, char *argv[]) { { WithOpenFile file("outfile"); } do_something(file.fp); return 0; } But Lisps will happily accept this: (defun do-something (foo) (1+ foo)) (with-open-file (file "outfile") (do-something file))
        rbanffy 15y ago
        Fine. But now you'll have to accept both files and network sockets. What do you do?
pavelludiq 15y ago
The biggest programs I've worked on are way smaller than 1MLOC, but I've dug around a few large code-bases written in dynamic languages(emacs most notably, 1M lines of Lisp, 1/4M lines of C), and my unproven theory of large systems is that it might be a good idea to make mistakes have smaller impact, rather than try to avoid them(they are inevitable), by focusing on better abstractions, and thus keeping problems more local, focusing on the protocols rather than on the implementation. Static typing might help here, but designing a protocol/interface/api/dsl is hard and I need to experiment a lot with it, so I'm willing to sacrifice some safety for better overall design and flexibility(which will hopefully make debugging easier). IOW I agree you need DSLs, polymorphism and all that good jazz. Where my disagreement comes in is that It's a trade off, and not necessarily an absolute need to have a lot of static typing. In my personal work I'm willing to make the trade off and work with clay rather than marble. Other people work and think differently and I accept that; absolute truths are rare in computer science, and don't exist in software engineering, its all trade-offs here :) ps My hobby vaporware CL project will probably be called mudball, I'm trying to learn how to design systems with hackability and extreme flexibility in mind :)

by dons 15y ago

rbanffy 15y ago

A 1M line codebase can mean many things. It can mean code that belongs to many systems mashed together as if it were a single thing. It can mean that the code that should belong to different systems is tightly coupled into one monolithic entity that should have been several little entities. It can mean lots of boilerplate code too. It can indicate a lack of timely refactoring on an aging codebase so interconnected nobody has the courage to separate in more manageable pieces. None of those can be solved by clever language choices alone. You also mention things going wrong in unanticipated ways - this may signal that the problematic code where errors bubble up is not problematic at all - it is called by code written by people who don't really understand what the functions do and who probably didn't write adequate tests for that - because the tests should catch the unanticipated parts. The problematic code is the one calling the parts where errors bubble up. The canary is not responsible for the gases in the mine. While you may be right that, in order to deal with multi-million-line codebases you need static typing, I'd much rather split that codebase into smaller units that could be more easily managed. Wearing a straitjacket is often a consequence of an underlying condition that can, sometimes, be corrected.

route66 15y ago
I would expect that dons is talking about a 1M codebase which already consists out of manageable pieces. It's only, that the pieces have to work together, talk to each other, know about each other (but not too much). Sometimes software solves problems which provoke incidental complexity because of sheer size and my experience (albeit not above 500k) tells me that indeed, all bits help, also compiler enforced type checks. I would never bet my life on tests. As you write: "because the tests should catch the unanticipated parts", that's the point: tests never catch unanticipated parts by their very nature. Sometimes, by sheer luck, yes.
- rbanffy 15y ago
  > tests never catch unanticipated parts No, but calling a function with arguments of the wrong type is something tests would catch.
  - maximilianburke 15y ago
    > No, but calling a function with arguments of the wrong type is something tests would catch. Or the compiler.
    - rbanffy 15y ago
      A compiler can only go so far. It'll happily compile: #include <stdio.h> int main(int argc, char *argv[]) { FILE *fp = fopen("outfile", "w"); fclose(fp); do_something(fp); return 0; } int do_something(FILE *f) { fprintf(f, "This should work if you know what you're doing"); } And it won't work.
      - maximilianburke 15y ago
        In C++, proper scoping with RAII (similar to CL's with-blah idioms) would alleviate that issue as do_something in the code below won't compile. struct WithOpenFile { FILE *fp; WithOpenFile(const char *fileName) : fp(fopen(fileName, "w")) {} ~WithOpenFile() { fclose(fp); } }; int do_something(FILE *f) { fprintf(f, "This should work if you know what you're doing"); } int main(int argc, char *argv[]) { { WithOpenFile file("outfile"); } do_something(file.fp); return 0; } But Lisps will happily accept this: (defun do-something (foo) (1+ foo)) (with-open-file (file "outfile") (do-something file))
        rbanffy 15y ago
        Fine. But now you'll have to accept both files and network sockets. What do you do?

pavelludiq 15y ago

The biggest programs I've worked on are way smaller than 1MLOC, but I've dug around a few large code-bases written in dynamic languages(emacs most notably, 1M lines of Lisp, 1/4M lines of C), and my unproven theory of large systems is that it might be a good idea to make mistakes have smaller impact, rather than try to avoid them(they are inevitable), by focusing on better abstractions, and thus keeping problems more local, focusing on the protocols rather than on the implementation. Static typing might help here, but designing a protocol/interface/api/dsl is hard and I need to experiment a lot with it, so I'm willing to sacrifice some safety for better overall design and flexibility(which will hopefully make debugging easier). IOW I agree you need DSLs, polymorphism and all that good jazz. Where my disagreement comes in is that It's a trade off, and not necessarily an absolute need to have a lot of static typing. In my personal work I'm willing to make the trade off and work with clay rather than marble. Other people work and think differently and I accept that; absolute truths are rare in computer science, and don't exist in software engineering, its all trade-offs here :) ps My hobby vaporware CL project will probably be called mudball, I'm trying to learn how to design systems with hackability and extreme flexibility in mind :)