14 ms·

Languages have always come and gone; Python has always mattered, it's simply finally showing itself as a much more suited tool for certain tasks (such as CS edu

Languages have always come and gone; Python has always mattered, it's simply finally showing itself as a much more suited tool for certain tasks (such as CS education, lightweight scripting by data scientists, glueing together large frameworks, ...). While popular, it can't really hit every single spot required in software development; try to maintain > 10k LOC codebases in Python to understand what I mean. The point that I'm trying to say is that, while we're living in a moment where python is at the center of everything, it won't last forever; it never does. Something better always shows up, given enough time.

pmiller2 8y ago
I’ve worked on Python codebases >100k LOC with no issues. What actually is your point?
- toolslive 8y ago
  having been on the receiving end of a large python project, I can say that refactoring is (trying to stay polite here) a challenge.
  - pmiller2 8y ago
    I don’t see how that has anything to do with Python per se. It’s more of an issue with large codebases.
    - toolslive 8y ago
      Dynamically typed languages make it harder to refactor a large codebase than statically typed languages do.
      - jernfrost 8y ago
        In some ways and some ways not. Making a global change is much harder in a dynamic language. However static languages by their very nature tie the code together more tightly. It is much easier to decouple something in a dynamic language. The re-factoring needs in a dynamic language is thus entirely different.
        TimJYoung 8y ago
        Yes, but these are two sides of the same coin: being able to inject code dynamically in a way that cannot be statically checked at compilation time results in code that will cause runtime errors if there are issues (even basic issues). Also, I would argue that your use of the words "tie the code together" implies something that isn't actually occurring. With a static type system, verifying that code is using the correct types for classes, methods, functions, etc. is not binding them together in any way. If you drop down to assembler, you can do anything you want with any of it, just as you would with a dynamic language. A static type system is simply a compile-time integrity check on the validity of your code. Also, the de-coupling and modularization of functionality has been a thing long before dynamic languages became so popular. You simply need an FFI, and then the compiler makes sure that everything is straight before you get anywhere near deployment.
    - adrianN 8y ago
      Type systems generally are thought to help with large scale refactoring. Unless you have perfect path coverage with your unit tests, there can always be paths lurking after refactoring your duck objects to be geese.
  - est 8y ago
    refactoring a equivalent Java project is way more challenging. I'd argue. Refactoring a python project is usually one man's job. Refactoring a Java project takes a whole department and process management crew.
    - qalmakka 8y ago
      Java is absolute garbage, and it is definitely not the kind of language I've in mind, which is something like Go, Rust, C#, Kotlin, Swift, ocaml... Even C++ is perhaps easier to understand than Java, if not because it tends to have half boilerplate than a similarly sized Java project.
    - potatoyogurt 8y ago
      Why do you think Java is so much harder to refactor? Java and python are not that much different, and Java has static types which aid refactoring as well as a really strong tooling ecosystem. I haven't worked with large python codebases, but it hasn't really been my experience that refactoring Java is unusually hard, except insofar as average code quality among projects written in Java is probably lower.
      - toolslive 8y ago
        Java IDEs have good support for refactoring since the late 90s: - rename/move class (and autoupdate all references) - change method signature most of it is just stating what you want and the tool does it for you. I don't see that level of support from python IDEs. https://news.ycombinator.com/item?id=9671783 https://news.ycombinator.com/item?id=9671783
    - senorjazz 8y ago
      > refactoring a equivalent Java project is way more challenging. I'd argue. Sorry but you are wrong. Just plain wrong. There are benefits for both languages. But to say python is easier to refactor is like saying Java is more concise. It isn't. >Refactoring a Java project takes a whole department and process management crew. Has nothing to do with the language, but the company. A small company will refactor and have an update out same day in Java. An enterprise using some python will be as you say
  - tom_ 8y ago
    It's impossible. Once your program gets past ~1,000 lines, you're stuck with it, basically :( I've always found C++ much easier to work with, if not to actually write.
  - fabioz 8y ago
    Personally, I think that refactoring Python is straightforward (and many refactorings you need to do are much simpler and easier to do without breaking existing APIs -- doing refactorings in Java usually needs changing much more code even for simple things). I think that the real problem is not having a good test coverage, and I agree that bad programs written in Python may be harder to work with than bad programs written in a statically typed language (but in a good codebase, even if large, I'd rather be working with Python). -- although note that in the latest incarnation of Python you can try to use Mypy (http://mypy-lang.org/ http://mypy-lang.org/) along with https://github.com/Instagram/MonkeyType https://github.com/Instagram/MonkeyType or https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate to add types to you and go from there if you really miss types too much and this could make a bad codebase bearable (I'm personally waiting for PEP 544 before adding any types though as I'm not a fan of the type system implemented in PEP 484 very much).
    - scarface74 8y ago
      Personally, I think that refactoring Python is straightforward (and many refactorings you need to do are much simpler and easier to do without breaking existing APIs -- doing refactorings in Java usually needs changing much more code even for simple things). Refactoring in a non statically typed language cant be as straightforward - many refactors in statically typed language can be automated and guaranteed safe.
- qalmakka 8y ago
  Like someone else said, I personally find that very large codebases written in dynamically typed languages tend to discourage refactoring. You may try to mitigate the issue by writing the whole application as standalone, independent modules, but we all know that a bit of interdependence always creeps inside even the best designed application. I love Python; that's the first language I've learnt and I can't really live without its REPL. I only think that the industry has a tendency to excessively obsess towards certain languages, even when they are not the best tool for the job. For instance, I strongly believe that the academia's morbid obsession with Java has crippled a whole generation of new programmers, which are now unable to think and code in no other way but the Java™ Way. I won't say that python is as bad as Java in this regard (it isn't even close), but it annoys me to death when it looks like we're heading once again towards a language monoculture; at least this time the language isn't utter crap.
  - devxpy 8y ago
    > looks like we're heading once again towards a language monoculture Isn't that's only true for data science? What about the JavaScript monoculture :( And academia still has courses for C java C++ etc. No respectable University teaches a single language!
    - qalmakka 8y ago
      In many Universities, C introductory courses are taught by people that hasn't touched C in at least two decades, or has close to zero experience to how programming modern C is. More often than not, this C is taught by people that have only used Java in years; I've seen horrible stuff, such as C89 still being thought like it's the only standard available, or gets() being used by professors that have no idea whatsoever of what's wrong with them.
      - devxpy 8y ago
        Here in India, 90% of teachers haven't touched what they're teaching, like ever. But you still _have_ to study it because you need to give the exam... Not like it solves anything -- just that there's this side of the argument too.. > I've seen horrible stuff, such as C89 still being thought like it's the only standard available, or gets()... Youre absolutely right -- although one could argue that the curious will have the decency to look this stuff up (like myself) The teaching system is still pretty rekt on the python side of things. Its not like you'll find people who actually code teaching at Universities anyway :( Like now we're just going into how bad the teaching system is. The original argument that python is reaching is a monoculture seems weird to me, when I can't even run the damn thing on a browser..
mherrmann 8y ago
I'm very happily maintaining a 15 kLoC code base (my file manager [1]). No issues at all. Though it must be said I'm alone, so know the code base inside out. My point: Likely, there is a project size where Python's dynamic nature becomes a challenge. But 10 kLoC isn't it. [1]: https://fman.io https://fman.io
- qalmakka 8y ago
  Writing code ≠ maintaining code someone else has written. Duck typed code relies in general in implicit behavioural assumption on how the values you're receiving should behave, and it's much better to be the one making choices than the once trying to understand them. I'm not arguing that it's not possible to do it; I just think that it requires a bit of extra steps and care to document which assumptions your code takes and how it behaves that could be more meaningfully described by a statically typed language. But these are just my 2 cents.
  - austinpray 8y ago
    What do you think of mypy? http://mypy-lang.org/ http://mypy-lang.org/
    - qalmakka 8y ago
      It seems definitely promising; it depends on how much adoption it's going to take. Even the most interesting languages (like Nim, for instance) struggle these days without corporate backing.
      - drcongo 8y ago
        Worth pointing out maybe that MyPy is not a language but a type checker that works with regular Python. I've been getting into the habit of using it on everything I do lately and find it extremely useful. You can even do runtime checking on the types with tools like Enforce [1], and the Molten framework that was posted on here last week [2] uses MyPy typing as a core part of the framework. [1] https://github.com/RussBaz/enforce https://github.com/RussBaz/enforce [2] https://news.ycombinator.com/item?id=18107818 https://news.ycombinator.com/item?id=18107818
pistolpeteDK 8y ago
Not trying to split hairs, but we have no problem maintaining our 50K LOC python/django codebase. What I like about Python is the ease of which you can layout a project to fit your needs. Maintaining a large codebase is more about layout/architecture than the language itself. Maybe other languages are simply better suited for programmers who lack the need of keeping their codebase clean and tidy :)
- sumofzeros 8y ago
  > 50K LOC python/django codebase. How much of it is generated?
  - andybak 8y ago
    I would guess none. Code generation is very rare in the Python community. Mainly because it's generally unnecessary given the nature of the language.
    - paulie_a 8y ago
      I frequently use code generation in python. why write thousands of lines of boilerplate code in any language when you can just generate it? Maybe I'm just lazy but the results are better and more consistent to get something done.
      - AstralStorm 8y ago
        Weird, typically you can use metaclasses or mixins to remove such boilerplate.
        paulie_a 8y ago
        I utilize that too but why not generate new models with forms, cbvs, drf endpoints, serializers, etc? You can pick and choose what you need, add the mixins and move on. Plus initial project using cookicutter to generate the original project in a much more fitting and featureful structure.
devilmoon 8y ago
I'd prefer maintaining a 10k LoC python project rather than a 10k LoC Java project (which would likely be 8k lines of boilerplate anyway)
- senorjazz 8y ago
  horses for courses. I would prefer a 100k java LOC program over a 10k loc python program, any day. The ease and safety you can make huge refactors is a huge win. Sure you can do it in python, but you will be writing x10 the amount of tests and final LOC counts including tests will probably have python come in closer to Java than the 10k > 100k loc indicates
Flip-per 8y ago
I'm contributing to a >20k LOC open-source python/django project (counting .py files only). It's quite clean with an architecture that makes sense, and thus way easier to understand and refactor than an actually smaller Java project at work. Admittedly, I have more of a Python background than a Java background.
- Jian-Yang 8y ago
  Java is just as bad as python. For maintainability, header files make life a lot easier.
  - TuringTest 8y ago
    Or you could use documentation.
gaius 8y ago
try to maintain > 10k LOC codebases in Python to understand what I mean. My previous job we had about 3 million lines of Python and it was fine.
devxpy 8y ago
I think your real problem is with duck typing and not specifically with python. There are efforts to have gradual typing suppprt in python though, and you might be interested in having at least partial typing suppprt.
- qalmakka 8y ago
  Yup, it's a complaint that can be applied to every duck typed language out there, not just Python.
  - devxpy 8y ago
    I actually believe that we can solve this problem for most cases. Take dart for instance -- it doesn't require any typing as such, but has an excellent type system that supports generics too. The type checker is pretty robust, but flexible at the same time...
devxpy 8y ago
> it won't last forever; it never does I would love for a real contender to arrive, that challenges python's level of abstraction in a real way...
- alehander42 8y ago
  wow, what's python's level of abstraction? there are languages with much better abstraction capabilities
  - devxpy 8y ago
    Please enlighten me! Would love to try out something new.
    - alehander42 8y ago
      well I wasn't sure what you mean by abstraction. typically languages like Haskell, OCaml are based on great abstractions (and are good to express new ones, especially from a type-system point of view). If you're looking for languages which are similarly "easy to use", I can suggest to try Nim or Crystal. Nim looks a lot like Python(indentation-based and similar constructs), it has a nice type system and very strong metaprogramming which helps to often express stuff in <= lines you would in Python (and also C and JS backends with excellent interop. Crystal is inspired by Ruby, but it's also statically typed and mostly used for web development from my impression. Its windows support is still under development but it's also very expressive.
- firdak 8y ago
  Do you think a statically-typed language could ever be a "contender" - i.e. compete with Python in terms of ease-of-use etc? Nowadays, I don't think any new, dynamically-typed language has a chance to become mainstream.
  - deleted 8y ago
    [deleted]
stevesimmons 8y ago
> try to maintain > 10k LOC codebases in Python to understand what I mean Lots of companies successfully manage far larger Python codebases. We have a 35M LOC Python codebase with 2500 active developers making 25000 commits per week and continuous deployment globally of new code. (I gave a talk on it at PyData London in April: "Python at Massive Scale")
- mcintyre1994 8y ago
  If anyone else is looking: https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.
  - stevesimmons 8y ago
    The most concise answer is 'tooling'. We have a DevTools team that builds the SDLC and automated testing infra, all tightly integrated into MS Visual Studio. That lets the thousands of developers focus on productive coding.
    - mino 8y ago
      Just watched it. Thanks, it was indeed a great talk, and indeed a lot of tooling. Do you have any extra writeup to suggest?
      - stevesimmons 8y ago
        There isn't a ton of detailed public information on this kind of stuff. My talk focused on the general principles of our approach, rather than the myriad of features and optimizations necessary at such a large scale. A good next step would be reading the 2016 ACM paper "Why Google Stores Billions of Lines of Code in a Single Repository" [1] then reading a recent HN discussion [2]. That has a ton of good links. [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371
gigatexal 8y ago
Dropbox has 1m lines of python — it’s doable.
- qalmakka 8y ago
  Dropbox has had several headaches from its reliance on python, first among all from the fact that Python is still bound on running once thread at once on a single core after all these years (thanks GIL), so it does not parallelize at all without abusing subprocess + IPC. That's the reason why they attempted writing a new runtime first (Pyston), and then settled down on rewriting their performance critical infrastructure in Go (https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-libraries https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-...) Google iself wrote grumpy for the same reason, which is that CPython scales pretty bad.
  - gigatexal 8y ago
    No I get that. And they employ the creator of the language just that it was doable but not necessarily enjoyable.
- senorjazz 8y ago
  doable sure. But enjoyable? ;-)
Rotareti 8y ago
> try to maintain > 10k LOC codebases in Python to understand what I mean. Since Python 3.5 you can use type annotations: def my_func(foo: str, bar: User) -> None: baz: int = User.qux This is a useful feature for larger code bases. In the long run, I'm sure some org will come up with a decent Python compiler, that: * compiles Python to C * makes use of (optional) type annotations * improves performance * creates single file binaries I guess this would be less of an effort than porting the whole Python ecosystem to e.g. Julia.
- demonshreder 8y ago
  Nuitka (http://nuitka.net/pages/overview.html http://nuitka.net/pages/overview.html) seems to be attempting to do those.
  - Rotareti 8y ago
    Nuitka is awesome. I used it to compile a Python command line tool recently and it worked fine. In fact, it worked so well that I added a Nuitka build script to that project... But Nuitka is written and maintained by a single person [0]. I think for such a huge task (with so many edge cases) it needs more attention/support in order to become mainstream. [0]: https://github.com/Nuitka/Nuitka/graphs/contributors https://github.com/Nuitka/Nuitka/graphs/contributors
TurboHaskal 8y ago
Everyone keeps saying that, but my experience is rather the opposite. Static typing is great when the program is small and you can keep the model in your head, but the larger the program, the more you benefit from dynamic typing. Sure, static typing helps with tiny refactorings, but making non trivial changes that don't play well with the existing model proved to be too much effort. I can't speak for Python but since the codebases were Perl and Common Lisp based, I believe it would be similar, should there be anything similar to the MOP for Python.
- kccqzy 8y ago
  In my experience, it's the opposite. Static typing is great but not essential when the program is small and you can keep the model in your head, but the larger the program, the more you benefit from static typing. Static typing helps with tiny refactorings, but it truly shines when making non-trivial changes that don't play well with the existing model; it's as effortless as making the desired changes to the model, then mindlessly fixing all compile errors, and it will likely just work afterwards.
  - sacado2 8y ago
    Plus, static typing acts as a formal documentation, which is more than welcome when you get to maintain someone else's code.
    - therockhead 8y ago
      For me this is by far the best feature of static typing.
amrx431 8y ago
Java. C#, C++ will never be replaced. Want job security learn and code in these languages. Enterprises love them and are not going to incur cost to replace their code-bases with anything else.
- rusk 8y ago
  Java code is a bitch to write from scratch but it never dies.
  - simonh 8y ago
    Oracle's commercial licensing department is relying on that fact.
- TuringTest 8y ago
  People said the same thing about COBOL. It was replaced anyway. You're using a really short version of "never".
  - asien 8y ago
    Banking and Government are still making services in COBOL . It’ll never disappear.
    - 3rdAccount 8y ago
      There are a lot of Cobol replacement projects underway. Yes, it'll still be around for awhile, but there's less of it in the wild every year. The sad thing is that many companies prefer the mainframe Cobol service as it has run reliably for decades and the Java replacement does not have that trait.
  - oblio 8y ago
    Cobol had 10x fewer developers at its peakc and probably 100-1000x fewer lines out of code out there, in the wild. Plus it didn't have a huge Open Source community.
oliwarner 8y ago
I have personal projects that are bigger than 10KLOC and professional projects that are 10× that. I'm not sure what you're getting at but code management in Python is really no harder than anything else. It can be clean and well delineated, or you can completely screw it up... But that's not Python's fault.
simonh 8y ago
>it won't last forever; it never does. The rest of my life will do fine. I'm sure you're right, but just for fun I'll make some counter-arguments. All technologies follow an S-curve in their 'bang for buck' life cycle. Initial improvement and return on investment ramps up slowly, then it hits a middle era where all the low hanging fruit get picked for massive returns, then it enters a third stage where most of the easy benefits have been reaped and further improvement is incremental. We see this with the internal combustion engine, jet engines, nuclear power, all sorts of thing. Eventually another technology might come along that supplants the old technology for many or most use cases. The question I'm pondering is, are languages technologies in this respect, or are programming languages in general a technology? If the former then Python will top out in the benefits it provides and will almost certainly be supplanted by a successor. If the latter, maybe Python is near the top of the development curve (along with many other modern languages) and further improvements will be too incremental to lead to anything supplanting it? But if that is the case, what technology might supplant programming languages?
- mch82 8y ago
  Interesting observation! Throwing a couple ideas out there... One possibility is visual languages that emphasize mistake proofing, interchangeable parts, and logic visualization over expressiveness. I’ve noticed an increasing number and use of visual languages (Blockly, Unreal Blueprints, NoFlo, WebGME). These are typically criticized as too limited for real programming, but seem to be climbing the S-curve. A more likely near-term possibility is continued adoption of systems that emphasize package management, which values reuse of proven code over ability to solve problems from scratch. It seems like the language package managers are a big draw in Python & NodeJS. Maybe this trend moves even farther toward stuff like IFTT, where most people working with code are really chaining together applets. We’re also seemingly headed toward the Star Trek TNG computer where humans query a computer with voice commands and work with it to solve problems.
jernfrost 8y ago
While I may personally prefer statically typed language for large code bases, I think people greatly exaggerate the problems of maintaining large code bases in dynamic languages. Python projects such has NumPy, SciPy and Pandas are several hundred thousand lines of Python. Pandas is 250k LOC in Python. GNU Emacs is 1.5 million lines of LISP code, another dynamic language. It is ultimately a question of sound software architecture. You don't have to care about how many lines of code is in the libraries you use daily or in the Operating System you use. In similar fashion any software can be modularized so you only need to concern yourself with your part.
tobmlt 8y ago
Chiming in anecdotally here: 50k ``real'' lines of Python and climbing. No problems maintaining it but I wrote it all for my dissertation. It's scientific/CAGD programming, -Not sure but it feels like that makes it easier. Pretty sure I need to re-write it to entice others in on the fun. Anecdotes aren't valid far from the experience, I am afraid. I, along with one other guy, maintain a >500k fortran 77 code base (bit of C/C++ sprinkled in) at work. It's way cleaner than my home-brew python. Maintainable is about wise practices, not language choice.

pmiller2 8y ago

I’ve worked on Python codebases >100k LOC with no issues. What actually is your point?

toolslive 8y ago
having been on the receiving end of a large python project, I can say that refactoring is (trying to stay polite here) a challenge.
- pmiller2 8y ago
  I don’t see how that has anything to do with Python per se. It’s more of an issue with large codebases.
  - toolslive 8y ago
    Dynamically typed languages make it harder to refactor a large codebase than statically typed languages do.
    - jernfrost 8y ago
      In some ways and some ways not. Making a global change is much harder in a dynamic language. However static languages by their very nature tie the code together more tightly. It is much easier to decouple something in a dynamic language. The re-factoring needs in a dynamic language is thus entirely different.
      - TimJYoung 8y ago
        Yes, but these are two sides of the same coin: being able to inject code dynamically in a way that cannot be statically checked at compilation time results in code that will cause runtime errors if there are issues (even basic issues). Also, I would argue that your use of the words "tie the code together" implies something that isn't actually occurring. With a static type system, verifying that code is using the correct types for classes, methods, functions, etc. is not binding them together in any way. If you drop down to assembler, you can do anything you want with any of it, just as you would with a dynamic language. A static type system is simply a compile-time integrity check on the validity of your code. Also, the de-coupling and modularization of functionality has been a thing long before dynamic languages became so popular. You simply need an FFI, and then the compiler makes sure that everything is straight before you get anywhere near deployment.
  - adrianN 8y ago
    Type systems generally are thought to help with large scale refactoring. Unless you have perfect path coverage with your unit tests, there can always be paths lurking after refactoring your duck objects to be geese.
- est 8y ago
  refactoring a equivalent Java project is way more challenging. I'd argue. Refactoring a python project is usually one man's job. Refactoring a Java project takes a whole department and process management crew.
  - qalmakka 8y ago
    Java is absolute garbage, and it is definitely not the kind of language I've in mind, which is something like Go, Rust, C#, Kotlin, Swift, ocaml... Even C++ is perhaps easier to understand than Java, if not because it tends to have half boilerplate than a similarly sized Java project.
  - potatoyogurt 8y ago
    Why do you think Java is so much harder to refactor? Java and python are not that much different, and Java has static types which aid refactoring as well as a really strong tooling ecosystem. I haven't worked with large python codebases, but it hasn't really been my experience that refactoring Java is unusually hard, except insofar as average code quality among projects written in Java is probably lower.
    - toolslive 8y ago
      Java IDEs have good support for refactoring since the late 90s: - rename/move class (and autoupdate all references) - change method signature most of it is just stating what you want and the tool does it for you. I don't see that level of support from python IDEs. https://news.ycombinator.com/item?id=9671783 https://news.ycombinator.com/item?id=9671783
  - senorjazz 8y ago
    > refactoring a equivalent Java project is way more challenging. I'd argue. Sorry but you are wrong. Just plain wrong. There are benefits for both languages. But to say python is easier to refactor is like saying Java is more concise. It isn't. >Refactoring a Java project takes a whole department and process management crew. Has nothing to do with the language, but the company. A small company will refactor and have an update out same day in Java. An enterprise using some python will be as you say
- tom_ 8y ago
  It's impossible. Once your program gets past ~1,000 lines, you're stuck with it, basically :( I've always found C++ much easier to work with, if not to actually write.
- fabioz 8y ago
  Personally, I think that refactoring Python is straightforward (and many refactorings you need to do are much simpler and easier to do without breaking existing APIs -- doing refactorings in Java usually needs changing much more code even for simple things). I think that the real problem is not having a good test coverage, and I agree that bad programs written in Python may be harder to work with than bad programs written in a statically typed language (but in a good codebase, even if large, I'd rather be working with Python). -- although note that in the latest incarnation of Python you can try to use Mypy (http://mypy-lang.org/ http://mypy-lang.org/) along with https://github.com/Instagram/MonkeyType https://github.com/Instagram/MonkeyType or https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate to add types to you and go from there if you really miss types too much and this could make a bad codebase bearable (I'm personally waiting for PEP 544 before adding any types though as I'm not a fan of the type system implemented in PEP 484 very much).
  - scarface74 8y ago
    Personally, I think that refactoring Python is straightforward (and many refactorings you need to do are much simpler and easier to do without breaking existing APIs -- doing refactorings in Java usually needs changing much more code even for simple things). Refactoring in a non statically typed language cant be as straightforward - many refactors in statically typed language can be automated and guaranteed safe.
qalmakka 8y ago
Like someone else said, I personally find that very large codebases written in dynamically typed languages tend to discourage refactoring. You may try to mitigate the issue by writing the whole application as standalone, independent modules, but we all know that a bit of interdependence always creeps inside even the best designed application. I love Python; that's the first language I've learnt and I can't really live without its REPL. I only think that the industry has a tendency to excessively obsess towards certain languages, even when they are not the best tool for the job. For instance, I strongly believe that the academia's morbid obsession with Java has crippled a whole generation of new programmers, which are now unable to think and code in no other way but the Java™ Way. I won't say that python is as bad as Java in this regard (it isn't even close), but it annoys me to death when it looks like we're heading once again towards a language monoculture; at least this time the language isn't utter crap.
- devxpy 8y ago
  > looks like we're heading once again towards a language monoculture Isn't that's only true for data science? What about the JavaScript monoculture :( And academia still has courses for C java C++ etc. No respectable University teaches a single language!
  - qalmakka 8y ago
    In many Universities, C introductory courses are taught by people that hasn't touched C in at least two decades, or has close to zero experience to how programming modern C is. More often than not, this C is taught by people that have only used Java in years; I've seen horrible stuff, such as C89 still being thought like it's the only standard available, or gets() being used by professors that have no idea whatsoever of what's wrong with them.
    - devxpy 8y ago
      Here in India, 90% of teachers haven't touched what they're teaching, like ever. But you still _have_ to study it because you need to give the exam... Not like it solves anything -- just that there's this side of the argument too.. > I've seen horrible stuff, such as C89 still being thought like it's the only standard available, or gets()... Youre absolutely right -- although one could argue that the curious will have the decency to look this stuff up (like myself) The teaching system is still pretty rekt on the python side of things. Its not like you'll find people who actually code teaching at Universities anyway :( Like now we're just going into how bad the teaching system is. The original argument that python is reaching is a monoculture seems weird to me, when I can't even run the damn thing on a browser..

mherrmann 8y ago

I'm very happily maintaining a 15 kLoC code base (my file manager [1]). No issues at all. Though it must be said I'm alone, so know the code base inside out. My point: Likely, there is a project size where Python's dynamic nature becomes a challenge. But 10 kLoC isn't it. [1]: https://fman.io https://fman.io

qalmakka 8y ago
Writing code ≠ maintaining code someone else has written. Duck typed code relies in general in implicit behavioural assumption on how the values you're receiving should behave, and it's much better to be the one making choices than the once trying to understand them. I'm not arguing that it's not possible to do it; I just think that it requires a bit of extra steps and care to document which assumptions your code takes and how it behaves that could be more meaningfully described by a statically typed language. But these are just my 2 cents.
- austinpray 8y ago
  What do you think of mypy? http://mypy-lang.org/ http://mypy-lang.org/
  - qalmakka 8y ago
    It seems definitely promising; it depends on how much adoption it's going to take. Even the most interesting languages (like Nim, for instance) struggle these days without corporate backing.
    - drcongo 8y ago
      Worth pointing out maybe that MyPy is not a language but a type checker that works with regular Python. I've been getting into the habit of using it on everything I do lately and find it extremely useful. You can even do runtime checking on the types with tools like Enforce [1], and the Molten framework that was posted on here last week [2] uses MyPy typing as a core part of the framework. [1] https://github.com/RussBaz/enforce https://github.com/RussBaz/enforce [2] https://news.ycombinator.com/item?id=18107818 https://news.ycombinator.com/item?id=18107818

pistolpeteDK 8y ago

Not trying to split hairs, but we have no problem maintaining our 50K LOC python/django codebase. What I like about Python is the ease of which you can layout a project to fit your needs. Maintaining a large codebase is more about layout/architecture than the language itself. Maybe other languages are simply better suited for programmers who lack the need of keeping their codebase clean and tidy :)

sumofzeros 8y ago
> 50K LOC python/django codebase. How much of it is generated?
- andybak 8y ago
  I would guess none. Code generation is very rare in the Python community. Mainly because it's generally unnecessary given the nature of the language.
  - paulie_a 8y ago
    I frequently use code generation in python. why write thousands of lines of boilerplate code in any language when you can just generate it? Maybe I'm just lazy but the results are better and more consistent to get something done.
    - AstralStorm 8y ago
      Weird, typically you can use metaclasses or mixins to remove such boilerplate.
      - paulie_a 8y ago
        I utilize that too but why not generate new models with forms, cbvs, drf endpoints, serializers, etc? You can pick and choose what you need, add the mixins and move on. Plus initial project using cookicutter to generate the original project in a much more fitting and featureful structure.

devilmoon 8y ago

I'd prefer maintaining a 10k LoC python project rather than a 10k LoC Java project (which would likely be 8k lines of boilerplate anyway)

senorjazz 8y ago
horses for courses. I would prefer a 100k java LOC program over a 10k loc python program, any day. The ease and safety you can make huge refactors is a huge win. Sure you can do it in python, but you will be writing x10 the amount of tests and final LOC counts including tests will probably have python come in closer to Java than the 10k > 100k loc indicates

Flip-per 8y ago

I'm contributing to a >20k LOC open-source python/django project (counting .py files only). It's quite clean with an architecture that makes sense, and thus way easier to understand and refactor than an actually smaller Java project at work. Admittedly, I have more of a Python background than a Java background.

Jian-Yang 8y ago
Java is just as bad as python. For maintainability, header files make life a lot easier.
- TuringTest 8y ago
  Or you could use documentation.

gaius 8y ago

try to maintain > 10k LOC codebases in Python to understand what I mean. My previous job we had about 3 million lines of Python and it was fine.

devxpy 8y ago

I think your real problem is with duck typing and not specifically with python. There are efforts to have gradual typing suppprt in python though, and you might be interested in having at least partial typing suppprt.

qalmakka 8y ago
Yup, it's a complaint that can be applied to every duck typed language out there, not just Python.
- devxpy 8y ago
  I actually believe that we can solve this problem for most cases. Take dart for instance -- it doesn't require any typing as such, but has an excellent type system that supports generics too. The type checker is pretty robust, but flexible at the same time...

devxpy 8y ago

> it won't last forever; it never does I would love for a real contender to arrive, that challenges python's level of abstraction in a real way...

alehander42 8y ago
wow, what's python's level of abstraction? there are languages with much better abstraction capabilities
- devxpy 8y ago
  Please enlighten me! Would love to try out something new.
  - alehander42 8y ago
    well I wasn't sure what you mean by abstraction. typically languages like Haskell, OCaml are based on great abstractions (and are good to express new ones, especially from a type-system point of view). If you're looking for languages which are similarly "easy to use", I can suggest to try Nim or Crystal. Nim looks a lot like Python(indentation-based and similar constructs), it has a nice type system and very strong metaprogramming which helps to often express stuff in <= lines you would in Python (and also C and JS backends with excellent interop. Crystal is inspired by Ruby, but it's also statically typed and mostly used for web development from my impression. Its windows support is still under development but it's also very expressive.
firdak 8y ago
Do you think a statically-typed language could ever be a "contender" - i.e. compete with Python in terms of ease-of-use etc? Nowadays, I don't think any new, dynamically-typed language has a chance to become mainstream.
- deleted 8y ago
  [deleted]

stevesimmons 8y ago

> try to maintain > 10k LOC codebases in Python to understand what I mean Lots of companies successfully manage far larger Python codebases. We have a 35M LOC Python codebase with 2500 active developers making 25000 commits per week and continuous deployment globally of new code. (I gave a talk on it at PyData London in April: "Python at Massive Scale")

mcintyre1994 8y ago
If anyone else is looking: https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.
- stevesimmons 8y ago
  The most concise answer is 'tooling'. We have a DevTools team that builds the SDLC and automated testing infra, all tightly integrated into MS Visual Studio. That lets the thousands of developers focus on productive coding.
  - mino 8y ago
    Just watched it. Thanks, it was indeed a great talk, and indeed a lot of tooling. Do you have any extra writeup to suggest?
    - stevesimmons 8y ago
      There isn't a ton of detailed public information on this kind of stuff. My talk focused on the general principles of our approach, rather than the myriad of features and optimizations necessary at such a large scale. A good next step would be reading the 2016 ACM paper "Why Google Stores Billions of Lines of Code in a Single Repository" [1] then reading a recent HN discussion [2]. That has a ton of good links. [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371

gigatexal 8y ago

Dropbox has 1m lines of python — it’s doable.

qalmakka 8y ago
Dropbox has had several headaches from its reliance on python, first among all from the fact that Python is still bound on running once thread at once on a single core after all these years (thanks GIL), so it does not parallelize at all without abusing subprocess + IPC. That's the reason why they attempted writing a new runtime first (Pyston), and then settled down on rewriting their performance critical infrastructure in Go (https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-libraries https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-...) Google iself wrote grumpy for the same reason, which is that CPython scales pretty bad.
- gigatexal 8y ago
  No I get that. And they employ the creator of the language just that it was doable but not necessarily enjoyable.
senorjazz 8y ago
doable sure. But enjoyable? ;-)

Rotareti 8y ago

> try to maintain > 10k LOC codebases in Python to understand what I mean. Since Python 3.5 you can use type annotations: def my_func(foo: str, bar: User) -> None: baz: int = User.qux This is a useful feature for larger code bases. In the long run, I'm sure some org will come up with a decent Python compiler, that: * compiles Python to C * makes use of (optional) type annotations * improves performance * creates single file binaries I guess this would be less of an effort than porting the whole Python ecosystem to e.g. Julia.

demonshreder 8y ago
Nuitka (http://nuitka.net/pages/overview.html http://nuitka.net/pages/overview.html) seems to be attempting to do those.
- Rotareti 8y ago
  Nuitka is awesome. I used it to compile a Python command line tool recently and it worked fine. In fact, it worked so well that I added a Nuitka build script to that project... But Nuitka is written and maintained by a single person [0]. I think for such a huge task (with so many edge cases) it needs more attention/support in order to become mainstream. [0]: https://github.com/Nuitka/Nuitka/graphs/contributors https://github.com/Nuitka/Nuitka/graphs/contributors

TurboHaskal 8y ago

Everyone keeps saying that, but my experience is rather the opposite. Static typing is great when the program is small and you can keep the model in your head, but the larger the program, the more you benefit from dynamic typing. Sure, static typing helps with tiny refactorings, but making non trivial changes that don't play well with the existing model proved to be too much effort. I can't speak for Python but since the codebases were Perl and Common Lisp based, I believe it would be similar, should there be anything similar to the MOP for Python.

kccqzy 8y ago
In my experience, it's the opposite. Static typing is great but not essential when the program is small and you can keep the model in your head, but the larger the program, the more you benefit from static typing. Static typing helps with tiny refactorings, but it truly shines when making non-trivial changes that don't play well with the existing model; it's as effortless as making the desired changes to the model, then mindlessly fixing all compile errors, and it will likely just work afterwards.
- sacado2 8y ago
  Plus, static typing acts as a formal documentation, which is more than welcome when you get to maintain someone else's code.
  - therockhead 8y ago
    For me this is by far the best feature of static typing.

amrx431 8y ago

Java. C#, C++ will never be replaced. Want job security learn and code in these languages. Enterprises love them and are not going to incur cost to replace their code-bases with anything else.

rusk 8y ago
Java code is a bitch to write from scratch but it never dies.
- simonh 8y ago
  Oracle's commercial licensing department is relying on that fact.
TuringTest 8y ago
People said the same thing about COBOL. It was replaced anyway. You're using a really short version of "never".
- asien 8y ago
  Banking and Government are still making services in COBOL . It’ll never disappear.
  - 3rdAccount 8y ago
    There are a lot of Cobol replacement projects underway. Yes, it'll still be around for awhile, but there's less of it in the wild every year. The sad thing is that many companies prefer the mainframe Cobol service as it has run reliably for decades and the Java replacement does not have that trait.
- oblio 8y ago
  Cobol had 10x fewer developers at its peakc and probably 100-1000x fewer lines out of code out there, in the wild. Plus it didn't have a huge Open Source community.

oliwarner 8y ago

I have personal projects that are bigger than 10KLOC and professional projects that are 10× that. I'm not sure what you're getting at but code management in Python is really no harder than anything else. It can be clean and well delineated, or you can completely screw it up... But that's not Python's fault.

simonh 8y ago

>it won't last forever; it never does. The rest of my life will do fine. I'm sure you're right, but just for fun I'll make some counter-arguments. All technologies follow an S-curve in their 'bang for buck' life cycle. Initial improvement and return on investment ramps up slowly, then it hits a middle era where all the low hanging fruit get picked for massive returns, then it enters a third stage where most of the easy benefits have been reaped and further improvement is incremental. We see this with the internal combustion engine, jet engines, nuclear power, all sorts of thing. Eventually another technology might come along that supplants the old technology for many or most use cases. The question I'm pondering is, are languages technologies in this respect, or are programming languages in general a technology? If the former then Python will top out in the benefits it provides and will almost certainly be supplanted by a successor. If the latter, maybe Python is near the top of the development curve (along with many other modern languages) and further improvements will be too incremental to lead to anything supplanting it? But if that is the case, what technology might supplant programming languages?

mch82 8y ago
Interesting observation! Throwing a couple ideas out there... One possibility is visual languages that emphasize mistake proofing, interchangeable parts, and logic visualization over expressiveness. I’ve noticed an increasing number and use of visual languages (Blockly, Unreal Blueprints, NoFlo, WebGME). These are typically criticized as too limited for real programming, but seem to be climbing the S-curve. A more likely near-term possibility is continued adoption of systems that emphasize package management, which values reuse of proven code over ability to solve problems from scratch. It seems like the language package managers are a big draw in Python & NodeJS. Maybe this trend moves even farther toward stuff like IFTT, where most people working with code are really chaining together applets. We’re also seemingly headed toward the Star Trek TNG computer where humans query a computer with voice commands and work with it to solve problems.

jernfrost 8y ago

While I may personally prefer statically typed language for large code bases, I think people greatly exaggerate the problems of maintaining large code bases in dynamic languages. Python projects such has NumPy, SciPy and Pandas are several hundred thousand lines of Python. Pandas is 250k LOC in Python. GNU Emacs is 1.5 million lines of LISP code, another dynamic language. It is ultimately a question of sound software architecture. You don't have to care about how many lines of code is in the libraries you use daily or in the Operating System you use. In similar fashion any software can be modularized so you only need to concern yourself with your part.

tobmlt 8y ago

Chiming in anecdotally here: 50k ``real'' lines of Python and climbing. No problems maintaining it but I wrote it all for my dissertation. It's scientific/CAGD programming, -Not sure but it feels like that makes it easier. Pretty sure I need to re-write it to entice others in on the fun. Anecdotes aren't valid far from the experience, I am afraid. I, along with one other guy, maintain a >500k fortran 77 code base (bit of C/C++ sprinkled in) at work. It's way cleaner than my home-brew python. Maintainable is about wise practices, not language choice.