7 ms·

Python is great for relatively small projects written by a single person or for glue code, but large Python projects are scary. Diving in and reading someone el

Python is great for relatively small projects written by a single person or for glue code, but large Python projects are scary. Diving in and reading someone else's code can be quite difficult (what are the types of these parameters? they're being used like dicts, but what if they're really a completely different class with different behavior? how can I trace this down when it goes through multiple IPC indirections since they chose to use multiple processes to get around the GIL? oh i'm missing packages and they don't seem to be in pip...). The bigger the project, the more places there are for runtime errors that a statically compiled language would have caught or at least has better tooling to find (syntax errors in an error path block that rarely gets hit, type errors, uncaught exceptions, missing methods, accidentally fat-fingering a variable name and instantiating a new variable). You end up leaning really heavily on tests that wouldn't have to be written in other languages and documentation that would otherwise be statically enforced in many other languages. Even when using Python for glue code, be wary that it will need to be in-charge (i.e. embedding a Python interpreter in your code versus embedding your code in Python especially with CPython). Not saying it's not possible and doesn't work well for a lot of companies, but I do think you take on technical debt to get up and running fast when you choose Python. It's a choice I would personally think twice about.

fideloper 12y ago
Is this different for Python over any dynamic language? This seems to be a better argument for statically compiled languages over dynamic languages rather than an argument against Python.
- robmccoll 12y ago
  That's a good point, and perhaps it is. The majority of my experience with dynamic languages is probably in Python. One thing that makes a difference between Python and other commonly used dynamic languages (here referring to JavaScript / ECMAScript and Lua) is that Python is dramatically more complex out of the box. This is both in terms of language features and Python's batteries-included standard libraries. For example, the amount of overloading that you can do in Python is very impressive, but the result is that common operations aren't necessarily predictable or dependable without more context (not that this isn't a problem in static languages - looking at you C++, but at least in the other languages you get type information to help out and some checking at compile time).
  - thom_nic 12y ago
    Funny, my experience has been that large/ complex python code is actually easier to follow than JavaScript because of the line noise that JS requires. Side by side, the same code will usually be easier to read in Python (or Ruby.) Same goes for Java, although in a slightly different sense. I've seen non-trivial Java projects balloon quickly where you have dozens or hundreds of classes & interfaces with no clear sense of structure. High line noise, lots of boilerplate. Sure it compiles but at the end of the day it's just as likely to hit a NPE so you have to rely on functional and unit tests regardless of static/dynamic/compiled/interpreted. Obviously a lot of it comes down to who's writing the code. For me, a language that lets me succinctly express my intentions with minimal cruft is what wins.
    - herriojr 12y ago
      I totally agree that it comes down to who the code is written by. I've done both python and java, and in python I would go so far as to define interfaces as a form of documentation in my python code so people who use it know what I expect out of their classes which want to interact with my object. It doesn't require the class to explicitly implement the interface but they can if it makes sense (fairly similar to Zope Interfaces I guess). Unfortunately, I don't see this a lot in most python applications and you're left searching around through calls to figure out exactly what is required of the object being passed in. Look at a lot of popular libraries -- you are still left searching through code just to see what can possibly be returned because they aren't always the same type -- also, exceptions aren't part of the function or method definition, which means if you don't document your exceptions, I'm up shit creek. I think a lot of the problem around the python ecosystem, again, that I've seen, is people just don't follow best practices -- documentation, it's a core component if python development -- It's one of the arguments they use against static typing. Personally, I think python developers need to be much more strict about development practices than java developers. I think it's a lot easier to make bad python code than it is to make bad java code. The same goes for C, which I used to also do as well. To be a good C developer, you have to be very strict and structured (however, for somewhat different reasons). With great power comes great responsibility. I've seen shit java code as well. A lot of bad java code usually revolves around things not being modular or not having some form of consistent development patterns or not breaking methods down into simpler sub-problems -- I think documentation isn't as important as it is in python for the fact that I know exactly what is being returned, what exceptions can be thrown, and what exactly needs to be passed in just by looking at a method. In terms of the business logic associated with the class, that still needs to be documented. That said, I understand why python is generally used at startups -- it allows fast initial development where at startups, time is critical. Long term development really relies a lot on the teams ability to make structured decisions and organize their code, which is a difficult task in any language.
mpdehaan2 12y ago
This is an argument for unit tests, not for a statically typed language. Type errors tend to be very few and far between compared to logic errors. Multiple processes in most web applications are best handled by a pre-forking webserver in front of something like mod_wsgi, and using an engine like Celery for asynchronous and long-running operations on the backend, often launched with something like supervisor. I do take issue with some parts of the article, namely saying there is a good type system (there's not, but it's all fine, duck typing, etc) and that twisted is a good framework. It is definitely true however that Python is a great fit for all kinds of serious applications, but with all things, it takes discipline.
- Retra 12y ago
  What is the benefit to using a dynamically typed language if you have to spend all your time writing tests and verifying the static properties of your code?
  - Vaskivo 12y ago
    The idea is that writing code + writing tests in a dynamically typed language is easier and faster just writing code in a statically typed language. (Of course it depends on the languages in question, the programmers and a number of other factors)
  - spdy 12y ago
    Mainly the boilerplate other languages force upon you. Whenever i look at Java Code i start to chuckle. Its so much more information that you need to load into your head. To quote a picture from the post: https://www.paypal-engineering.com/wordpress/wp-content/uploads/2014/12/cpp_py_medium.png https://www.paypal-engineering.com/wordpress/wp-content/uplo... And its not like you need static typing everywhere just in some places its better to enforce it for your own sanity. Type hinting looks like a good way to solve this problem in python.
    - Retra 12y ago
      That image isn't really a fair comparison (in that C++ / Java aren't the only statically typed languages.) I imagine a statically typed Python dialect would only have an extra 15 or so lines, and the benefits would be numerous. I love python as much as anyone, but having to keep a reference manual handy just to use someone's library because they had the audacity to use a variable is not my idea of a fun time. EDIT: I'm probably exaggerating too much. Would I use Python in a large, complicated, multi-developer project? Yes, I would. And my only real complaint would probably be that I like static typing so much that I'd miss it.
mmahemoff 12y ago
Most of these runtime errors rarely make their way to production and are easily identified and rectified in cases when they do slip through. More restricted languages, e.g. Java, do have advantages in terms of forcing developers to do some more self-documenting and better opportunities for tools to help with syntax highlighting, refactoring, and warning about those errors you mention. But those errors are usually caught easily even with basic test procedures and almost always if there's a half-decent automated testing regimen. A lot of it comes down to, who's writing this big project? If developers are talented, there's a lot to be said for the freedom granted by a language like Python or Ruby. If developers are more mediocre, the forcing function of a language like Java may help to produce a "boring" and larger but understandable code base.
- rtpg 12y ago
  >If developers are talented, there's a lot to be said for the freedom granted by a language like Python or Ruby. If developers are more mediocre, the forcing function of a language like Java may help to produce a "boring" and larger but understandable code base. This is a bit of a false dichotomy. Much like I don't think lisp is popular for its functional-ness or whatnot, I don't think python or ruby are popular for their dynamic properties. My guess is that 90% of it is just how the code ends up looking. And with type inference getting further along, I think we'll see more people realizing that typing is a great sanity test to have in your code (especially in big codebases). See the popularity of Go in pythonic circles. I have never been shown any non-tiny piece of robust code that was easier to deal with because of a language's dynamic nature. I have been on this Earth a bit less time than others, but my gut feeling is that it doesn't actually exist. (note: my day job involves a lot of python, and I love the language for many things. I am looking forward to optional type annotations, though, as are many of my coworkers)
golergka 12y ago
http://news.efinancialcareers.com/us-en/173476/investment-banking-tech-guru-quits-starts-firm/ http://news.efinancialcareers.com/us-en/173476/investment-ba... "...There are around 5,000 developers using [Python] at Bank of America, .. there are close to 10 million lines of Python code .. and we got close to 3,000 commits a day." IMHO, it would be scary with any language.

fideloper 12y ago

Is this different for Python over any dynamic language? This seems to be a better argument for statically compiled languages over dynamic languages rather than an argument against Python.

robmccoll 12y ago
That's a good point, and perhaps it is. The majority of my experience with dynamic languages is probably in Python. One thing that makes a difference between Python and other commonly used dynamic languages (here referring to JavaScript / ECMAScript and Lua) is that Python is dramatically more complex out of the box. This is both in terms of language features and Python's batteries-included standard libraries. For example, the amount of overloading that you can do in Python is very impressive, but the result is that common operations aren't necessarily predictable or dependable without more context (not that this isn't a problem in static languages - looking at you C++, but at least in the other languages you get type information to help out and some checking at compile time).
- thom_nic 12y ago
  Funny, my experience has been that large/ complex python code is actually easier to follow than JavaScript because of the line noise that JS requires. Side by side, the same code will usually be easier to read in Python (or Ruby.) Same goes for Java, although in a slightly different sense. I've seen non-trivial Java projects balloon quickly where you have dozens or hundreds of classes & interfaces with no clear sense of structure. High line noise, lots of boilerplate. Sure it compiles but at the end of the day it's just as likely to hit a NPE so you have to rely on functional and unit tests regardless of static/dynamic/compiled/interpreted. Obviously a lot of it comes down to who's writing the code. For me, a language that lets me succinctly express my intentions with minimal cruft is what wins.
  - herriojr 12y ago
    I totally agree that it comes down to who the code is written by. I've done both python and java, and in python I would go so far as to define interfaces as a form of documentation in my python code so people who use it know what I expect out of their classes which want to interact with my object. It doesn't require the class to explicitly implement the interface but they can if it makes sense (fairly similar to Zope Interfaces I guess). Unfortunately, I don't see this a lot in most python applications and you're left searching around through calls to figure out exactly what is required of the object being passed in. Look at a lot of popular libraries -- you are still left searching through code just to see what can possibly be returned because they aren't always the same type -- also, exceptions aren't part of the function or method definition, which means if you don't document your exceptions, I'm up shit creek. I think a lot of the problem around the python ecosystem, again, that I've seen, is people just don't follow best practices -- documentation, it's a core component if python development -- It's one of the arguments they use against static typing. Personally, I think python developers need to be much more strict about development practices than java developers. I think it's a lot easier to make bad python code than it is to make bad java code. The same goes for C, which I used to also do as well. To be a good C developer, you have to be very strict and structured (however, for somewhat different reasons). With great power comes great responsibility. I've seen shit java code as well. A lot of bad java code usually revolves around things not being modular or not having some form of consistent development patterns or not breaking methods down into simpler sub-problems -- I think documentation isn't as important as it is in python for the fact that I know exactly what is being returned, what exceptions can be thrown, and what exactly needs to be passed in just by looking at a method. In terms of the business logic associated with the class, that still needs to be documented. That said, I understand why python is generally used at startups -- it allows fast initial development where at startups, time is critical. Long term development really relies a lot on the teams ability to make structured decisions and organize their code, which is a difficult task in any language.

mpdehaan2 12y ago

This is an argument for unit tests, not for a statically typed language. Type errors tend to be very few and far between compared to logic errors. Multiple processes in most web applications are best handled by a pre-forking webserver in front of something like mod_wsgi, and using an engine like Celery for asynchronous and long-running operations on the backend, often launched with something like supervisor. I do take issue with some parts of the article, namely saying there is a good type system (there's not, but it's all fine, duck typing, etc) and that twisted is a good framework. It is definitely true however that Python is a great fit for all kinds of serious applications, but with all things, it takes discipline.

Retra 12y ago
What is the benefit to using a dynamically typed language if you have to spend all your time writing tests and verifying the static properties of your code?
- Vaskivo 12y ago
  The idea is that writing code + writing tests in a dynamically typed language is easier and faster just writing code in a statically typed language. (Of course it depends on the languages in question, the programmers and a number of other factors)
- spdy 12y ago
  Mainly the boilerplate other languages force upon you. Whenever i look at Java Code i start to chuckle. Its so much more information that you need to load into your head. To quote a picture from the post: https://www.paypal-engineering.com/wordpress/wp-content/uploads/2014/12/cpp_py_medium.png https://www.paypal-engineering.com/wordpress/wp-content/uplo... And its not like you need static typing everywhere just in some places its better to enforce it for your own sanity. Type hinting looks like a good way to solve this problem in python.
  - Retra 12y ago
    That image isn't really a fair comparison (in that C++ / Java aren't the only statically typed languages.) I imagine a statically typed Python dialect would only have an extra 15 or so lines, and the benefits would be numerous. I love python as much as anyone, but having to keep a reference manual handy just to use someone's library because they had the audacity to use a variable is not my idea of a fun time. EDIT: I'm probably exaggerating too much. Would I use Python in a large, complicated, multi-developer project? Yes, I would. And my only real complaint would probably be that I like static typing so much that I'd miss it.

mmahemoff 12y ago

Most of these runtime errors rarely make their way to production and are easily identified and rectified in cases when they do slip through. More restricted languages, e.g. Java, do have advantages in terms of forcing developers to do some more self-documenting and better opportunities for tools to help with syntax highlighting, refactoring, and warning about those errors you mention. But those errors are usually caught easily even with basic test procedures and almost always if there's a half-decent automated testing regimen. A lot of it comes down to, who's writing this big project? If developers are talented, there's a lot to be said for the freedom granted by a language like Python or Ruby. If developers are more mediocre, the forcing function of a language like Java may help to produce a "boring" and larger but understandable code base.

rtpg 12y ago
>If developers are talented, there's a lot to be said for the freedom granted by a language like Python or Ruby. If developers are more mediocre, the forcing function of a language like Java may help to produce a "boring" and larger but understandable code base. This is a bit of a false dichotomy. Much like I don't think lisp is popular for its functional-ness or whatnot, I don't think python or ruby are popular for their dynamic properties. My guess is that 90% of it is just how the code ends up looking. And with type inference getting further along, I think we'll see more people realizing that typing is a great sanity test to have in your code (especially in big codebases). See the popularity of Go in pythonic circles. I have never been shown any non-tiny piece of robust code that was easier to deal with because of a language's dynamic nature. I have been on this Earth a bit less time than others, but my gut feeling is that it doesn't actually exist. (note: my day job involves a lot of python, and I love the language for many things. I am looking forward to optional type annotations, though, as are many of my coworkers)

golergka 12y ago

http://news.efinancialcareers.com/us-en/173476/investment-banking-tech-guru-quits-starts-firm/ http://news.efinancialcareers.com/us-en/173476/investment-ba... "...There are around 5,000 developers using [Python] at Bank of America, .. there are close to 10 million lines of Python code .. and we got close to 3,000 commits a day." IMHO, it would be scary with any language.