4 ms·

I do love Python, but I often wish there was more rigor for some things. I currently work on a large project (>500K LOC) and have started to see it fall apart

I do love Python, but I often wish there was more rigor for some things. I currently work on a large project (>500K LOC) and have started to see it fall apart with a bigger team. Lack of enforced typing in function arguments, inability to create strict interfaces, inconsistency in standard library conventions, and package management would probably be my biggest gripes.

underwater 15y ago
I've seen projects that use decorators for type checking. You could turn it into a noop for production.
bugsbunnyak 15y ago
Curious: what is this 500k LOC project (if you can say) and how much of that is Python? If it's an in-house codebase that's a really neat datapoint because it would rank among the largest open-source Python projects.. Also, Traits (from Enthought) is an attempt to deal with some of the issues of typing and interfaces. It's led to some great software (the Chaco plotting package is fantastic, as is Mayavi), but I've yet to figure out how to debug a Traits error and stay sane. There is a lot of auto-configured magic that happens, and when something goes wrong the backtrace is often pages long and doesn't actually show the offending line.
- keypusher 15y ago
  Project is automated testing for an enterprise private cloud platform. 500K of tests and test library is all python, the application itself is probably 2M+ LOC, Java + C running on Linux.
cdavid 15y ago
On most projects I have seen where python was seen as problematic, I think that having types ala C++/Java would not have improved much. The real problem is that the function is not documented, nor is the function expectations. Typing in C++/java does not replace that (more advanced typing systems do much better, though). Package management and generally deployment is certainly a pain in python. I think it is one of the main issue in typical "corporate environments"
- nostrademons 15y ago
  In most C++/Java code, the function is not documented either. The nice thing about manifest types is that the source code becomes the documentation (shitty documentation, yes, but usually good enough that you can figure out how to use it by looking at the source). Type inference with a doc generator would work too, as long as you run the doc generator regularly and everybody knows where to find the docs. History has shown that programmers are too lazy/busy to write good documentation, and when they do, they don't keep it in sync with what the code does. If it's not checked by the compiler or automated tests, it's probably wrong.
  - deleted 15y ago
    [deleted]
  - cdavid 15y ago
    It is not my experience that you can figure out how to call functions correctly just from the function signature. Certainly, that's not true from C++ when you need to figure out lifetime issues, etc… It may be possible to figure out from the source code, if the code is written well enough, but that's true independently of the language IMO. Developer not writing decent doc is an institutional problem, and if this cannot be enforced, then the institution is most likely not capable of producing very good code either. I think it is more interesting to talk from the POV of how to write code that is actually readable: type may help, but that's only a not that important part of the equation in my experience.
    - nostrademons 15y ago
      C++ with good coding conventions can encode pretty much all the information you need in the type. If a parameter is read-only, pass by const reference. If it's writable but ownership is not transferred, pass by pointer. If ownership is transferred, pass by unique_ptr. If it's owned within an object or function lifetime, use a scoped_ptr. If it can't be copied, inherit from boost::noncopyable.
- gtaylor 15y ago
  This, this, and this. We document our stuff using ReST and Sphinx. You can optionally specify data types in the docstrings, which does a ton to help with this. It isn't time-consuming, and it makes your team not hate reading your stuff.
spenrose 15y ago
500K LOC in Python is what, 2.5M->5M in a statically typed language? At that size I would expect any project to require explicit convention and communication management, simply because no one person can hold the whole thing in their mind at a resolution that reveals the code's meaningful features.
- jimbokun 15y ago
  "500K LOC in Python is what, 2.5M->5M in a statically typed language?" Depends if the statically typed language is Java or Haskell.
- tikhonj 15y ago
  And it's probably 200k lines in Haskell, which is very statically typed :) (For reference, GHC--a very complicated compiler with clever optimizations, language extensions, multiple backends...etc--is something like 125k lines of Haskell.) I've spent a significant amount of time with Python, both at work and in my classes and yet my Haskell code is usually 2-3x shorter for very similar programs. I think it's also easier to read, but that's indubitably a matter of taste.
  - attractivechaos 15y ago
    That is interesting. According to the computer language benchmark games, Python programs always have fewer LOCs: http://shootout.alioth.debian.org/u64/benchmark.php?test=all&lang=ghc&lang2=python3 http://shootout.alioth.debian.org/u64/benchmark.php?test=all... I know most programs there are written in an unusual way, but that at least means something.
    - tikhonj 15y ago
      There are a couple of things to note about that particular comparison. First, Haskell--particularly on GHC--gives you a ton of options for speeding your code up, including unsafe* functions and pragmas. So the fastest code is going to use things like {-# INLINE #-} and unsafeAt as opposed to ! (the array index operator). However, you would only use these things in performance critical bits of your code. Secondly, the length of the identifier (! vs unsafeAt) actually matters because the benchmarks do not measure the number of lines--they measure the compressed size of the code. [1] [1]: http://shootout.alioth.debian.org/help.php#gzbytes http://shootout.alioth.debian.org/help.php#gzbytes Finally, these are all small algorithmic problems. The goal is to write the fastest code possible, not something readable or maintainable. Some of the biggest reductions in Haskell come from generalizing and reusing functions in different contexts; I expect this does not happen in the benchmarks because of limited size and the focus on performance.

underwater 15y ago

I've seen projects that use decorators for type checking. You could turn it into a noop for production.

bugsbunnyak 15y ago

Curious: what is this 500k LOC project (if you can say) and how much of that is Python? If it's an in-house codebase that's a really neat datapoint because it would rank among the largest open-source Python projects.. Also, Traits (from Enthought) is an attempt to deal with some of the issues of typing and interfaces. It's led to some great software (the Chaco plotting package is fantastic, as is Mayavi), but I've yet to figure out how to debug a Traits error and stay sane. There is a lot of auto-configured magic that happens, and when something goes wrong the backtrace is often pages long and doesn't actually show the offending line.

keypusher 15y ago
Project is automated testing for an enterprise private cloud platform. 500K of tests and test library is all python, the application itself is probably 2M+ LOC, Java + C running on Linux.

cdavid 15y ago

On most projects I have seen where python was seen as problematic, I think that having types ala C++/Java would not have improved much. The real problem is that the function is not documented, nor is the function expectations. Typing in C++/java does not replace that (more advanced typing systems do much better, though). Package management and generally deployment is certainly a pain in python. I think it is one of the main issue in typical "corporate environments"

nostrademons 15y ago
In most C++/Java code, the function is not documented either. The nice thing about manifest types is that the source code becomes the documentation (shitty documentation, yes, but usually good enough that you can figure out how to use it by looking at the source). Type inference with a doc generator would work too, as long as you run the doc generator regularly and everybody knows where to find the docs. History has shown that programmers are too lazy/busy to write good documentation, and when they do, they don't keep it in sync with what the code does. If it's not checked by the compiler or automated tests, it's probably wrong.
- deleted 15y ago
  [deleted]
- cdavid 15y ago
  It is not my experience that you can figure out how to call functions correctly just from the function signature. Certainly, that's not true from C++ when you need to figure out lifetime issues, etc… It may be possible to figure out from the source code, if the code is written well enough, but that's true independently of the language IMO. Developer not writing decent doc is an institutional problem, and if this cannot be enforced, then the institution is most likely not capable of producing very good code either. I think it is more interesting to talk from the POV of how to write code that is actually readable: type may help, but that's only a not that important part of the equation in my experience.
  - nostrademons 15y ago
    C++ with good coding conventions can encode pretty much all the information you need in the type. If a parameter is read-only, pass by const reference. If it's writable but ownership is not transferred, pass by pointer. If ownership is transferred, pass by unique_ptr. If it's owned within an object or function lifetime, use a scoped_ptr. If it can't be copied, inherit from boost::noncopyable.
gtaylor 15y ago
This, this, and this. We document our stuff using ReST and Sphinx. You can optionally specify data types in the docstrings, which does a ton to help with this. It isn't time-consuming, and it makes your team not hate reading your stuff.

spenrose 15y ago

500K LOC in Python is what, 2.5M->5M in a statically typed language? At that size I would expect any project to require explicit convention and communication management, simply because no one person can hold the whole thing in their mind at a resolution that reveals the code's meaningful features.

jimbokun 15y ago
"500K LOC in Python is what, 2.5M->5M in a statically typed language?" Depends if the statically typed language is Java or Haskell.
tikhonj 15y ago
And it's probably 200k lines in Haskell, which is very statically typed :) (For reference, GHC--a very complicated compiler with clever optimizations, language extensions, multiple backends...etc--is something like 125k lines of Haskell.) I've spent a significant amount of time with Python, both at work and in my classes and yet my Haskell code is usually 2-3x shorter for very similar programs. I think it's also easier to read, but that's indubitably a matter of taste.
- attractivechaos 15y ago
  That is interesting. According to the computer language benchmark games, Python programs always have fewer LOCs: http://shootout.alioth.debian.org/u64/benchmark.php?test=all&lang=ghc&lang2=python3 http://shootout.alioth.debian.org/u64/benchmark.php?test=all... I know most programs there are written in an unusual way, but that at least means something.
  - tikhonj 15y ago
    There are a couple of things to note about that particular comparison. First, Haskell--particularly on GHC--gives you a ton of options for speeding your code up, including unsafe* functions and pragmas. So the fastest code is going to use things like {-# INLINE #-} and unsafeAt as opposed to ! (the array index operator). However, you would only use these things in performance critical bits of your code. Secondly, the length of the identifier (! vs unsafeAt) actually matters because the benchmarks do not measure the number of lines--they measure the compressed size of the code. [1] [1]: http://shootout.alioth.debian.org/help.php#gzbytes http://shootout.alioth.debian.org/help.php#gzbytes Finally, these are all small algorithmic problems. The goal is to write the fastest code possible, not something readable or maintainable. Some of the biggest reductions in Haskell come from generalizing and reusing functions in different contexts; I expect this does not happen in the benchmarks because of limited size and the focus on performance.