7 ms·

i've been coding in python professionally for 4 years now, and i'm currently working on the biggest project i've ever worked on. to me, being scared of changing

by bsaul 13y ago

i've been coding in python professionally for 4 years now, and i'm currently working on the biggest project i've ever worked on. to me, being scared of changing the signature of a function because the static analyser will not be able to spot all the places i've used this function is a real problem ( along with incomplete autocomplete ). i do unit test everything, but i'd like to keep unit tests for things a computer can not theorically do. since i'm still in the early phase of the project, i know that python expressiveness is an edge, but i'm looking right now at what's going to be the "definitive" language i'm going to rebuild my product for the next 3 to 4 years. Python badly needs optional typing. really. i'm pretty sure that would solve both the speed and tooling issues. right now, for me, it starts to become unsuitable as soon as you reach 5-10k lines of code and a team of 2.

hosay123 13y ago
Test suites are awesome
- drdaeman 13y ago
  Running tests, even with coverage and quickcheck does not replace static verification. At very best, such tests can assert that program should probably work with some known probability. In contrast, strong statically-checked typing may be used prove that program produces results correctly to the extent type system may express the necessary proof.
  - alayne 13y ago
    Most type systems in statically typed languages can't make that guarantee. For example I can use Java reflection to call a method, change the method's parameters, and my code will only fail at runtime.
    - SCdF 13y ago
      That's kind of silly. If you're using Java and altering method signatures at runtime using reflection you're either in a very niche field where perhaps you shouldn't be using Java, or you're nuts. Just because you can get around a feature doesn't invalidate that feature. I'm not sure how I feel about static typing in Java, but it does do what OP talks about, because by default you're not playing devils advocate with reflection.
      - alayne 13y ago
        This problem happens with Spring all the time in Java which uses reflection. It's certainly not "silly" or a "niche field".
        Buttons840 13y ago
        I agree. With Java + Spring + XML you can get one language for the price of two! How useful is Java's type checking when one typo in an XML file breaks your program? At least in Python I can follow the logic of the language and find where the mistake is. Spring's XML files don't have such logic and usually all you have is a confusing traceback to work with.
      - quacker 13y ago
        I'm not sure how I feel about static typing in Java, but it does do what OP talks about, because by default you're not playing devils advocate with reflection. Static typing can guarantee the functionality of quite a lot of code, but it cannot do this for any arbitrary piece of code. As an example, no compiler in any language can know what whether a user will numbers or garbage, when you're expecting numbers. You need to check for an exception in Java even in this simple case.
  - nostrademons 13y ago
    When you're dealing with real-world systems you're always dealing with "probably work with some known probability". The hardware might burn out, or someone might trip over the cable, or a cosmic ray might randomly flip a bit and make your program go haywire. A proven-correct program doesn't help when you can't trust the physical infrastructure it runs on. Instead, you need to figure out what level of reliability you require and then work out a strategy to get there. Static-checking can be one element of this strategy, since you can use it to eliminate whole classes of bugs. But I've usually found that architectural solutions like canaries, automatic restarts, feature disablement, fallback codepaths, etc. are more effective at reducing errors per programmer-hour spent on them, and when you have those the reliability thresholds for the actual code you write can be much looser. (You probably still want to verify the hell out of the framework code that implements these architectural mechanisms, though.)
andrewcooke 13y ago
why did you choose python?!
- bsaul 13y ago
  That's a good question. There was my line of reasonning : 1_ I just coded something close to what i'm doing in java (using spring), and that made me feel to stay as far as i could from that bloated ecosystem (although the language in itself is ok) 2_ I looked at golang but was scared away when i saw that there wasn't generics or anything that would code some "meta level" functions. Plus, it still felt not broadly enough adopted at that time. Yet it IS the language i'm planning to rebuild my project for now (although i'll probably first rebuild just a small part of my api to see how the language behave with business process modeling and DB interactions, because i'm still very skeptical). 3_ I love C# , frankly the best language hands down, yet i don't like the idea of depending on the MS ecosystem (i've always felt like one day they'll come and make me pay big time). Plus windows seems like a falling platform to me. 4_ I just coded a moderately big website on django, and a big "sudoko-like" generator in pure python and loved the expressiveness and power of that language (as long as you're using the default data structures). Speed never was an issue, and since it was basically just a website, reading the django documentation was everything i needed. I had a look at ruby but frankly there wasn't a big enough difference to make me want to switch. Here's the story...
  - andrewcooke 13y ago
    thanks for the explanation. i know some people (the european southern observatory here in santiago) that use zope interfaces to address your concerns. i find the whole idea somewhat odd, but they seem to like it and make decent software, from what i've seen. http://stackoverflow.com/questions/2521189/purpose-of-zope-interfaces http://stackoverflow.com/questions/2521189/purpose-of-zope-i... http://docs.zope.org/zope.interface/ http://docs.zope.org/zope.interface/
    - bsaul 13y ago
      Thanks for the links. I didn't know that thing existed. I'm not sure i'm going to use it though. That's another library i'll be dependent on, for something the language itself should provide. To me, the fact that some projects had to create those things is a symptom that the language needs to evolve.
boothead 13y ago
I'm in a similar boat... Been programming python for ages, and now working on a huge, shared "hyper-agile" (yes they really call it that) code base. It's constantly breaking, even with unit tests! Personally, I'd love to be doing multi person large project stuff in Haskell... I can dream :-)
- bsaul 13y ago
  I had a look at Haskell, but i wonder whether Haskell couldn't become an abstraction mess either once the project reaches a certain size. Every time i have a look at Haskell snippets, i feel like i'm reading math. I'd like to read code like i read english...
  - marcosdumay 13y ago
    I guess it was meant to read like math. I think Haskell has the same problem as Lisp. Since you can abstract lots and lots of things away, there are tons of codebase specific abstractions that developers must keep on their heads before they can start coding. That said, the Haskell situation is better than Lisp's, because it has a big set of "default" abstractions that become part of the language, reducing the number of codebase specific ones.
  - kruipen 13y ago
    Interesting that you would say that. To me Haskell often feels like the most reads-like-English language; e.g. Parsec parsers written in applicative style, or function in pointfree style. Of course that is not true of all Haskell constructs...
  - boothead 13y ago
    I find that the unit of abstraction is smaller and more manageable in Haskell. What I mean by that it that there are a core set of abstractions based on mathematical laws that are ubiquitous and compose-able. Once you understand them and how they work together you see them everywhere and can very quickly understand new code you see. Contrast this with python or other more mainstream languages; you have similar "patterns" with inconsistent naming and implementation details which you first have to grok - every time mind - before you can start to understand the code.
smortaz 13y ago
bsaul, are you on windows by any chance? we're looking for some good large scale test cases for our python static analysis engine. would love to see how PTVS (pytools.codeplex.com) does on your project. thx.
- bsaul 13y ago
  sorry, i'm on a mac. I would love to try your tool though.
  - smortaz 13y ago
    awesome! if you have access to a windows box, you can install https://pytools.codeplex.com/downloads/get/697389 https://pytools.codeplex.com/downloads/get/697389 (400MB!) - it has PTVS + VS in one package (free). see this for a quick video overview: http://www.youtube.com/watch?v=TuewiStNT0M http://www.youtube.com/watch?v=TuewiStNT0M but basically you should do a "Create project from existing code", then wait for a while until the analysis engine parties on your code. if you run into any issues/questions, feel free to contact me at zatroms@microsoft.com (my login is spelled backwards). thanks!
spamizbad 13y ago
I'm surprised you're hitting that boundary at just 5-10K lines. I work on a team of 6* with about 60,000 logical Python statements (Not including tests) and aren't running into any language or framework induced roadblocks. And this isn't even a case of "If you do everything perfect like me, [Language] works great!" - We only have 45% code coverage, and our code architecture, in certain places, is really sub-optimal (Caused by us, not the language). When I'm about to make a major change to a function (different inputs or different outputs) I'll always start by grepping first to get an idea of what I'm about to get myself into, write or adjust tests for the new 'signature', change the function, and code/debug/test until it works. My team is in the loop before my code is committed to a shared branch. * We have other responsibilities besides the Python code. Edit: If your team is stepping all over each other at such a low LOC, my hunch is that its related to a communication problem with the team, your code is too tightly coupled, or there's not enough architecture planning (Too much is bad, but so is too little).
- runT1ME 13y ago
  It's also possible you have a 60k loc python app because you aren't able to get near the amount of code reuse or abstractions due to the lack of static typing, and this is why you haven't run into issues with refactoring.
  - Buttons840 13y ago
    If Wikipedia is to be believed: "Dynamic typing typically allows duck typing, which enables easier code reuse." I would expect to see more code reuse in Python than in any statically typed language.
    - coolsunglasses 13y ago
      It's a true statement if you qualify it properly. It wins you more code reusability with less code written to accomplish aforementioned reusability. I always run into silly inheritance chains, type coercion or adapter fns/methods to handle reusing code for unlike-types, rather than relying on a common base of functionality regardless of where it was derived from (function API, inheritance, mixin, etc). Best way I've seen this done is in Clojure. Good mix of the best of both worlds in terms of static and dynamic typing, multimethods, protocols, essentially structurally typed arguments, etc.
gcr 13y ago
Consider looking into Cython, a Python-->C source translator. http://docs.cython.org/index.html http://docs.cython.org/index.html Cython can turn Python source into a binary `.so` that you import, thus removing the overhead of the bytecode compiler/interpreter. You can also annotate any or all of your Python variables with C types. This way, Cython will generate raw high-performaning C using Python syntax. Other Python functions can call your optimized C versions, allowing you to optimize only the hotspots. Further, if you need to integrate with other languages, Cython can import other libraries' C functions and call them. Conversely, it can generate a header file for your Cython code, thus allowing other languages like C to call your existing Python or Cython functions.
- mercurial 13y ago
  My experience with Cython is limited to some hacking on lxml, but I found the compilation phase extremely slow. So, sure, you get performance but you get a big hit in terms of development speed.
  - chadzawistowski 13y ago
    I don't recall Cython's default optimization level, but you may be able to decrease compilation time by adding a -O0 flag for debug builds. This StackOverflow question addresses adding optimization flags to a Cython setup.py script: http://stackoverflow.com/q/16285011/1628916 http://stackoverflow.com/q/16285011/1628916
  - gcr 13y ago
    Huh. I find that if you have a short cython script, you can just hack aruond in the IPython Notebook. Evaluating a 200-line cython script takes under a second for me, which is fine for my purposes.
reinhardt 13y ago
(ack-)grep is your friend.
- anymane 13y ago
  Or the pythonic replacement grin: https://pypi.python.org/pypi/grin https://pypi.python.org/pypi/grin
njharman 13y ago
I thought python had optional typing http://www.python.org/dev/peps/pep-3107/ http://www.python.org/dev/peps/pep-3107/
- chubot 13y ago
  Yeah that's only Python 3, which most people aren't using yet. Also it's not optional typing -- it's optional annotations. Someone would still have to actually implement the type checking -- not even sure that's been done. I have wanted to play around with implementing some type checking, but none of my stuff is Python 3 unfortunately.
  - timmipetit 13y ago
    Check out MyPy: http://www.mypy-lang.org/ http://www.mypy-lang.org/ It uses the Python 3 annotations syntax for type checking.
    - bsaul 13y ago
      MyPy is the only thing that would make me keep coding in python in the long term. Do you know how mature it is ?
- bsaul 13y ago
  I read that and it gave me 5 seconds of hope. But then "The only way that annotations take on meaning is when they are interpreted by third-party libraries" and "Following from point 2, this PEP makes no attempt to introduce any kind of standard semantics, even for the built-in types. This work will be left to third-party libraries" means the author really isn't serious about it. If you don't have a standard semantic you've got nothing useful.
pjmlp 13y ago
This is why I only advise the use of dynamic languages for small projects. This is specially an issue in the enterprise world where very few people care about unit testing.

by bsaul 13y ago

hosay123 13y ago

Test suites are awesome

drdaeman 13y ago
Running tests, even with coverage and quickcheck does not replace static verification. At very best, such tests can assert that program should probably work with some known probability. In contrast, strong statically-checked typing may be used prove that program produces results correctly to the extent type system may express the necessary proof.
- alayne 13y ago
  Most type systems in statically typed languages can't make that guarantee. For example I can use Java reflection to call a method, change the method's parameters, and my code will only fail at runtime.
  - SCdF 13y ago
    That's kind of silly. If you're using Java and altering method signatures at runtime using reflection you're either in a very niche field where perhaps you shouldn't be using Java, or you're nuts. Just because you can get around a feature doesn't invalidate that feature. I'm not sure how I feel about static typing in Java, but it does do what OP talks about, because by default you're not playing devils advocate with reflection.
    - alayne 13y ago
      This problem happens with Spring all the time in Java which uses reflection. It's certainly not "silly" or a "niche field".
      - Buttons840 13y ago
        I agree. With Java + Spring + XML you can get one language for the price of two! How useful is Java's type checking when one typo in an XML file breaks your program? At least in Python I can follow the logic of the language and find where the mistake is. Spring's XML files don't have such logic and usually all you have is a confusing traceback to work with.
    - quacker 13y ago
      I'm not sure how I feel about static typing in Java, but it does do what OP talks about, because by default you're not playing devils advocate with reflection. Static typing can guarantee the functionality of quite a lot of code, but it cannot do this for any arbitrary piece of code. As an example, no compiler in any language can know what whether a user will numbers or garbage, when you're expecting numbers. You need to check for an exception in Java even in this simple case.
- nostrademons 13y ago
  When you're dealing with real-world systems you're always dealing with "probably work with some known probability". The hardware might burn out, or someone might trip over the cable, or a cosmic ray might randomly flip a bit and make your program go haywire. A proven-correct program doesn't help when you can't trust the physical infrastructure it runs on. Instead, you need to figure out what level of reliability you require and then work out a strategy to get there. Static-checking can be one element of this strategy, since you can use it to eliminate whole classes of bugs. But I've usually found that architectural solutions like canaries, automatic restarts, feature disablement, fallback codepaths, etc. are more effective at reducing errors per programmer-hour spent on them, and when you have those the reliability thresholds for the actual code you write can be much looser. (You probably still want to verify the hell out of the framework code that implements these architectural mechanisms, though.)

andrewcooke 13y ago

why did you choose python?!

bsaul 13y ago
That's a good question. There was my line of reasonning : 1_ I just coded something close to what i'm doing in java (using spring), and that made me feel to stay as far as i could from that bloated ecosystem (although the language in itself is ok) 2_ I looked at golang but was scared away when i saw that there wasn't generics or anything that would code some "meta level" functions. Plus, it still felt not broadly enough adopted at that time. Yet it IS the language i'm planning to rebuild my project for now (although i'll probably first rebuild just a small part of my api to see how the language behave with business process modeling and DB interactions, because i'm still very skeptical). 3_ I love C# , frankly the best language hands down, yet i don't like the idea of depending on the MS ecosystem (i've always felt like one day they'll come and make me pay big time). Plus windows seems like a falling platform to me. 4_ I just coded a moderately big website on django, and a big "sudoko-like" generator in pure python and loved the expressiveness and power of that language (as long as you're using the default data structures). Speed never was an issue, and since it was basically just a website, reading the django documentation was everything i needed. I had a look at ruby but frankly there wasn't a big enough difference to make me want to switch. Here's the story...
- andrewcooke 13y ago
  thanks for the explanation. i know some people (the european southern observatory here in santiago) that use zope interfaces to address your concerns. i find the whole idea somewhat odd, but they seem to like it and make decent software, from what i've seen. http://stackoverflow.com/questions/2521189/purpose-of-zope-interfaces http://stackoverflow.com/questions/2521189/purpose-of-zope-i... http://docs.zope.org/zope.interface/ http://docs.zope.org/zope.interface/
  - bsaul 13y ago
    Thanks for the links. I didn't know that thing existed. I'm not sure i'm going to use it though. That's another library i'll be dependent on, for something the language itself should provide. To me, the fact that some projects had to create those things is a symptom that the language needs to evolve.

boothead 13y ago

I'm in a similar boat... Been programming python for ages, and now working on a huge, shared "hyper-agile" (yes they really call it that) code base. It's constantly breaking, even with unit tests! Personally, I'd love to be doing multi person large project stuff in Haskell... I can dream :-)

bsaul 13y ago
I had a look at Haskell, but i wonder whether Haskell couldn't become an abstraction mess either once the project reaches a certain size. Every time i have a look at Haskell snippets, i feel like i'm reading math. I'd like to read code like i read english...
- marcosdumay 13y ago
  I guess it was meant to read like math. I think Haskell has the same problem as Lisp. Since you can abstract lots and lots of things away, there are tons of codebase specific abstractions that developers must keep on their heads before they can start coding. That said, the Haskell situation is better than Lisp's, because it has a big set of "default" abstractions that become part of the language, reducing the number of codebase specific ones.
- kruipen 13y ago
  Interesting that you would say that. To me Haskell often feels like the most reads-like-English language; e.g. Parsec parsers written in applicative style, or function in pointfree style. Of course that is not true of all Haskell constructs...
- boothead 13y ago
  I find that the unit of abstraction is smaller and more manageable in Haskell. What I mean by that it that there are a core set of abstractions based on mathematical laws that are ubiquitous and compose-able. Once you understand them and how they work together you see them everywhere and can very quickly understand new code you see. Contrast this with python or other more mainstream languages; you have similar "patterns" with inconsistent naming and implementation details which you first have to grok - every time mind - before you can start to understand the code.

smortaz 13y ago

bsaul, are you on windows by any chance? we're looking for some good large scale test cases for our python static analysis engine. would love to see how PTVS (pytools.codeplex.com) does on your project. thx.

bsaul 13y ago
sorry, i'm on a mac. I would love to try your tool though.
- smortaz 13y ago
  awesome! if you have access to a windows box, you can install https://pytools.codeplex.com/downloads/get/697389 https://pytools.codeplex.com/downloads/get/697389 (400MB!) - it has PTVS + VS in one package (free). see this for a quick video overview: http://www.youtube.com/watch?v=TuewiStNT0M http://www.youtube.com/watch?v=TuewiStNT0M but basically you should do a "Create project from existing code", then wait for a while until the analysis engine parties on your code. if you run into any issues/questions, feel free to contact me at zatroms@microsoft.com (my login is spelled backwards). thanks!

spamizbad 13y ago

I'm surprised you're hitting that boundary at just 5-10K lines. I work on a team of 6* with about 60,000 logical Python statements (Not including tests) and aren't running into any language or framework induced roadblocks. And this isn't even a case of "If you do everything perfect like me, [Language] works great!" - We only have 45% code coverage, and our code architecture, in certain places, is really sub-optimal (Caused by us, not the language). When I'm about to make a major change to a function (different inputs or different outputs) I'll always start by grepping first to get an idea of what I'm about to get myself into, write or adjust tests for the new 'signature', change the function, and code/debug/test until it works. My team is in the loop before my code is committed to a shared branch. * We have other responsibilities besides the Python code. Edit: If your team is stepping all over each other at such a low LOC, my hunch is that its related to a communication problem with the team, your code is too tightly coupled, or there's not enough architecture planning (Too much is bad, but so is too little).

runT1ME 13y ago
It's also possible you have a 60k loc python app because you aren't able to get near the amount of code reuse or abstractions due to the lack of static typing, and this is why you haven't run into issues with refactoring.
- Buttons840 13y ago
  If Wikipedia is to be believed: "Dynamic typing typically allows duck typing, which enables easier code reuse." I would expect to see more code reuse in Python than in any statically typed language.
  - coolsunglasses 13y ago
    It's a true statement if you qualify it properly. It wins you more code reusability with less code written to accomplish aforementioned reusability. I always run into silly inheritance chains, type coercion or adapter fns/methods to handle reusing code for unlike-types, rather than relying on a common base of functionality regardless of where it was derived from (function API, inheritance, mixin, etc). Best way I've seen this done is in Clojure. Good mix of the best of both worlds in terms of static and dynamic typing, multimethods, protocols, essentially structurally typed arguments, etc.

gcr 13y ago

Consider looking into Cython, a Python-->C source translator. http://docs.cython.org/index.html http://docs.cython.org/index.html Cython can turn Python source into a binary `.so` that you import, thus removing the overhead of the bytecode compiler/interpreter. You can also annotate any or all of your Python variables with C types. This way, Cython will generate raw high-performaning C using Python syntax. Other Python functions can call your optimized C versions, allowing you to optimize only the hotspots. Further, if you need to integrate with other languages, Cython can import other libraries' C functions and call them. Conversely, it can generate a header file for your Cython code, thus allowing other languages like C to call your existing Python or Cython functions.

mercurial 13y ago
My experience with Cython is limited to some hacking on lxml, but I found the compilation phase extremely slow. So, sure, you get performance but you get a big hit in terms of development speed.
- chadzawistowski 13y ago
  I don't recall Cython's default optimization level, but you may be able to decrease compilation time by adding a -O0 flag for debug builds. This StackOverflow question addresses adding optimization flags to a Cython setup.py script: http://stackoverflow.com/q/16285011/1628916 http://stackoverflow.com/q/16285011/1628916
- gcr 13y ago
  Huh. I find that if you have a short cython script, you can just hack aruond in the IPython Notebook. Evaluating a 200-line cython script takes under a second for me, which is fine for my purposes.

reinhardt 13y ago

(ack-)grep is your friend.

anymane 13y ago
Or the pythonic replacement grin: https://pypi.python.org/pypi/grin https://pypi.python.org/pypi/grin

njharman 13y ago

I thought python had optional typing http://www.python.org/dev/peps/pep-3107/ http://www.python.org/dev/peps/pep-3107/

chubot 13y ago
Yeah that's only Python 3, which most people aren't using yet. Also it's not optional typing -- it's optional annotations. Someone would still have to actually implement the type checking -- not even sure that's been done. I have wanted to play around with implementing some type checking, but none of my stuff is Python 3 unfortunately.
- timmipetit 13y ago
  Check out MyPy: http://www.mypy-lang.org/ http://www.mypy-lang.org/ It uses the Python 3 annotations syntax for type checking.
  - bsaul 13y ago
    MyPy is the only thing that would make me keep coding in python in the long term. Do you know how mature it is ?
bsaul 13y ago
I read that and it gave me 5 seconds of hope. But then "The only way that annotations take on meaning is when they are interpreted by third-party libraries" and "Following from point 2, this PEP makes no attempt to introduce any kind of standard semantics, even for the built-in types. This work will be left to third-party libraries" means the author really isn't serious about it. If you don't have a standard semantic you've got nothing useful.

pjmlp 13y ago

This is why I only advise the use of dynamic languages for small projects. This is specially an issue in the enterprise world where very few people care about unit testing.