7 ms·

Former Cython developer here. To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if yo

by dagss 10y ago

Former Cython developer here. To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia -- all specifically made for scientific programming with 0% general purpose/web/etc. development going on. So I would say that the scientific Python community has been doing extremely well in terms of even using a language that isn't designed ground up for scientific computing. I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) -- I'll refrain, I just wanted to say that were you see something and get frustrated, I see the same picture and think it's actually an incredible success, to bring so many scientists onto at least the same ball park as other programmers, even if they are still playing their own game.

p4wnc6 10y ago
Slight aside -- I am a long-time user of Cython and would love to contribute, but poking around the source code feels daunting -- like there is a huge learning curve to be able to contribute, and that other contributors will only view newcomers as an annoyance who will drain their time with questions. Given all this, what's the best way for someone to pick up and start contributing to Cython?
- dagss 10y ago
  It definitely is daunting; in particular because Cython isn't so much a goal as itself as a tool that developers are motivated to work on it because they use it in some other project they care about (e.g., lxml, Sage, ...). So it's been pragmatic development. If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer). Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere). Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code. It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.
whyrt12 10y ago
Julia has tons of general purpose development going, including an mvc web framework (genie.jl) and a reactive web app framework called (escher.jl)
- dagss 10y ago
  First, sorry for being a bit heavy on the hyperbole and saying "0%"; that is almost guaranteed to be wrong as a statement and also let me say I don't know Julia very well at all. I want to clarify that when I say "0% web development going on", I don't mean that scientific programmers don't do web development (they do! a lot!), I meant that people don't pick it those languages in general if they are only doing web development without a numerical/scientific/statistical aspect to it. What would be interesting is, do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background? The scientific Python community certainly makes good use of all the Python web tools!, and being a "scientific stack" in no way precludes the need for general purpose frameworks that can also be used by others. It's at least as much about people and community and habits as about the tooling...
  - Jcol1 10y ago
    I think its just a matter of time before this happens (its mature enough or an important app is released). Julia is designed for general purpose computing from day one, but this community is not dealing with the same painpoints that scientists have been.
  - tavert 10y ago
    > do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background? It's catching on among early-adopters in finance. What do you consider that?
    - coldtea 10y ago
      Is it? Where are they? And how many are they?
Athas 10y ago
> I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) Please do! I'm a PhD student in CS, and I don't think I "get" scientific computing (I'm in compilers myself).
- kaeluka 10y ago
  +1, I'm a PhD student in programming languages/PL design
- enriquto 10y ago
  You want to process (typically, with simple arithmetical operations) huge arrays of numbers. Imagine that you want to sum two arrays of 800 MB of floating point numbers each. This is one step inside the loop of your algorithm. You can do that natively in C, and the result is very fast. You can do that natively in Fortran. And in Matlab, etc. You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
  - kevin_thibedeau 10y ago
    > You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower. NumPy + Cython would beg to differ.
    - enriquto 10y ago
      Of course, but not natively. Try doing that in plain python using a list of 8000 lists of 8000 lists of 3 numbers (e.g. a mid-resolution microscopy image).
    - dagss 10y ago
      NumPy is about one order of magnitude slower in very many situations. Cython is in some ways a reimplementation of Fortran in places -- I mean, you can consider it a better alternative but have to use features that are not in Python. Personally I gave up writing numerical code in Cython and instead wrote it in Fortran and merely wrapped it with Cython... But yes, being 2x-10x slower is something one can live with for productivity, vs sometimes 1000x slower of Python.
dalke 10y ago
As an aside, these sorts of comments about the needs of a "scientific programmer" always irk me. I've been doing scientific software development now for 20 years. I do non-numerical scientific computing, originally structural biology, then bioinformatics, and now chemical informatics, the last dominated by graph theory. I rarely use NumPy and effectively never the languages you mentioned. Last year on one project I did use a hypergeometric survival function from SciPy, then re-implemented it in Python so I wouldn't have the large dependency for what was a few tens of lines of code. Biopython, as another example, has almost no dependencies on NumPy, and works under PyPy.
- forgotpwtomain 10y ago
  It would be awesome if you have a blog post or another resource that contains some of your work experience that you can link? I'd be very interested.
  - dalke 10y ago
    The best I can offer are my previous HN comments on the topic, at https://hn.algolia.com/?query=dalke%20numpy&sort=byPopularity&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=dalke%20numpy&sort=byPopularit... and http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html http://www.dalkescientific.com/writings/diary/archive/2011/1... . I don't see what's awesome or all that interesting about it.
hyperbovine 10y ago
> MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia You are so correct. I have not used Julia, but what do the first four have in common? As languages, they suck. Each in their own way can be used to do amazing, incredible things. But from a development perspective, they are pure torture to code in. The joy of writing scientific code in Python is that there is a whole set of users, the majority really, who have nothing to do with science. The language must stand on its own, so Python cannot suck.

by dagss 10y ago

p4wnc6 10y ago

Slight aside -- I am a long-time user of Cython and would love to contribute, but poking around the source code feels daunting -- like there is a huge learning curve to be able to contribute, and that other contributors will only view newcomers as an annoyance who will drain their time with questions. Given all this, what's the best way for someone to pick up and start contributing to Cython?

dagss 10y ago
It definitely is daunting; in particular because Cython isn't so much a goal as itself as a tool that developers are motivated to work on it because they use it in some other project they care about (e.g., lxml, Sage, ...). So it's been pragmatic development. If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer). Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere). Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code. It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.

whyrt12 10y ago

Julia has tons of general purpose development going, including an mvc web framework (genie.jl) and a reactive web app framework called (escher.jl)

dagss 10y ago
First, sorry for being a bit heavy on the hyperbole and saying "0%"; that is almost guaranteed to be wrong as a statement and also let me say I don't know Julia very well at all. I want to clarify that when I say "0% web development going on", I don't mean that scientific programmers don't do web development (they do! a lot!), I meant that people don't pick it those languages in general if they are only doing web development without a numerical/scientific/statistical aspect to it. What would be interesting is, do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background? The scientific Python community certainly makes good use of all the Python web tools!, and being a "scientific stack" in no way precludes the need for general purpose frameworks that can also be used by others. It's at least as much about people and community and habits as about the tooling...
- Jcol1 10y ago
  I think its just a matter of time before this happens (its mature enough or an important app is released). Julia is designed for general purpose computing from day one, but this community is not dealing with the same painpoints that scientists have been.
- tavert 10y ago
  > do you know of any teams or companies using Julia in anger in a non-scientific setting, with programmers from a non-scientific background? It's catching on among early-adopters in finance. What do you consider that?
  - coldtea 10y ago
    Is it? Where are they? And how many are they?

Athas 10y ago

> I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) Please do! I'm a PhD student in CS, and I don't think I "get" scientific computing (I'm in compilers myself).

kaeluka 10y ago
+1, I'm a PhD student in programming languages/PL design
enriquto 10y ago
You want to process (typically, with simple arithmetical operations) huge arrays of numbers. Imagine that you want to sum two arrays of 800 MB of floating point numbers each. This is one step inside the loop of your algorithm. You can do that natively in C, and the result is very fast. You can do that natively in Fortran. And in Matlab, etc. You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower.
- kevin_thibedeau 10y ago
  > You cannot do that at all in Python. Well, you can, but it will be orders of magnitude slower. NumPy + Cython would beg to differ.
  - enriquto 10y ago
    Of course, but not natively. Try doing that in plain python using a list of 8000 lists of 8000 lists of 3 numbers (e.g. a mid-resolution microscopy image).
  - dagss 10y ago
    NumPy is about one order of magnitude slower in very many situations. Cython is in some ways a reimplementation of Fortran in places -- I mean, you can consider it a better alternative but have to use features that are not in Python. Personally I gave up writing numerical code in Cython and instead wrote it in Fortran and merely wrapped it with Cython... But yes, being 2x-10x slower is something one can live with for productivity, vs sometimes 1000x slower of Python.

dalke 10y ago

As an aside, these sorts of comments about the needs of a "scientific programmer" always irk me. I've been doing scientific software development now for 20 years. I do non-numerical scientific computing, originally structural biology, then bioinformatics, and now chemical informatics, the last dominated by graph theory. I rarely use NumPy and effectively never the languages you mentioned. Last year on one project I did use a hypergeometric survival function from SciPy, then re-implemented it in Python so I wouldn't have the large dependency for what was a few tens of lines of code. Biopython, as another example, has almost no dependencies on NumPy, and works under PyPy.

forgotpwtomain 10y ago
It would be awesome if you have a blog post or another resource that contains some of your work experience that you can link? I'd be very interested.
- dalke 10y ago
  The best I can offer are my previous HN comments on the topic, at https://hn.algolia.com/?query=dalke%20numpy&sort=byPopularity&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=dalke%20numpy&sort=byPopularit... and http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html http://www.dalkescientific.com/writings/diary/archive/2011/1... . I don't see what's awesome or all that interesting about it.

hyperbovine 10y ago

> MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia You are so correct. I have not used Julia, but what do the first four have in common? As languages, they suck. Each in their own way can be used to do amazing, incredible things. But from a development perspective, they are pure torture to code in. The joy of writing scientific code in Python is that there is a whole set of users, the majority really, who have nothing to do with science. The language must stand on its own, so Python cannot suck.