5 ms·

If you subscribe to the idea of a division of labor between systems and scripting languages, then Python + its C based extensions (Cython, Numpy, etc.) is ideal

If you subscribe to the idea of a division of labor between systems and scripting languages, then Python + its C based extensions (Cython, Numpy, etc.) is ideal, as it already has an extensive scientific ecosystem. Julia combines the two in a single language. See this blog post and the one that follows it: http://graydon2.dreamwidth.org/3186.html http://graydon2.dreamwidth.org/3186.html

marcodena 12y ago
nice post!
lottin 12y ago
Do one thing and do it well, seems like a good idea. Of course, do everything better than anyone else is even better, but is it really possible? I'm not sold on the idea of a one-size-fits-all programming language.
- coldtea 12y ago
  Well, if there was some theorical barrier, and if we were indeed talking of making the "100% perfect" language" yes. But in real life things preventing using a language in more domains are mostly mere bad design, lack of some features and legacy baggage. Case in point with Python. It's not that we want it to be faster or lower level than C to use in kernels. We just want it to be 10x, 20x faster, something entirely possible (Javascript JITs prove it can be done for even the most dynamic of languages), and we just want it to be able to tap into multicore CPUs (something that would also be entirely possible if it was slightly better designed). So it's not like there's some huge theoritical barrier to achieving this stuff. It's mostly money (like those spend by Apple, Google and Mozilla to their JIT JS engines), and the will to drop some backwards compatibility (which they did with 3.0 but for marginal benefits instead of addressing useful stuff).
srean 12y ago
It is light years far from ideal. There are two variants of this much parroted line (i) Numpy, Scipy are just C loops, and (ii) 'just do the intensive parts in C', both leave a lot more to be desired. Yes numpy, scipy indeed dispatch to precompiled C and sometimes Fortran loops but the problem lies elsewhere, in its vectorization paradigm. It is just extremely wasteful. There are two problems: (a) it is not expressive enough to capture efficient computation without generating unnecessary intermediate arrays whose sole objective is to make it possible to write the computation as a vectorized operation. Unlike Matlab in the past, numpy, scipy are at least smart about broadcasting. This often allows one to avoid constructing those intermediates in memory. However, this comes at an extra indirection that affects all array operations via the stride vector. You pay the cost of indirection whether you need it or not. (b) The second problem is generation of temporaries when you chain several binary operations. These temporaries get allocated, filled and destroyed over and over again within a single expression which itself might be in a loop. This costs computation, memory, not to mention GC pressure. There is of course numexpr but it is also quite limiting. For instance you cannot index or slice from within a numexpr expression. It offers limited set of reduce operations, in an expression you can use only one, and it must be the last one in the sequence of operations. Then there is 'write to C'. If we have not eliminated the need to write C we have not really solved the hard problems have we. I think the whole point was to avoid writing low level code because it is error prone, tedious and that it often comes at the cost of productivity. The drop down to C imposes an unnecessary break in flow and forces you to tackle the impedance mismatch. Tools like Cython eases this a bit. You cannot for example use numpy array expressions efficiently from Cython, you have to write those tedious low level indexing code. If I were to write C, I would rather write it in C syntax and take advantage of the decades of tooling around C syntax. Cython is great and an awesome community effort, but it still quite a simple compiler and has limitations. So far I have been talking only about ease of use, quality of programming experience etc etc, but that is not the only issue here. The problem is calls to C and more importantly callbacks from C to Python are expensive enough to be non-ignorable. If you have a hot loop where you go back and forth between C/Fortran world and Python thats going to incur a serious hit. The solution is to make the containing piece of Code into C/Fortran/Cython, so it ends up swallowing more and more of the application logic, leaving only but a shell of I/O in the Python world. Its not the end of the world but not quite the rosy picture you give. Another issue is cultural, its common among many newer programmers not to have really experienced fast runtimes, of course this is a generalization and does not apply to all, but have seen it happen frequently enough. They are greatly amazed by what I would call only modest improvement in runtimes and they would be cheering "Wow! so speed much fast!" etc. All of these make me be really hopeful about Julia. Interacting with the community gives me the feeling that they get it. Julia is an expressive language, already quite performant and not saddled by limitations of vectorization. I do like the terseness of vectorized expressions over loops, this is being filled by devectorize.jl. Yes there are more libraries available in R or Scipy but given the ease with which one can code in Julia I dont see this to be an unsurmountable problem. Every language has to begin somewhere and unlike say other competing solutions like Torch7 I find the community very friendly, responsive and pragmatic. It seems they spend conscious effort to keep it that way. So, Julia community, here is wishing all the best. I do love Python a lot, and I mean really really a whole lot (except for its OOP parts) but this self cheering gets a little too much at times.
- andreasvc 12y ago
  Your points (a) and (b) are specific to numerical code; perhaps Numba addresses these problems well but it's not something I've worked with. With regards to C and Cython, I wouldn't say "the whole point was to avoid writing low level code", because that is what can give you the most performance in the end. If you do want to avoid low level code then you'll prefer to approach with a single language such as Julia or Java, but this means trading the bare-to-the-metal performance for having a VM and JIT between the code and the machine. The argument of preferring writing C code to Cython code is moot because you have the choice of writing code in Cython or using it to wrap existing C code. While the second option allows you to write in C as you say you'd prefer, the first options offers seamless integration of C code and manipulation of Python datastructures, so there is an advantage to using Cython not just for wrappers but also for code. I'm a bit at a loss as to why you would claim I gave a rosy picture, or what this "self cheering" is about (note that my use of "ideal" was part of a conditional). I actually presented it as a dilemma (systems & scripting language, or a single language), and I think it's not clear which approach is better or whether the fact that there seems to be a dilemma is accidental or necessary.
  - srean 12y ago
    The 'cheering' part was not directed at you, I should have made that clearer and indeed my comments are specific to numeric code. You mention Java, it so happened that I commented why Java is not a good solution for numerics just yesterday https://news.ycombinator.com/item?id=8214922 https://news.ycombinator.com/item?id=8214922 However, if you follow that thread you would see that it is not clear yet whether writing C is what would give you the most performant and correct code. > The argument of preferring writing C code to Cython code is moot because you have the choice of writing code in Cython or using it to wrap existing C code. I am not convinced about this and I have mentioned why I am not so enamored by this approach. Although quite a feat in itself Cython is not a very sophisticated compiler, so if you are writing C code you are better of taking charge and writing the C yourself. You get to enjoy the tooling that have accumulated over the years around the C syntax language. Otherwise you often get yourself in a solution that you have to debug the autogenerated C, not very pleasant. The second reason is that the bridge between Cython and C is by no means cheap. Note Cython's objective is to produce a Python C modules, not native C binaries, so it will talk to see with all the disadvantages of talking to C from a Python C module. Cython is indeed great if you have legacy code in Python that you would want to marry to C, or to get some speedup with minimal intervention, but if you are not so constrained and want more speedup than this affords perhaps better approaches are needed. I would also point out that Julia does not always yield faster solutions than Cython yet. My main motivation was to point out some design issues that you are saddled with when you are a denizen in the Python world.
ced 12y ago
That was a really nice read (it's a quick history of programming languages through a particular lens, with a glowing endorsement of Julia at the end). I couldn't find the link to the second post (which is the one about Julia), it's this: http://graydon2.dreamwidth.org/189377.html http://graydon2.dreamwidth.org/189377.html

marcodena 12y ago

nice post!

lottin 12y ago

Do one thing and do it well, seems like a good idea. Of course, do everything better than anyone else is even better, but is it really possible? I'm not sold on the idea of a one-size-fits-all programming language.

coldtea 12y ago
Well, if there was some theorical barrier, and if we were indeed talking of making the "100% perfect" language" yes. But in real life things preventing using a language in more domains are mostly mere bad design, lack of some features and legacy baggage. Case in point with Python. It's not that we want it to be faster or lower level than C to use in kernels. We just want it to be 10x, 20x faster, something entirely possible (Javascript JITs prove it can be done for even the most dynamic of languages), and we just want it to be able to tap into multicore CPUs (something that would also be entirely possible if it was slightly better designed). So it's not like there's some huge theoritical barrier to achieving this stuff. It's mostly money (like those spend by Apple, Google and Mozilla to their JIT JS engines), and the will to drop some backwards compatibility (which they did with 3.0 but for marginal benefits instead of addressing useful stuff).

srean 12y ago

It is light years far from ideal. There are two variants of this much parroted line (i) Numpy, Scipy are just C loops, and (ii) 'just do the intensive parts in C', both leave a lot more to be desired. Yes numpy, scipy indeed dispatch to precompiled C and sometimes Fortran loops but the problem lies elsewhere, in its vectorization paradigm. It is just extremely wasteful. There are two problems: (a) it is not expressive enough to capture efficient computation without generating unnecessary intermediate arrays whose sole objective is to make it possible to write the computation as a vectorized operation. Unlike Matlab in the past, numpy, scipy are at least smart about broadcasting. This often allows one to avoid constructing those intermediates in memory. However, this comes at an extra indirection that affects all array operations via the stride vector. You pay the cost of indirection whether you need it or not. (b) The second problem is generation of temporaries when you chain several binary operations. These temporaries get allocated, filled and destroyed over and over again within a single expression which itself might be in a loop. This costs computation, memory, not to mention GC pressure. There is of course numexpr but it is also quite limiting. For instance you cannot index or slice from within a numexpr expression. It offers limited set of reduce operations, in an expression you can use only one, and it must be the last one in the sequence of operations. Then there is 'write to C'. If we have not eliminated the need to write C we have not really solved the hard problems have we. I think the whole point was to avoid writing low level code because it is error prone, tedious and that it often comes at the cost of productivity. The drop down to C imposes an unnecessary break in flow and forces you to tackle the impedance mismatch. Tools like Cython eases this a bit. You cannot for example use numpy array expressions efficiently from Cython, you have to write those tedious low level indexing code. If I were to write C, I would rather write it in C syntax and take advantage of the decades of tooling around C syntax. Cython is great and an awesome community effort, but it still quite a simple compiler and has limitations. So far I have been talking only about ease of use, quality of programming experience etc etc, but that is not the only issue here. The problem is calls to C and more importantly callbacks from C to Python are expensive enough to be non-ignorable. If you have a hot loop where you go back and forth between C/Fortran world and Python thats going to incur a serious hit. The solution is to make the containing piece of Code into C/Fortran/Cython, so it ends up swallowing more and more of the application logic, leaving only but a shell of I/O in the Python world. Its not the end of the world but not quite the rosy picture you give. Another issue is cultural, its common among many newer programmers not to have really experienced fast runtimes, of course this is a generalization and does not apply to all, but have seen it happen frequently enough. They are greatly amazed by what I would call only modest improvement in runtimes and they would be cheering "Wow! so speed much fast!" etc. All of these make me be really hopeful about Julia. Interacting with the community gives me the feeling that they get it. Julia is an expressive language, already quite performant and not saddled by limitations of vectorization. I do like the terseness of vectorized expressions over loops, this is being filled by devectorize.jl. Yes there are more libraries available in R or Scipy but given the ease with which one can code in Julia I dont see this to be an unsurmountable problem. Every language has to begin somewhere and unlike say other competing solutions like Torch7 I find the community very friendly, responsive and pragmatic. It seems they spend conscious effort to keep it that way. So, Julia community, here is wishing all the best. I do love Python a lot, and I mean really really a whole lot (except for its OOP parts) but this self cheering gets a little too much at times.

andreasvc 12y ago
Your points (a) and (b) are specific to numerical code; perhaps Numba addresses these problems well but it's not something I've worked with. With regards to C and Cython, I wouldn't say "the whole point was to avoid writing low level code", because that is what can give you the most performance in the end. If you do want to avoid low level code then you'll prefer to approach with a single language such as Julia or Java, but this means trading the bare-to-the-metal performance for having a VM and JIT between the code and the machine. The argument of preferring writing C code to Cython code is moot because you have the choice of writing code in Cython or using it to wrap existing C code. While the second option allows you to write in C as you say you'd prefer, the first options offers seamless integration of C code and manipulation of Python datastructures, so there is an advantage to using Cython not just for wrappers but also for code. I'm a bit at a loss as to why you would claim I gave a rosy picture, or what this "self cheering" is about (note that my use of "ideal" was part of a conditional). I actually presented it as a dilemma (systems & scripting language, or a single language), and I think it's not clear which approach is better or whether the fact that there seems to be a dilemma is accidental or necessary.
- srean 12y ago
  The 'cheering' part was not directed at you, I should have made that clearer and indeed my comments are specific to numeric code. You mention Java, it so happened that I commented why Java is not a good solution for numerics just yesterday https://news.ycombinator.com/item?id=8214922 https://news.ycombinator.com/item?id=8214922 However, if you follow that thread you would see that it is not clear yet whether writing C is what would give you the most performant and correct code. > The argument of preferring writing C code to Cython code is moot because you have the choice of writing code in Cython or using it to wrap existing C code. I am not convinced about this and I have mentioned why I am not so enamored by this approach. Although quite a feat in itself Cython is not a very sophisticated compiler, so if you are writing C code you are better of taking charge and writing the C yourself. You get to enjoy the tooling that have accumulated over the years around the C syntax language. Otherwise you often get yourself in a solution that you have to debug the autogenerated C, not very pleasant. The second reason is that the bridge between Cython and C is by no means cheap. Note Cython's objective is to produce a Python C modules, not native C binaries, so it will talk to see with all the disadvantages of talking to C from a Python C module. Cython is indeed great if you have legacy code in Python that you would want to marry to C, or to get some speedup with minimal intervention, but if you are not so constrained and want more speedup than this affords perhaps better approaches are needed. I would also point out that Julia does not always yield faster solutions than Cython yet. My main motivation was to point out some design issues that you are saddled with when you are a denizen in the Python world.

ced 12y ago

That was a really nice read (it's a quick history of programming languages through a particular lens, with a glowing endorsement of Julia at the end). I couldn't find the link to the second post (which is the one about Julia), it's this: http://graydon2.dreamwidth.org/189377.html http://graydon2.dreamwidth.org/189377.html