9 ms·

I see your comment grayed out, and I just want to chime in, as some who does a lot of numerical stuff (more than a decade, published stuff, support multiple lab

I see your comment grayed out, and I just want to chime in, as some who does a lot of numerical stuff (more than a decade, published stuff, support multiple lab research projects etc), I want to second this point of view. When it’s time to get real work done Python is more than good enough, and there’s plenty of strategies for acceleration where required. And when I want Julia’s promise of fast loops, I use Numba. If all the effort gone into Julia had instead been spent on fixing remaining warts in Python workflow for science, we wouldn’t even havee this conversation.

celrod 8y ago
I think it is easier to write fast, complicated code in Julia than in C, C++, or Fortran. Not just the syntax, but because of great support for generic code by default and metaprogramming making it relatively simple to write code to generate the code you actually want for any given scenario. Interactive benchmarking and profiling are a boon too. An example of the value of generic code is that forward mode AD is extremely easy, and almost always just works on whatever code you run it on. Then, once that's done, multiple dispatch (and possible macros for a DSL) allows for a much cleaner user interface than Python offers for numeric code. I have a lot more experience with R than Python, but seeing more of the scientist/mathematician/researchers side of things, I have to strongly disagree with the view that they should write slow code and contact a CS guy to write fast code in another language when they need it. Do you honestly think that's practical for grad student's projects? Recently, one of my friends wrote a simulation in R. Most of the work was done by hcubature -- a package written in C -- integrating a function written in R. Could just have easily been written in Python. That function was slow, and the simulation ran for days. Before an error caused it to crash, losing days of compute time. I -- a statistics grad student -- helped him rewrite it in Julia, and it finished in 2 hours. That C/C++ code will still run slowly if they have to call your R/Python code is a problem. They also can't apply things like AD easily. A common solution, used by Stan for example, is to create a whole new modeling language and have users interface through that. Learning a new language -- albeit relatively simple/domain specific -- which they then cannot debug interactively, is another pain point. All this can be avoided by simply using Julia.
- tavert 8y ago
  The reason I ran away from Julia and don't plan on ever using it again, and don't recommend anyone use it outside of academia, is that so much of the community is made up of grad students. So you get a lot of research code and people who have never been professional programmers maintaining most of the ecosystem. Julia Computing is largely made up of people they've hired from the community straight out of grad school.
  - celrod 8y ago
    I hope that if/as Julia gets adopted in industry, more libraries get written and maintained by professionals. If the language is successful, that may change. Although AFAIK it hasn't really in the case of R, outside of tidyverse. As a grad student without a CS background, I don't think I'm qualified to say much more on this.
    - baldfat 8y ago
      The issue isn't "Professionals" it is domain specialist. In the past data specilist didn't have Computer Science specialty. They were awesome in stats and numbers but lacked strong programming skills. Hadley Wickham is special because he has both the stats, data science AND programming skills. data.table is also an amazing library. R to me is the most improved language in the history of programming languages over the past 5 years. Also R allows anyone with basic hackery R skills to create libraries easily and that is why so many of them are not optimal.
  - Accipitriform 8y ago
    > Julia Computing is largely made up of people they've hired from the community straight out of grad school. Where do you think most companies get "professional programmers" from, exactly? Julia's been designed and implemented by some very bright people, and it shows.
    - jhayward 8y ago
      Industry gets professional programmers by hiring people who have been hammering out shipping code in paying products for years, and years, doing support, maintenance, and new product development and research. Grad students may be brilliant but that does not help give them any insight in to what makes a good ecosystem, toolchain, and feature set good.
chappi42 8y ago
> And when I want Julia’s promise of fast loops, I use Numba. This only works (easily) as long as you don't have user-defined types > If all the effort gone into Julia had instead been spent on fixing remaining warts in Python workflow for science, we wouldn’t even havee this conversation. Python is too dynamic, you cannot just fix remaining warts. From Julia documentation I know that Julia language has been designed for speed and e.g. some dynamic possibilities have been omitted in order to be able to generate fast code. For a general idea about Julia speed see e.g. this 7 hours old excellent Juliacon video: https://www.youtube.com/watch?v=XWIZ_dCO6X8 https://www.youtube.com/watch?v=XWIZ_dCO6X8 Python certainly works. But already for syntax alone, if you have written Julia, it's hard to - in my case - go back to R.
- marmaduke 8y ago
  Numba has user defined types, https://numba.pydata.org/numba-doc/dev/user/jitclass.html https://numba.pydata.org/numba-doc/dev/user/jitclass.html No one intends to fix Python but it’s straightforward to do things like Numba: use a decorator to read out the AST for a function, reimplement it however you like and pass back the compiled function, and document the semantics.
  - chappi42 8y ago
    thank you
  - ChrisRackauckas 8y ago
    But you have to define the types for them to work. That's exactly the problem and that's what Julia has already solved.
    - marmaduke 8y ago
      What are user defined types that don’t require being defined?
      - ChrisRackauckas 8y ago
Accipitriform 8y ago
> And when I want Julia’s promise of fast loops, I use Numba. If all the effort gone into Julia had instead been spent on fixing remaining warts in Python workflow for science, we wouldn’t even havee this conversation. Python is rather a mess. Code written in Python can't be sped up without pain/cost, and apparently it will never support concurrency natively. It also suffers from the bane of weakly typed languages, errors at run time instead of compile time. I think the sweet spot for a language with most of Python's benefits that fixes many of its glaring warts is enormous.
- marmaduke 8y ago
  Python supports thread and process based concurrency. Most optimizations require effort, or they wouldn’t be called optimizations. It’s not weakly types either: you can’t add a list to a string. It’s dynamically typed.
jernfrost 8y ago
The point is that Python presents a very complex environment where you have to deal with different languages and technologies. For package developers it is a lot eaiser to use Julia. Saying one should focus more on Python and we would not have these problems is missing the point. Enormous resources by countless companies has been poured in to solve the performance problems of Python. It is almost impossible to do due to the language design of Python. You cannot fix it without breaking the language. Julia in contrast required minimum effort and resources to get fast. It is almost a toy project compare to Python. It is all down to clever language design which allowed them to use a rather dumb and simple compiler while letting LLVM do most of the heavy lifting. Every advance of Python is going to require 10x the effort of advancing Julia. It will just be a question of time before Julia catches up. Python has a huge lead so that will still probably take years but it will happen.
- marmaduke 8y ago
  I’m not missing the point: I’ve tried doing the same thing in Julia before as I do in Python, and it’s not a 10x difference. No language (relevant to this discussion) is ever going to cut down on lines of code required to do error checking, code gen, plotting etc. Sure, Julia is great as a high-level interface to LLVM, but so is Numba. Python’s design leaves something to be desired performance-wise for those coming from JVM or native languages, but it’s a trade off, not an obvious win (for Julia), and the problem goes away as programmers get wise to performance strategies in Python.
  - ffriend 8y ago
    I have quite limited experience with Cython and tried Numba just a couple of times, but I'm curious how much would it take to rewrite one of my Julia libraries to them. The library is for reverse-mode automatic differentiation, but let's put AD itself aside and talk about code generation. As an input to code generator, I have a computational graph (or "tape") - a list of functions connecting input and intermediate variables. As an output I want a compiled function for CPU/GPU. (Note: Theano used to do exactly this, but that's a separate huge system not relevant no Numba or Cython). In Julia I follow the following steps: 1. Convert all operations on the tape to expressions (~approx 1 line per operation type). 2. Eliminate common subexpressions. 3. Fuse broadcasting, e.g. rewrite: c .= a .* b e .= c .+ d into e .= a .* b .+ d Dot near operations means that they are applied elementwise without creating intermediate arrays. On CPU, Julia compiler / LLVM then generates code that reads and writes to memory exactly once (unlike e.g. what you would get with several separate operations on numpy arrays). On GPU, CUDAnative generates a single CUDA kernel which on my tests is ~1.5 times faster then several separate kernels. Note that `.=` also means that the result of operation is directly written to a (buffered) destination, so it no memory is allocated in the hot loop. 4. Rewrite everything I can into in-place operations. Notably, matrix multiplication `A * B` is replaced with BLAS/CUBLAS alternative. 5. Add to the expression function header, buffers and JIT-compile the result. In Python, I imagine using `ast` module for code parsing and transformations like common expression elimination (how hard it would be?). Perhaps, Numba can be used to compile Python code to fast CPU and GPU code, but does it fit with AST? Also, do Numba or Cython do optimizations like broadcasting and kernel fusion? I'd love to see side-by-side comparison of capabilities in such a scenario!

celrod 8y ago

I think it is easier to write fast, complicated code in Julia than in C, C++, or Fortran. Not just the syntax, but because of great support for generic code by default and metaprogramming making it relatively simple to write code to generate the code you actually want for any given scenario. Interactive benchmarking and profiling are a boon too. An example of the value of generic code is that forward mode AD is extremely easy, and almost always just works on whatever code you run it on. Then, once that's done, multiple dispatch (and possible macros for a DSL) allows for a much cleaner user interface than Python offers for numeric code. I have a lot more experience with R than Python, but seeing more of the scientist/mathematician/researchers side of things, I have to strongly disagree with the view that they should write slow code and contact a CS guy to write fast code in another language when they need it. Do you honestly think that's practical for grad student's projects? Recently, one of my friends wrote a simulation in R. Most of the work was done by hcubature -- a package written in C -- integrating a function written in R. Could just have easily been written in Python. That function was slow, and the simulation ran for days. Before an error caused it to crash, losing days of compute time. I -- a statistics grad student -- helped him rewrite it in Julia, and it finished in 2 hours. That C/C++ code will still run slowly if they have to call your R/Python code is a problem. They also can't apply things like AD easily. A common solution, used by Stan for example, is to create a whole new modeling language and have users interface through that. Learning a new language -- albeit relatively simple/domain specific -- which they then cannot debug interactively, is another pain point. All this can be avoided by simply using Julia.

tavert 8y ago
The reason I ran away from Julia and don't plan on ever using it again, and don't recommend anyone use it outside of academia, is that so much of the community is made up of grad students. So you get a lot of research code and people who have never been professional programmers maintaining most of the ecosystem. Julia Computing is largely made up of people they've hired from the community straight out of grad school.
- celrod 8y ago
  I hope that if/as Julia gets adopted in industry, more libraries get written and maintained by professionals. If the language is successful, that may change. Although AFAIK it hasn't really in the case of R, outside of tidyverse. As a grad student without a CS background, I don't think I'm qualified to say much more on this.
  - baldfat 8y ago
    The issue isn't "Professionals" it is domain specialist. In the past data specilist didn't have Computer Science specialty. They were awesome in stats and numbers but lacked strong programming skills. Hadley Wickham is special because he has both the stats, data science AND programming skills. data.table is also an amazing library. R to me is the most improved language in the history of programming languages over the past 5 years. Also R allows anyone with basic hackery R skills to create libraries easily and that is why so many of them are not optimal.
- Accipitriform 8y ago
  > Julia Computing is largely made up of people they've hired from the community straight out of grad school. Where do you think most companies get "professional programmers" from, exactly? Julia's been designed and implemented by some very bright people, and it shows.
  - jhayward 8y ago
    Industry gets professional programmers by hiring people who have been hammering out shipping code in paying products for years, and years, doing support, maintenance, and new product development and research. Grad students may be brilliant but that does not help give them any insight in to what makes a good ecosystem, toolchain, and feature set good.

chappi42 8y ago

> And when I want Julia’s promise of fast loops, I use Numba. This only works (easily) as long as you don't have user-defined types > If all the effort gone into Julia had instead been spent on fixing remaining warts in Python workflow for science, we wouldn’t even havee this conversation. Python is too dynamic, you cannot just fix remaining warts. From Julia documentation I know that Julia language has been designed for speed and e.g. some dynamic possibilities have been omitted in order to be able to generate fast code. For a general idea about Julia speed see e.g. this 7 hours old excellent Juliacon video: https://www.youtube.com/watch?v=XWIZ_dCO6X8 https://www.youtube.com/watch?v=XWIZ_dCO6X8 Python certainly works. But already for syntax alone, if you have written Julia, it's hard to - in my case - go back to R.

marmaduke 8y ago
Numba has user defined types, https://numba.pydata.org/numba-doc/dev/user/jitclass.html https://numba.pydata.org/numba-doc/dev/user/jitclass.html No one intends to fix Python but it’s straightforward to do things like Numba: use a decorator to read out the AST for a function, reimplement it however you like and pass back the compiled function, and document the semantics.
- chappi42 8y ago
  thank you
- ChrisRackauckas 8y ago
  But you have to define the types for them to work. That's exactly the problem and that's what Julia has already solved.
  - marmaduke 8y ago
    What are user defined types that don’t require being defined?
    - ChrisRackauckas 8y ago

Accipitriform 8y ago

> And when I want Julia’s promise of fast loops, I use Numba. If all the effort gone into Julia had instead been spent on fixing remaining warts in Python workflow for science, we wouldn’t even havee this conversation. Python is rather a mess. Code written in Python can't be sped up without pain/cost, and apparently it will never support concurrency natively. It also suffers from the bane of weakly typed languages, errors at run time instead of compile time. I think the sweet spot for a language with most of Python's benefits that fixes many of its glaring warts is enormous.

marmaduke 8y ago
Python supports thread and process based concurrency. Most optimizations require effort, or they wouldn’t be called optimizations. It’s not weakly types either: you can’t add a list to a string. It’s dynamically typed.

jernfrost 8y ago

The point is that Python presents a very complex environment where you have to deal with different languages and technologies. For package developers it is a lot eaiser to use Julia. Saying one should focus more on Python and we would not have these problems is missing the point. Enormous resources by countless companies has been poured in to solve the performance problems of Python. It is almost impossible to do due to the language design of Python. You cannot fix it without breaking the language. Julia in contrast required minimum effort and resources to get fast. It is almost a toy project compare to Python. It is all down to clever language design which allowed them to use a rather dumb and simple compiler while letting LLVM do most of the heavy lifting. Every advance of Python is going to require 10x the effort of advancing Julia. It will just be a question of time before Julia catches up. Python has a huge lead so that will still probably take years but it will happen.

marmaduke 8y ago
I’m not missing the point: I’ve tried doing the same thing in Julia before as I do in Python, and it’s not a 10x difference. No language (relevant to this discussion) is ever going to cut down on lines of code required to do error checking, code gen, plotting etc. Sure, Julia is great as a high-level interface to LLVM, but so is Numba. Python’s design leaves something to be desired performance-wise for those coming from JVM or native languages, but it’s a trade off, not an obvious win (for Julia), and the problem goes away as programmers get wise to performance strategies in Python.
- ffriend 8y ago
  I have quite limited experience with Cython and tried Numba just a couple of times, but I'm curious how much would it take to rewrite one of my Julia libraries to them. The library is for reverse-mode automatic differentiation, but let's put AD itself aside and talk about code generation. As an input to code generator, I have a computational graph (or "tape") - a list of functions connecting input and intermediate variables. As an output I want a compiled function for CPU/GPU. (Note: Theano used to do exactly this, but that's a separate huge system not relevant no Numba or Cython). In Julia I follow the following steps: 1. Convert all operations on the tape to expressions (~approx 1 line per operation type). 2. Eliminate common subexpressions. 3. Fuse broadcasting, e.g. rewrite: c .= a .* b e .= c .+ d into e .= a .* b .+ d Dot near operations means that they are applied elementwise without creating intermediate arrays. On CPU, Julia compiler / LLVM then generates code that reads and writes to memory exactly once (unlike e.g. what you would get with several separate operations on numpy arrays). On GPU, CUDAnative generates a single CUDA kernel which on my tests is ~1.5 times faster then several separate kernels. Note that `.=` also means that the result of operation is directly written to a (buffered) destination, so it no memory is allocated in the hot loop. 4. Rewrite everything I can into in-place operations. Notably, matrix multiplication `A * B` is replaced with BLAS/CUBLAS alternative. 5. Add to the expression function header, buffers and JIT-compile the result. In Python, I imagine using `ast` module for code parsing and transformations like common expression elimination (how hard it would be?). Perhaps, Numba can be used to compile Python code to fast CPU and GPU code, but does it fit with AST? Also, do Numba or Cython do optimizations like broadcasting and kernel fusion? I'd love to see side-by-side comparison of capabilities in such a scenario!