5 ms·

In the preface you list: - Generate high-performance code - Specialize on multiple arguments - Evaluate code interactively - Exploit parallel hardware > Th

In the preface you list: - Generate high-performance code - Specialize on multiple arguments - Evaluate code interactively - Exploit parallel hardware > This list of requirements eliminates Python, R and other mainstream languages used for data science. Can you elaborate on why/how? Awesome work by the way

wodenokoto 3y ago
Python and R do not generate high performing code. At best they generate calls to high performing code.
- trostaft 3y ago
  > At best they generate calls to high performing code. It should be noted that this is usually sufficient. But particularly for earth scale problems it can often not be.
- th0ma5 3y ago
  Julia is designed to seem to win arguments as best I can tell... If you complain about the need to break abstractions and the lack of general purpose application you're accused of not understanding. When you say it slow they say you can inline assembler, and when you say that's dumb why have a high level language then, they then say well you don't have to it is fast as is and everyone else is slow, and it just devolves into circular arguments. Abstractions exist in layers for reasons.
  - sainez 3y ago
    You can obviously provide the same abstraction with different implementations that yield different performance characteristics. Julia provides the same level of flexibility (if not more) as Python without any of the design decisions which cause Python to be so slow. I fail to see how this is a contentious point.
    - th0ma5 3y ago
      Yeah this is a great example of what I'm saying. You obviously don't understand Python then.
  - adgjlsfhk1 3y ago
    when you say Julia is slow, what are you talking about? even without any fancy tricks, normal Julia code is usually the same speed as the equivalent normal C code
    - th0ma5 3y ago
      Yes this is another great example... If Julia is fast then why do you need to inline assembler ?
      - adgjlsfhk1 3y ago
        For the same reason that C/C++ allow inline assembly? Languages come in roughly 3 speeds. Slow (e.g. python/R), mostly not slow (e.g. Java/Go), and not slow (e.g. C/Rust). If you want actually fast code (e.g. the speed of BLAS/FFTW etc) you need the combination of a not slow language, code generation, and often hand-coded assembly for the most performance critical parts.
        th0ma5 3y ago
        I noticed you didn't mention Julia explicitly this time because when you outline the abstractions like this it seems silly to claim something about Julia magically solves the position and purpose of these layers. I can write a PySpark job based on a tutorial that would run circles around a single core Julia process that was designed with contradictory requirements. I just don't see how Julia gets away with claiming it solves all of this in the first page of their documentation without a ton of qualifiers... Except to say that is Julia that's what they do, they make bold claims that obfuscate what performance is and where it comes from.
        adgjlsfhk1 3y ago
        To be explicit about where julia fits in here, Julia is a "not slow" language (you could make an argument for it being on the faster end of "mostly not slow" due to GC) that also has enough high level features (higher order functions, macros, memory management, general ease of use) to work as a high level language. You absolutely can write a distributed python codebase that runs faster than single core julia, but doing so will likely be harder than writing the distributed/threaded Julia code that is way faster than PySpark.
- BrandonS113 3y ago
  I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pretty good at such vector calculations.
  - n4r9 3y ago
    Is this what you mean by nlopt? https://github.com/stevengj/nlopt https://github.com/stevengj/nlopt If so, it looks like you're interfacing from R to high-performing code written in C. Isn't that exactly what OP was describing?
    - BrandonS113 3y ago
      no, the function it calls is pure R and that is where the the code spends all its time.
      - n4r9 3y ago
        Interesting. Have you published the code and/or benchmarks anywhere? This flies against everything I've read about Julia and R.
        BrandonS113 3y ago
        No, its 4 lines of code. I just benchmarked for myself. All it is for 2 vectors x and y of average length 50, and 5 parameters, with exponential, addition, and multiplication, and ultimately sum to return to the optimizer. I was also surprised as I expected c to be much faster. And with Rccp, its actually slower than the R, overhead I guess. When I looked into it, apparently R has really fast code for such vector calculations. With julia, admittingly I did not use simd which would likely make it faster. Now, I generally use Julia for heavy computes, and usually its much faster than R. But not always. And this little bit of code runs for hours on the largest instance on AWS every day. Why I was looking so speed it up.
ekianjo 3y ago
R can exploit parallel hardware just fine with Parallel, Future and other libraries like Mirai. The problem is that execution speed is going to be a bottleneck for anything large and when you reach some optimizations, maybe R is not the best language to do the job. But it depends a lot on the use case.
- BrandonS113 3y ago
  I much prefer parallel in R with mclapply() to the Julia implementation of parallel. One of the few areas where I prefer R to julia (other being R data.tables to julia dataframes)

wodenokoto 3y ago

Python and R do not generate high performing code. At best they generate calls to high performing code.

trostaft 3y ago
> At best they generate calls to high performing code. It should be noted that this is usually sufficient. But particularly for earth scale problems it can often not be.
th0ma5 3y ago
Julia is designed to seem to win arguments as best I can tell... If you complain about the need to break abstractions and the lack of general purpose application you're accused of not understanding. When you say it slow they say you can inline assembler, and when you say that's dumb why have a high level language then, they then say well you don't have to it is fast as is and everyone else is slow, and it just devolves into circular arguments. Abstractions exist in layers for reasons.
- sainez 3y ago
  You can obviously provide the same abstraction with different implementations that yield different performance characteristics. Julia provides the same level of flexibility (if not more) as Python without any of the design decisions which cause Python to be so slow. I fail to see how this is a contentious point.
  - th0ma5 3y ago
    Yeah this is a great example of what I'm saying. You obviously don't understand Python then.
- adgjlsfhk1 3y ago
  when you say Julia is slow, what are you talking about? even without any fancy tricks, normal Julia code is usually the same speed as the equivalent normal C code
  - th0ma5 3y ago
    Yes this is another great example... If Julia is fast then why do you need to inline assembler ?
    - adgjlsfhk1 3y ago
      For the same reason that C/C++ allow inline assembly? Languages come in roughly 3 speeds. Slow (e.g. python/R), mostly not slow (e.g. Java/Go), and not slow (e.g. C/Rust). If you want actually fast code (e.g. the speed of BLAS/FFTW etc) you need the combination of a not slow language, code generation, and often hand-coded assembly for the most performance critical parts.
      - th0ma5 3y ago
        I noticed you didn't mention Julia explicitly this time because when you outline the abstractions like this it seems silly to claim something about Julia magically solves the position and purpose of these layers. I can write a PySpark job based on a tutorial that would run circles around a single core Julia process that was designed with contradictory requirements. I just don't see how Julia gets away with claiming it solves all of this in the first page of their documentation without a ton of qualifiers... Except to say that is Julia that's what they do, they make bold claims that obfuscate what performance is and where it comes from.
        adgjlsfhk1 3y ago
        To be explicit about where julia fits in here, Julia is a "not slow" language (you could make an argument for it being on the faster end of "mostly not slow" due to GC) that also has enough high level features (higher order functions, macros, memory management, general ease of use) to work as a high level language. You absolutely can write a distributed python codebase that runs faster than single core julia, but doing so will likely be harder than writing the distributed/threaded Julia code that is way faster than PySpark.
BrandonS113 3y ago
I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pretty good at such vector calculations.
- n4r9 3y ago
  Is this what you mean by nlopt? https://github.com/stevengj/nlopt https://github.com/stevengj/nlopt If so, it looks like you're interfacing from R to high-performing code written in C. Isn't that exactly what OP was describing?
  - BrandonS113 3y ago
    no, the function it calls is pure R and that is where the the code spends all its time.
    - n4r9 3y ago
      Interesting. Have you published the code and/or benchmarks anywhere? This flies against everything I've read about Julia and R.
      - BrandonS113 3y ago
        No, its 4 lines of code. I just benchmarked for myself. All it is for 2 vectors x and y of average length 50, and 5 parameters, with exponential, addition, and multiplication, and ultimately sum to return to the optimizer. I was also surprised as I expected c to be much faster. And with Rccp, its actually slower than the R, overhead I guess. When I looked into it, apparently R has really fast code for such vector calculations. With julia, admittingly I did not use simd which would likely make it faster. Now, I generally use Julia for heavy computes, and usually its much faster than R. But not always. And this little bit of code runs for hours on the largest instance on AWS every day. Why I was looking so speed it up.

ekianjo 3y ago

R can exploit parallel hardware just fine with Parallel, Future and other libraries like Mirai. The problem is that execution speed is going to be a bottleneck for anything large and when you reach some optimizations, maybe R is not the best language to do the job. But it depends a lot on the use case.

BrandonS113 3y ago
I much prefer parallel in R with mclapply() to the Julia implementation of parallel. One of the few areas where I prefer R to julia (other being R data.tables to julia dataframes)