5 ms·

Julia is a very interesting language but big data processing is not a matter of the programming language itself, but how your platform and your architecture is

Julia is a very interesting language but big data processing is not a matter of the programming language itself, but how your platform and your architecture is put together. And nearly always your bottleneck is not the language itself..

staticassertion 10y ago
I think that if this were true, the data science libraries in Python wouldn't all be written in C under the hood, linking to Fortran, etc. Language can have a huge impact on performance with big data. Julia is just a small step forward. It still has performance problems where the JIT can't help much - like the DataFrame structure, which is effectively a black box and therefor hard to optimize.
- VodkaHaze 10y ago
  If you need performance on these kind of structures, you can use structured arrays in python. The main reason I like the idea of Julia is that I don't see the problem of parallelism being addressed in python, at least not without cumbersome libraries (disclaimer: I haven't tried dask yet). If python had an equivalent to #pragma omp parallel for I wouldn't have Julia ready in mind for my next project that requires creating a high performance algorithm.
- hadsed 10y ago
  Python numerical libraries links to LAPACK/BLAS/etc because that's how people used to do scientific computing before Python. Also for scientific computing you're mostly CPU bound which means that no one would use Python if the libraries were slower than Fortran or C. But big data problems are different in the sense that you're typically bound by IO, so while it certainly helps to have fast code it wouldn't be the first problem you'd want to solve.
  - urschrei 10y ago
    I would say they link to LAPACK/BLAS because there's nothing to touch the combination of speed and robustness they provide.
- quasarken 10y ago
  It depends on what kind of data you're working with and what kind of operations you perform on it. If protein folding, then that's computationally expensive and the language will matter, doing mapreduce jobs, language matters little as the network overhead is going to be your bottleneck in a distributed environment.
- johnmyleswhite 10y ago
  This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Julia, like any language, has intrinsic limitations, but this is not a good example: it's an example of how good API design for Julia differs from good API design for other systems.
  - staticassertion 10y ago
    > This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. Sorry if I implied otherwise - what I meant is that the current DF structure is hard for the JIT to optimize, not that this is a fundamental limitation of Julia. Just that JIT's are limited. I shouldn't have said Julia is only a small step forward, I think. A JIT gets you a long way. > There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Can you link to information on this? I'd love to see information on the design around this.
    - mandelken 10y ago
      >Can you link to information on this? This blog post is a good starting point on the current shortcomings and design solutions of dataframes in Julia: http://julialang.org/blog/2016/10/StructuredQueries http://julialang.org/blog/2016/10/StructuredQueries
      - staticassertion 10y ago
        Thanks a lot.
    - elcritch 10y ago
GeneralMayhem 10y ago
Depends on whether you're optimizing for latency or throughput - and of course, in some systems, that decision changes on an instant-to-instant basis. For throughput, indexing and shuffling is probably going to be the bottleneck, and that's dependent on network and filesystem I/O. But for the low-latency path, choice of language absolutely matters, so if you want to be able to share code between the two you end up needing both a fast language and fast architecture.
importantbrian 10y ago
I agree that this is generally true, but Finance is sort of a special case. In a world where firms pay the premium for colocated servers and direct data feeds in order to shave off a few milliseconds of latency there is also a strong push to use the vary fastest language possible + have the very best possible architecture, all to shave off as many extra milliseconds as possible.
- polskibus 10y ago
  When raw performance is so important you end up implementing the solution on FPGAs. Switch to Julia does not really matter in such cases.
  - importantbrian 10y ago
    True. You're pretty much stuck with verilog at that point.
- tikhonj 10y ago
  That's only true for a very niche sort of trading (only high-frequency/latency arbitrage type stuff) which makes up a tiny portion of the financial industry[1]—and that tiny portion is certainly not using Julia for production trading systems. Everyone else has more reasonable performance constraints. [1]: I can't find my source right now, but I remember seeing a credible estimate that the HFT industry in the US takes in $10 billion in revenue. That's a drop in the bucket of an industry that's measured in trillions just in the US. Intuitively this makes a lot of sense because HFT strategies make a tiny amount of money per trade and have to do a ton of trades across a ton of different securities—which means the capacity of HFT strategies is relatively small.
  - the_duke 10y ago
    HFT trading is very big. Maybe not in terms of earnings, but in volume. Estimates range from 30-70%, depending on the market. I'm digging up references now for an edit...
pitaj 10y ago
Well, interpreted languages like MATLAB and R can be annoyingly slow at times

staticassertion 10y ago

I think that if this were true, the data science libraries in Python wouldn't all be written in C under the hood, linking to Fortran, etc. Language can have a huge impact on performance with big data. Julia is just a small step forward. It still has performance problems where the JIT can't help much - like the DataFrame structure, which is effectively a black box and therefor hard to optimize.

VodkaHaze 10y ago
If you need performance on these kind of structures, you can use structured arrays in python. The main reason I like the idea of Julia is that I don't see the problem of parallelism being addressed in python, at least not without cumbersome libraries (disclaimer: I haven't tried dask yet). If python had an equivalent to #pragma omp parallel for I wouldn't have Julia ready in mind for my next project that requires creating a high performance algorithm.
hadsed 10y ago
Python numerical libraries links to LAPACK/BLAS/etc because that's how people used to do scientific computing before Python. Also for scientific computing you're mostly CPU bound which means that no one would use Python if the libraries were slower than Fortran or C. But big data problems are different in the sense that you're typically bound by IO, so while it certainly helps to have fast code it wouldn't be the first problem you'd want to solve.
- urschrei 10y ago
  I would say they link to LAPACK/BLAS because there's nothing to touch the combination of speed and robustness they provide.
quasarken 10y ago
It depends on what kind of data you're working with and what kind of operations you perform on it. If protein folding, then that's computationally expensive and the language will matter, doing mapreduce jobs, language matters little as the network overhead is going to be your bottleneck in a distributed environment.
johnmyleswhite 10y ago
This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Julia, like any language, has intrinsic limitations, but this is not a good example: it's an example of how good API design for Julia differs from good API design for other systems.
- staticassertion 10y ago
  > This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. Sorry if I implied otherwise - what I meant is that the current DF structure is hard for the JIT to optimize, not that this is a fundamental limitation of Julia. Just that JIT's are limited. I shouldn't have said Julia is only a small step forward, I think. A JIT gets you a long way. > There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Can you link to information on this? I'd love to see information on the design around this.
  - mandelken 10y ago
    >Can you link to information on this? This blog post is a good starting point on the current shortcomings and design solutions of dataframes in Julia: http://julialang.org/blog/2016/10/StructuredQueries http://julialang.org/blog/2016/10/StructuredQueries
    - staticassertion 10y ago
      Thanks a lot.
  - elcritch 10y ago

GeneralMayhem 10y ago

Depends on whether you're optimizing for latency or throughput - and of course, in some systems, that decision changes on an instant-to-instant basis. For throughput, indexing and shuffling is probably going to be the bottleneck, and that's dependent on network and filesystem I/O. But for the low-latency path, choice of language absolutely matters, so if you want to be able to share code between the two you end up needing both a fast language and fast architecture.

importantbrian 10y ago

I agree that this is generally true, but Finance is sort of a special case. In a world where firms pay the premium for colocated servers and direct data feeds in order to shave off a few milliseconds of latency there is also a strong push to use the vary fastest language possible + have the very best possible architecture, all to shave off as many extra milliseconds as possible.

polskibus 10y ago
When raw performance is so important you end up implementing the solution on FPGAs. Switch to Julia does not really matter in such cases.
- importantbrian 10y ago
  True. You're pretty much stuck with verilog at that point.
tikhonj 10y ago
That's only true for a very niche sort of trading (only high-frequency/latency arbitrage type stuff) which makes up a tiny portion of the financial industry[1]—and that tiny portion is certainly not using Julia for production trading systems. Everyone else has more reasonable performance constraints. [1]: I can't find my source right now, but I remember seeing a credible estimate that the HFT industry in the US takes in $10 billion in revenue. That's a drop in the bucket of an industry that's measured in trillions just in the US. Intuitively this makes a lot of sense because HFT strategies make a tiny amount of money per trade and have to do a ton of trades across a ton of different securities—which means the capacity of HFT strategies is relatively small.
- the_duke 10y ago
  HFT trading is very big. Maybe not in terms of earnings, but in volume. Estimates range from 30-70%, depending on the market. I'm digging up references now for an edit...

pitaj 10y ago

Well, interpreted languages like MATLAB and R can be annoyingly slow at times