4 ms·

For me it's really interesting to read all the complaints about precompilation times etc. I'm definitely not saying the complaints are false, just that there ar

For me it's really interesting to read all the complaints about precompilation times etc. I'm definitely not saying the complaints are false, just that there are also some of us out there where they matter close to 0 (at current level). I use Julia from the REPL, running mostly self constructed algorithms. Only fairly basic modules are used. My main project is a backtesting framework for algorithmic stock trading. A typical run took more than a month of runtime on my computer when the code was in optimized Numpy and now takes less than 1 minute in optimized (but likely far from optimal) Julia code. The only way to come at all close to the speed of Julia with Python would probably be something like Numba, which I figured would be more immature than Julia. Python is only usable for heavy computing on large data when the glue can be in the outermost loop so to speak.

arthurcolle 6y ago
What kind of architecture are you using for your backtesting framework? I have been collecting data to attempt to build something similar to that with close to around 2.3TB of tick data stretching around 20 years across 75 major pairs (indices, fx, major commods, etc) and I am always overwhelmed about design of systems to process large datasets like this.
- Reelin 6y ago
  I realize it's a bit off topic, but I just have to ask - where did you get so much ticker data?
  - arthurcolle 6y ago
    It's been a work in progress for quite a while. Involved writing many different scrapers and other data ingestion systems that systematically downloaded everything I could get my hands on. I'm building an API to eventually sell some of the information, so I'd prefer not to say specific sources.
- kruxigt 6y ago
  I'm not sure what you mean with architecture? The computers I currently run on are ordinary Macs of the latest generations. (Before Julia came and saved me I ran on Google cloud) That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?
  - arthurcolle 6y ago
    I'd love to. I'll contact you soon.
pirocks 6y ago
I find the month to minute transformation rather shocking. Are you sure you weren't doing something algorithmicly wrong in python? Did you profile it? Like I would never leave something running for a month without putting a fair bit of time into making sure it was fast.
- kruxigt 6y ago
  I don't think I did anything algorithmically wrong. If I could have done it with more of the data handling being done in c functions and just using Python as outer glue? Most likely but then that would not make the framework very extensible to other models.
- kruxigt 6y ago
  Not only did I profile it, put a lot of time into optimization and learned how to use cloud computing, but I also learned a new language (Julia) and ported the whole application. :)
  - kruxigt 6y ago
    In Julia I can use things as data oriented design (few cache misses) to further reduce runtime. I'm not sure you get much out of such efforts in Python.
    - snicker7 6y ago
      This. Having tight control over memory layout + heap allocations + vector instructions can literally decrease runtume by several orders of magnitude. Performance is so much more than having the "right algorithm". The low-level details matter so much.
xiphias2 6y ago
Time series analysis especially goes well with Julia, as the number of dependencies is in the same order of magnitude as the number of data points. Numpy / torch shines when the slowest part of the data processing is big matrix multiplication. In that case actually waiting for Julia compilation can slow down experimental iterations.
- FridgeSeal 6y ago
  That said, the Julia community has done some pretty amazing work in that subfield, last I checked they were getting performance in pure-Julia very close to the of BLAS/LAPACK, which is pretty incredible.
  - eigenspace 6y ago
    https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...
  - xiphias2 6y ago
    Julia is improving, but I wouldn't compare it to BLAS/LAPACK, cuBLAS is a much better comparision. PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation. Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly. Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year. We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away. I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).

arthurcolle 6y ago

What kind of architecture are you using for your backtesting framework? I have been collecting data to attempt to build something similar to that with close to around 2.3TB of tick data stretching around 20 years across 75 major pairs (indices, fx, major commods, etc) and I am always overwhelmed about design of systems to process large datasets like this.

Reelin 6y ago
I realize it's a bit off topic, but I just have to ask - where did you get so much ticker data?
- arthurcolle 6y ago
  It's been a work in progress for quite a while. Involved writing many different scrapers and other data ingestion systems that systematically downloaded everything I could get my hands on. I'm building an API to eventually sell some of the information, so I'd prefer not to say specific sources.
kruxigt 6y ago
I'm not sure what you mean with architecture? The computers I currently run on are ordinary Macs of the latest generations. (Before Julia came and saved me I ran on Google cloud) That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?
- arthurcolle 6y ago
  I'd love to. I'll contact you soon.

pirocks 6y ago

I find the month to minute transformation rather shocking. Are you sure you weren't doing something algorithmicly wrong in python? Did you profile it? Like I would never leave something running for a month without putting a fair bit of time into making sure it was fast.

kruxigt 6y ago
I don't think I did anything algorithmically wrong. If I could have done it with more of the data handling being done in c functions and just using Python as outer glue? Most likely but then that would not make the framework very extensible to other models.
kruxigt 6y ago
Not only did I profile it, put a lot of time into optimization and learned how to use cloud computing, but I also learned a new language (Julia) and ported the whole application. :)
- kruxigt 6y ago
  In Julia I can use things as data oriented design (few cache misses) to further reduce runtime. I'm not sure you get much out of such efforts in Python.
  - snicker7 6y ago
    This. Having tight control over memory layout + heap allocations + vector instructions can literally decrease runtume by several orders of magnitude. Performance is so much more than having the "right algorithm". The low-level details matter so much.

xiphias2 6y ago

Time series analysis especially goes well with Julia, as the number of dependencies is in the same order of magnitude as the number of data points. Numpy / torch shines when the slowest part of the data processing is big matrix multiplication. In that case actually waiting for Julia compilation can slow down experimental iterations.

FridgeSeal 6y ago
That said, the Julia community has done some pretty amazing work in that subfield, last I checked they were getting performance in pure-Julia very close to the of BLAS/LAPACK, which is pretty incredible.
- eigenspace 6y ago
  https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...
- xiphias2 6y ago
  Julia is improving, but I wouldn't compare it to BLAS/LAPACK, cuBLAS is a much better comparision. PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation. Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly. Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year. We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away. I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).