8 ms·

I don't have much experience with any kind of stock market analysis/ HFT programming. I've seen people use python (or R) a lot in online tutorials and courses

by ktta 10y ago

I don't have much experience with any kind of stock market analysis/ HFT programming. I've seen people use python (or R) a lot in online tutorials and courses (there's one on coursera[1] check it out). It's understandable that python is preferred since it's an easy language to get started with, but I've never really seen C/C++ used by people online (Although I've heard about the use of FPGAs by the hardcore guys) Is the only reason C++ isn't recommended because it is difficult? The performance difference seems rather large to me. And especially if you're doing lots of number crunching, it would really benefit using C/C++ wouldn't it? Do people use them but I don't hear about them because anyone above 'tutorial level' doesn't share their code or talk about it? Or are the R and Python Computational libraries close to the performance of C/C++ that it's a viable compromise? [1]: https://www.coursera.org/learn/computational-investing https://www.coursera.org/learn/computational-investing

tnecniv 10y ago
It mostly depends on what you want to do. If you want to try to beat the speed of the internet like some of the HFT stories you hear, you're going to want to cut as much fat out as you can. If you are more interested in doing some analysis of historic data where you don't care about speed so much, it's a lot easier to play around with things in a language like R or Python.
- cgmil 10y ago
  I hear that R/Python's role in the project is initial modelling and testing. You train trading algorithms in R or Python, then deploy them in a C/C++ algorithm for speed.
  - xapata 10y ago
    NumPy is C++, Fortran, and sometimes CUDA. Depending on usage.
rampage101 10y ago
People use Python for finance and time series because of how easy it to read in the data. There is a library called pandas which allows you to read in a txt, csv, or excel file easily and do any type of array manipulation you want. The pandas library is written in C++, so it's basically a python wrapper. Conversely I don't think C++ has this same type of library for reading in data in a single line. While C++ is faster than Python overall, for most financial simulations Python is definitely fast enough.
- em500 10y ago
  Pandas was started as a skunkworks project inside a big hedge fund (AQR) by Wes McKinney (who after a few years at Cloudera is now at another hedge fund, Two Sigma). So it's no coincidence that pandas is very well adapted to financial / investment analysis.
- jzwinck 10y ago
  The Pandas library is not written in C at all actually. You can view the source here: https://github.com/pandas-dev/pandas/tree/master/pandas https://github.com/pandas-dev/pandas/tree/master/pandas It's mostly Python with a bit of Cython, and pull requests that are not pure Python are more likely to be rejected. There's basically zero C++ in Pandas itself.
cluoma 10y ago
Speaking only for R, most of the performance-critical functions included in packages these days are just wrappers to a C++ implementation. It isn't until you need to implement something custom that performance starts to be an issue. That combined with it's ease-of-use working with tabular data, and the myriad of packages available make it a very popular choice.
- cgmil 10y ago
  If Rcpp were not a thing, R might have died from being slow. Since it is a thing, though, people can easily write packages that are certainly fast enough for what they want to do, and that keeps the language useful.
dagw 10y ago
HFT is only a small part of financial data analysis. Everybody I know who works with mathematical modelling in finance use tools like SAS, Matlab, R and even Excel to build and run their models. If you're working at something like a pension fund and thinking years into the future then shaving a few milliseconds or even minutes here and there isn't just isn't a priority.
j7ake 10y ago
The main bottle neck in data analysis is not running the code but thinking about your data, visualizing it in different ways, and looking for interesting corner cases your model is not good at predicting. In that sense it is much faster to do these analyses in Python or R than in C++.
ryankennedyio 10y ago
I don't think any retailer can trade on the scale and speed that would require C/C++ over Python. With lower capital and significantly higher transaction costs, retail traders (mostly) need to have holding periods for at least hours, so speed isn't really an issue. It's more about finding longer term anomalies to deploy maybe at most a few dozen positions, as opposed to reacting to billions of quotes over any trading day. I think Python is a great tool for finance, particularly with libraries like Pandas, Numpy, etc. Most of the work in those libraries is C/Fortan/etc anyway. Great community. For more 'academic' style work/exploration, I think R might have an edge, but I personally find it more difficult to write and maintain high quality code in R. I'd like to just throw a quick plug in for an open source project I'm working on [1]. It's an event-driven trading library written in Python. We're pretty close to having live trade execution ready in an alpha state. The motivation is that you can put together a pretty simply strategy in ~40 lines of code. The codebase is designed to be pretty flexible & modular. Mike from http://quantstart.com http://quantstart.com has some great resources & many of his recent articles use the same library. [1]: https://github.com/mhallsmoore/qstrader https://github.com/mhallsmoore/qstrader
- MR4D 10y ago
  Given the network latency of 43ms between, say, Houston, and NYC, you are completely correct. Source: https://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html https://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html
branchless 10y ago
Kdb is used in many large banks.
Ntrails 10y ago
Worth noting that there's a difference between analysis/research and a trading platform etc. I'd guess that low effort threshold to try and backtest ideas is very attractive even if you're going to have to rewrite fast/better/stronger in other languages to trade in real time?
collyw 10y ago
Numpy for Python provides wrappers around C arrays, so (apparently) comes close in terms of performance.
bicubic 10y ago
>Or are the R and Python Computational libraries close to the performance of C/C++ that it's a viable compromise? That's a big part of it. Python's data analysis tooling is generally written on top of Numpy, which is insanely optimised code that an average C/C++ developer couldn't compete with in terms of perf. Numpy will completely smoke naive C code to do common data manipulation tasks. So it's a double win, really. You get very good perf out of the box, and you get a high level dynamic typed language which lets you focus on high level logic and iterate quickly. The other part is the ecosystem. For a variety of reasons, Python has become the premier language for Big Data™, and over the past 5 years has accreted a huge collection of libraries for analytics, visualization, ML, distributed compute, etc. A single developer can now click these libraries together to achieve stuff that could only be done with huge teams before. You can build a distributed training cluster for a deep learning algorithm and deploy it to Amazon in maybe 2k lines of code. C++? I don't even know where to begin.
- neshibble 10y ago
  This. Python's stength as a language IMO comes from it's ability to interface with more efficient code. You can write the meat of your library at near C levels of efficiency, and have python treat it like a "black box" essentially. That seems to be how NN code works often as well.
  - robertk 10y ago
    Given that the R interpreter is written in C and admits trivial FFI bindings, as demonstrated by libraries like glmnet or gbm calling out to it, I don't see how this is an inherent advantage of Python.
    - EmlynC 10y ago
      While it doesn't have an inherent advantage, it has the mindshare and momentum of a community that has these tools now. R could be just as capable as Python, but I think Python has largely won the race to be the most popular language for data analysis which in turn encourage more developers to commit to it, cementing Python's advantage. R still has solid lead in statistics and a good mindshare amongst academics.
baldfat 10y ago
> It's understandable that python is preferred since it's an easy language to get started with Python is a good choice but R is amazingly good language that doesn't deserve to be pushed back negatively. It's not easier to learn Python then R. They both have plus and minuses but knowing both if I was to teach someone statistical programming I absolutely would teach R over Python.

by ktta 10y ago

tnecniv 10y ago

It mostly depends on what you want to do. If you want to try to beat the speed of the internet like some of the HFT stories you hear, you're going to want to cut as much fat out as you can. If you are more interested in doing some analysis of historic data where you don't care about speed so much, it's a lot easier to play around with things in a language like R or Python.

cgmil 10y ago
I hear that R/Python's role in the project is initial modelling and testing. You train trading algorithms in R or Python, then deploy them in a C/C++ algorithm for speed.
- xapata 10y ago
  NumPy is C++, Fortran, and sometimes CUDA. Depending on usage.

rampage101 10y ago

People use Python for finance and time series because of how easy it to read in the data. There is a library called pandas which allows you to read in a txt, csv, or excel file easily and do any type of array manipulation you want. The pandas library is written in C++, so it's basically a python wrapper. Conversely I don't think C++ has this same type of library for reading in data in a single line. While C++ is faster than Python overall, for most financial simulations Python is definitely fast enough.

em500 10y ago
Pandas was started as a skunkworks project inside a big hedge fund (AQR) by Wes McKinney (who after a few years at Cloudera is now at another hedge fund, Two Sigma). So it's no coincidence that pandas is very well adapted to financial / investment analysis.
jzwinck 10y ago
The Pandas library is not written in C at all actually. You can view the source here: https://github.com/pandas-dev/pandas/tree/master/pandas https://github.com/pandas-dev/pandas/tree/master/pandas It's mostly Python with a bit of Cython, and pull requests that are not pure Python are more likely to be rejected. There's basically zero C++ in Pandas itself.

cluoma 10y ago

Speaking only for R, most of the performance-critical functions included in packages these days are just wrappers to a C++ implementation. It isn't until you need to implement something custom that performance starts to be an issue. That combined with it's ease-of-use working with tabular data, and the myriad of packages available make it a very popular choice.

cgmil 10y ago
If Rcpp were not a thing, R might have died from being slow. Since it is a thing, though, people can easily write packages that are certainly fast enough for what they want to do, and that keeps the language useful.

dagw 10y ago

HFT is only a small part of financial data analysis. Everybody I know who works with mathematical modelling in finance use tools like SAS, Matlab, R and even Excel to build and run their models. If you're working at something like a pension fund and thinking years into the future then shaving a few milliseconds or even minutes here and there isn't just isn't a priority.

j7ake 10y ago

The main bottle neck in data analysis is not running the code but thinking about your data, visualizing it in different ways, and looking for interesting corner cases your model is not good at predicting. In that sense it is much faster to do these analyses in Python or R than in C++.

ryankennedyio 10y ago

I don't think any retailer can trade on the scale and speed that would require C/C++ over Python. With lower capital and significantly higher transaction costs, retail traders (mostly) need to have holding periods for at least hours, so speed isn't really an issue. It's more about finding longer term anomalies to deploy maybe at most a few dozen positions, as opposed to reacting to billions of quotes over any trading day. I think Python is a great tool for finance, particularly with libraries like Pandas, Numpy, etc. Most of the work in those libraries is C/Fortan/etc anyway. Great community. For more 'academic' style work/exploration, I think R might have an edge, but I personally find it more difficult to write and maintain high quality code in R. I'd like to just throw a quick plug in for an open source project I'm working on [1]. It's an event-driven trading library written in Python. We're pretty close to having live trade execution ready in an alpha state. The motivation is that you can put together a pretty simply strategy in ~40 lines of code. The codebase is designed to be pretty flexible & modular. Mike from http://quantstart.com http://quantstart.com has some great resources & many of his recent articles use the same library. [1]: https://github.com/mhallsmoore/qstrader https://github.com/mhallsmoore/qstrader

MR4D 10y ago
Given the network latency of 43ms between, say, Houston, and NYC, you are completely correct. Source: https://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html https://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html

branchless 10y ago

Kdb is used in many large banks.

Ntrails 10y ago

Worth noting that there's a difference between analysis/research and a trading platform etc. I'd guess that low effort threshold to try and backtest ideas is very attractive even if you're going to have to rewrite fast/better/stronger in other languages to trade in real time?

collyw 10y ago

Numpy for Python provides wrappers around C arrays, so (apparently) comes close in terms of performance.

bicubic 10y ago

>Or are the R and Python Computational libraries close to the performance of C/C++ that it's a viable compromise? That's a big part of it. Python's data analysis tooling is generally written on top of Numpy, which is insanely optimised code that an average C/C++ developer couldn't compete with in terms of perf. Numpy will completely smoke naive C code to do common data manipulation tasks. So it's a double win, really. You get very good perf out of the box, and you get a high level dynamic typed language which lets you focus on high level logic and iterate quickly. The other part is the ecosystem. For a variety of reasons, Python has become the premier language for Big Data™, and over the past 5 years has accreted a huge collection of libraries for analytics, visualization, ML, distributed compute, etc. A single developer can now click these libraries together to achieve stuff that could only be done with huge teams before. You can build a distributed training cluster for a deep learning algorithm and deploy it to Amazon in maybe 2k lines of code. C++? I don't even know where to begin.

neshibble 10y ago
This. Python's stength as a language IMO comes from it's ability to interface with more efficient code. You can write the meat of your library at near C levels of efficiency, and have python treat it like a "black box" essentially. That seems to be how NN code works often as well.
- robertk 10y ago
  Given that the R interpreter is written in C and admits trivial FFI bindings, as demonstrated by libraries like glmnet or gbm calling out to it, I don't see how this is an inherent advantage of Python.
  - EmlynC 10y ago
    While it doesn't have an inherent advantage, it has the mindshare and momentum of a community that has these tools now. R could be just as capable as Python, but I think Python has largely won the race to be the most popular language for data analysis which in turn encourage more developers to commit to it, cementing Python's advantage. R still has solid lead in statistics and a good mindshare amongst academics.

baldfat 10y ago

> It's understandable that python is preferred since it's an easy language to get started with Python is a good choice but R is amazingly good language that doesn't deserve to be pushed back negatively. It's not easier to learn Python then R. They both have plus and minuses but knowing both if I was to teach someone statistical programming I absolutely would teach R over Python.