7 ms·

An Introduction to Stock Market Data Analysis with R – Part 1

267 points by alecsx6 10y ago

fapjacks 10y ago
> DISCLAIMER: THIS IS NOT FINANCIAL ADVICE!!! Furthermore, I have ZERO experience as a trader (a lot of this knowledge comes from a one-semester course on stock trading I took at Salt Lake Community College)! This disclaimer is a good idea, but I'd have gone as far as saying that getting into HFT as a hobbyist programmer is going to be throwing your money into a giant hole.
- ccc111 10y ago
  agree. i will say as an advent stock guy i think most of this statistical stuff is bs. i think we like to find patterns and only the ones that work in the past dont really work in the future. stocks are run off news and articles and weather the big bank sells today or buys among a myriad of other things. i believe hft are only really meant for big companies that have to trade 300 positions and maintain certain standards. Not as a average guy running one in his basement to make $200 dollar a day like clock work, only if he worked a little harder and studied more phd statistics and found this "amazing unbeatable forumla". by the way would you like to make $5242 a month from working at home? insertscamwebsitehere,com
- justonepost 10y ago
  Not if you are careful, the problem is reliably beating just putting your money in a low cost index fund to a point it does better than what you'd be making in a real job. Good luck with that.
- justonepost 10y ago
  Not if you are careful, the problem is reliably beating just putting your money in a low cost index fund to a point it does better than what you'd be making in a real job. Good luck with that.
- csomar 10y ago
  I disagree, strongly. While the learning experience was super stressful and expensive and had me on stress-drugs, the financial advantage was nothing but insane. I'm a small time trader, started from 5 figures and now up to a good 6 figures of trading capital; and took close to a 6 figures of earnings already.
  - TheColorYellow 10y ago
    Really? I haven't many people to have claimed to have gotten an "insane" financial advantage from the stock market, yet alone HFT. I've met a few but none of them had such a positive reaction that it seemed worth investing a significant amount of time. Any advice?
    - csomar 10y ago
      Well, that's the catch. I'm very unlikely to give you any hints on what markets I trade and the advantages I'm using. I'm also very unlikely to take any person that does that very seriously (I really don't). Also remember that this market (or any market with big money) is brutal and will eat you alive. I don't expect the average person and his capital to last for a long time.
      - neshibble 10y ago
        I think this is one of the reasons Econ as a field has stagnated. (THIs is v hard to defend but bear with me). People have no reason to share / peer review their findings. It is extremely advantageous to share no, or bad advice. Hence why you get the 2007/08 housing bubble, as hundreds of econ professors peddle COMPLETELY false economic theory and results based analysis. In fact, many still are. Not much has changed in the field of economics. How data science isn't a mandatory requirement for such a data driven field just shows to me how immature the field is.
        arca_vorago 10y ago
        Looks like my data science degree is gonna be worth it then!
ktta 10y ago
I don't have much experience with any kind of stock market analysis/ HFT programming. I've seen people use python (or R) a lot in online tutorials and courses (there's one on coursera[1] check it out). It's understandable that python is preferred since it's an easy language to get started with, but I've never really seen C/C++ used by people online (Although I've heard about the use of FPGAs by the hardcore guys) Is the only reason C++ isn't recommended because it is difficult? The performance difference seems rather large to me. And especially if you're doing lots of number crunching, it would really benefit using C/C++ wouldn't it? Do people use them but I don't hear about them because anyone above 'tutorial level' doesn't share their code or talk about it? Or are the R and Python Computational libraries close to the performance of C/C++ that it's a viable compromise? [1]: https://www.coursera.org/learn/computational-investing https://www.coursera.org/learn/computational-investing
- tnecniv 10y ago
  It mostly depends on what you want to do. If you want to try to beat the speed of the internet like some of the HFT stories you hear, you're going to want to cut as much fat out as you can. If you are more interested in doing some analysis of historic data where you don't care about speed so much, it's a lot easier to play around with things in a language like R or Python.
  - cgmil 10y ago
    I hear that R/Python's role in the project is initial modelling and testing. You train trading algorithms in R or Python, then deploy them in a C/C++ algorithm for speed.
    - xapata 10y ago
      NumPy is C++, Fortran, and sometimes CUDA. Depending on usage.
- rampage101 10y ago
  People use Python for finance and time series because of how easy it to read in the data. There is a library called pandas which allows you to read in a txt, csv, or excel file easily and do any type of array manipulation you want. The pandas library is written in C++, so it's basically a python wrapper. Conversely I don't think C++ has this same type of library for reading in data in a single line. While C++ is faster than Python overall, for most financial simulations Python is definitely fast enough.
jdonaldson 10y ago
You can still profit off of HFT, even if you can't compete directly. The key is to find the opportunity they create: HFT will typically exit HARD on a stock with weak earnings or bad news. They need to find a better short term growth option. I like to find those situations, buy low, and hang on for a year.
soheil 10y ago
It'd be great if this was posted on Kaggle.
ccc111 10y ago
Awesome article! have you guys been on https://www.quantopian.com/ https://www.quantopian.com/ very informative very good blog statistic articles on top of backtesting
tempodox 10y ago
You might just as well “analyze” dice rolls at the casino. Except that people routinely lie to you when they want your money (in case you didn't know already), there's no insight to be had. It is already a well established fact that in gambling, the house always wins.
- user5994461 10y ago
  > It is already a well established fact that in gambling, the house always wins. I had a friend who used to run a financial exchange. That was his favorite sentence. The house always wins.
empath75 10y ago
Is there a similar article for analyzing stock fundamentals? (P/e, revenue, etc)
- cgmil 10y ago
  No, but I'm always looking for new content ideas. People like my R/Python for Stocks posts, and ask for more, but I'm not really sure where to start in terms of giving them what they want.

267 points by alecsx6 10y ago

> DISCLAIMER: THIS IS NOT FINANCIAL ADVICE!!! Furthermore, I have ZERO experience as a trader (a lot of this knowledge comes from a one-semester course on stock trading I took at Salt Lake Community College)! This disclaimer is a good idea, but I'd have gone as far as saying that getting into HFT as a hobbyist programmer is going to be throwing your money into a giant hole.

ccc111 10y ago
agree. i will say as an advent stock guy i think most of this statistical stuff is bs. i think we like to find patterns and only the ones that work in the past dont really work in the future. stocks are run off news and articles and weather the big bank sells today or buys among a myriad of other things. i believe hft are only really meant for big companies that have to trade 300 positions and maintain certain standards. Not as a average guy running one in his basement to make $200 dollar a day like clock work, only if he worked a little harder and studied more phd statistics and found this "amazing unbeatable forumla". by the way would you like to make $5242 a month from working at home? insertscamwebsitehere,com
justonepost 10y ago
Not if you are careful, the problem is reliably beating just putting your money in a low cost index fund to a point it does better than what you'd be making in a real job. Good luck with that.
justonepost 10y ago
Not if you are careful, the problem is reliably beating just putting your money in a low cost index fund to a point it does better than what you'd be making in a real job. Good luck with that.
csomar 10y ago
I disagree, strongly. While the learning experience was super stressful and expensive and had me on stress-drugs, the financial advantage was nothing but insane. I'm a small time trader, started from 5 figures and now up to a good 6 figures of trading capital; and took close to a 6 figures of earnings already.
- TheColorYellow 10y ago
  Really? I haven't many people to have claimed to have gotten an "insane" financial advantage from the stock market, yet alone HFT. I've met a few but none of them had such a positive reaction that it seemed worth investing a significant amount of time. Any advice?
  - csomar 10y ago
    Well, that's the catch. I'm very unlikely to give you any hints on what markets I trade and the advantages I'm using. I'm also very unlikely to take any person that does that very seriously (I really don't). Also remember that this market (or any market with big money) is brutal and will eat you alive. I don't expect the average person and his capital to last for a long time.
    - neshibble 10y ago
      I think this is one of the reasons Econ as a field has stagnated. (THIs is v hard to defend but bear with me). People have no reason to share / peer review their findings. It is extremely advantageous to share no, or bad advice. Hence why you get the 2007/08 housing bubble, as hundreds of econ professors peddle COMPLETELY false economic theory and results based analysis. In fact, many still are. Not much has changed in the field of economics. How data science isn't a mandatory requirement for such a data driven field just shows to me how immature the field is.
      - arca_vorago 10y ago
        Looks like my data science degree is gonna be worth it then!

ktta 10y ago

I don't have much experience with any kind of stock market analysis/ HFT programming. I've seen people use python (or R) a lot in online tutorials and courses (there's one on coursera[1] check it out). It's understandable that python is preferred since it's an easy language to get started with, but I've never really seen C/C++ used by people online (Although I've heard about the use of FPGAs by the hardcore guys) Is the only reason C++ isn't recommended because it is difficult? The performance difference seems rather large to me. And especially if you're doing lots of number crunching, it would really benefit using C/C++ wouldn't it? Do people use them but I don't hear about them because anyone above 'tutorial level' doesn't share their code or talk about it? Or are the R and Python Computational libraries close to the performance of C/C++ that it's a viable compromise? [1]: https://www.coursera.org/learn/computational-investing https://www.coursera.org/learn/computational-investing

tnecniv 10y ago
It mostly depends on what you want to do. If you want to try to beat the speed of the internet like some of the HFT stories you hear, you're going to want to cut as much fat out as you can. If you are more interested in doing some analysis of historic data where you don't care about speed so much, it's a lot easier to play around with things in a language like R or Python.
- cgmil 10y ago
  I hear that R/Python's role in the project is initial modelling and testing. You train trading algorithms in R or Python, then deploy them in a C/C++ algorithm for speed.
  - xapata 10y ago
    NumPy is C++, Fortran, and sometimes CUDA. Depending on usage.
rampage101 10y ago
People use Python for finance and time series because of how easy it to read in the data. There is a library called pandas which allows you to read in a txt, csv, or excel file easily and do any type of array manipulation you want. The pandas library is written in C++, so it's basically a python wrapper. Conversely I don't think C++ has this same type of library for reading in data in a single line. While C++ is faster than Python overall, for most financial simulations Python is definitely fast enough.

jdonaldson 10y ago

You can still profit off of HFT, even if you can't compete directly. The key is to find the opportunity they create: HFT will typically exit HARD on a stock with weak earnings or bad news. They need to find a better short term growth option. I like to find those situations, buy low, and hang on for a year.

soheil 10y ago

It'd be great if this was posted on Kaggle.

ccc111 10y ago

Awesome article! have you guys been on https://www.quantopian.com/ https://www.quantopian.com/ very informative very good blog statistic articles on top of backtesting

tempodox 10y ago

You might just as well “analyze” dice rolls at the casino. Except that people routinely lie to you when they want your money (in case you didn't know already), there's no insight to be had. It is already a well established fact that in gambling, the house always wins.

user5994461 10y ago
> It is already a well established fact that in gambling, the house always wins. I had a friend who used to run a financial exchange. That was his favorite sentence. The house always wins.

empath75 10y ago

Is there a similar article for analyzing stock fundamentals? (P/e, revenue, etc)

cgmil 10y ago
No, but I'm always looking for new content ideas. People like my R/Python for Stocks posts, and ask for more, but I'm not really sure where to start in terms of giving them what they want.