7 ms·

Show HN: How I built a trading signal by scraping Nasdaq for short interest

89 points by fawce 14y ago

chatmasta 14y ago
Can somebody explain to me why, if this really works, you would publish it in a blogpost? Shouldn't you be hunting down investments of $X to turn $1.093X?
- jhowell 14y ago
  IMHO, when developing a trading strategy it helps to document and share your strategy with others as you'll come to better understand it from the questions and observations others make. No one strategy can or will be successful forever. Many algorithms stop performing when market conditions (lasting hours, days, weeks, months) change. Having a deep understanding of your algorithm and what makes it successful for any given period of time can better help you make adjustments when needed. Lastly, this may be where the algo started but not necessarily what they will run in production. It's much more likely to no longer be discussed at this point. Perhaps similar to ideas are worthless, execution is everything for startups. edit: typo
  - codex 14y ago
    If an algorithm stops performing after hours or days, it's likely you haven't discovered anything, but are simply seeing the effects of random noise on your hundreds, thousands, or millions of signal possibilities.
    - jhowell 14y ago
      One example that comes to my mind could be an algo closely related to the price of another security or index of what have you. At times, this algo could be highly correlative and at other times less so. I agree with you about random noise. Ultimately I'm just looking for something to make me feel like I'm taking an "informed position." You never really know what's going to happen.
  - chatmasta 14y ago
    Have their been any studies on the accuracy of backtesting data in predicting actual returns? Comments like yours always appear on these articles... "You can backtest, but you never know what's going to happen!" Well, obviously. But at what point is testing against 12 years of backdata not enough? Is it a matter of understanding exactly what your algorithm depends on and watching for those conditions to fail?
    - jhowell 14y ago
      To me, backtests are constructive but like you said, not predictive and perhaps questionably meaningful. Any good strategy using technical analysis will be able to identify a trend in any market. What differs are entries and exits. These factors can be influenced by current events making them tough to model into an algorithm especially with one with a shorter timeframe.
- prodigal_erik 14y ago
  Like selling pickaxes during a gold rush, Quantopian is probably better off encouraging hobbyists to start speculating via their platform.
  - pathdependent 14y ago
    Yes. Which seems like a better investment: dedication to a platform that certainly will attract speculators and is underserved in general or dedication to a trading algorithm in an arena overpopulated by very smart people competing in a zero sum game -- against you? Both provide the potential for scalable, massive returns, yet in the quant realm, selling pickaxes is a much safer bet and is equally -- if not more -- lucrative. (In the contextually correct parlance, the sharpe ratio is much higher for selling Quantopian.)
- steven2012 14y ago
  I'd be hard-pressed to call this algo as "working". If you look at the PnL, it makes most of its profit from Sept 15 2012 to Nov 15 2012. The rest of the time, it's either drawn down or chopping back and forth. 2 months of gains over a one year period sounds more like noise to me than anything else.
tokenadult 14y ago
My comment in the last thread opened with a post from this source: Past performance does not guarantee future results" is still the operative principle here. Data-mining discovers patterns, but it doesn't lead to deep insight into causes, and markets are perturbed by many events that you don't put into your training algorithm. "The market can remain irrational longer than you can remain solvent" is still important investment advice. You can never build a trading signal just by scraping historical data, unless you like losing your shirt. Can you tell I'm reading Antifragile: Things That Gain from Disorder just now? I'm very sensitive to errors in statistical thinking today.
- fchollet 14y ago
  You can do it; you just need to only evaluate your algorithm on data it hasn't been trained on. The same as with any machine learning problem really. Though it is indeed dismaying how many programmers dabbling into ML tend to do so without any scientific rigor... Whether the stockmarket can or cannot be predicted on the short term based on its past is another question... but I've been gathering some convincing evidence that it cannot (as in, its variations have no intrinsic structure; though it can still be predicted based on various external factors).
  - fawce 14y ago
    what's an example of an external factor? maybe I can find it on quandl and backtest it.
    - fchollet 14y ago
      The stockmarket is primarily driven by the news, both on the ultra-short term (minute scale) and at the scale of a few days. So, scraping the web and performing semantic analysis of investor sentiment can be a good way to get an edge in the game. Though in order to be really effective you'd need to do it before Wall Street traders' reactions adjust the prices, ie. get access to Bloomberg's B-Pipe [1], do real time semantic analysis and place orders with ultra low latency. Which quite a few trading firms are already doing... Other factors include P/E ratio (only accurate in the longer term), div yield, recent growth... [1] http://www.bloomberg.com/enterprise/enterprise_products/data_optimization/data_feeds/ http://www.bloomberg.com/enterprise/enterprise_products/data...
niggler 14y ago
Did anyone actually try this with real money? Does the model include transaction costs and market impact effect?
- fawce 14y ago
  If you click on the code and search for commissions, you'll see how those costs are taken into account. The big missing thing is the market for borrowing the stock to do the short side of the trade. No money has traded on my version no. But, I understand that asset management firms have licensed the more sophisticated one Jess wrote at TR, so I would think they use it with real money. From what I understand, firms look at numerous signals like this, and then make investments based on a combination of the signals.
dkhenry 14y ago
I have for two years now been playing around with Algorithmic trading as a hobby and I am amazed by people who think wave riders or simple mathmatical transforms will get them profits in the market. I have found that the best method is still a good mix of modeling and trader input. I don't think a model exists that you can just turn on and have it print you money. So attempts like this to make one of those really are a waste of time. Your systems should be tuned to listen to you and then take what input you have and do what you cannot ( make decision in sub-second windows )
- megaframe 14y ago
  Completely agree, been doing the same for last 3 years using a combination of machine learning techniques which without some human input are only at best as successful as putting money into a savings account, or in most cases would loose money.
- fawce 14y ago
  what do you mean by trader input?
  - cinquemb 14y ago
    news events? info from discussions on trading/economic/product related forums? (insider info? dont tell the SEC :P)
- vecter 14y ago
  I don't know where you're getting your data from, but I know of at least one high frequency algorithmic trading firm that make ~$1B a year using mathematical models. The models aren't simple, but they're entirely automated and they behave exactly the opposite of how you describe them: you turn them on and they print unbelievable gobs of money.
  - aortega 14y ago
    Anybody else think this is like, inherently bad? I mean making money from nothing, producing nothing, doing no service to anybody. The only way you could possible get that billion without doing nothing is to take it from other people, essentially stealing it. Why is this legal?
ikea_meatballs 14y ago
Back testing is a real bitch. I've been building my own app for back testing recently, my specific interest being how published insider buys (SEC Form 4 transactions) affect the prices of stocks in the short near and long term. You can get dividend data and stock splits easily enough from some public feeds. But where do you get a database of ticker changes, bankruptcy events, and spin-offs, especially on the OTC markets? You can't unless you're willing to shell out a lot of money. Back testing properly is probably out of the cost range of the individual investor. Some examples: * Lehhman's ticker changes on the way down * GM going bankrupt and then coming back from the dead! * Skye International used to trade under SKYY (at 0.35c/share), but now SKYY tracks a cloud SaaS ETF 20.60/share). Think you got a big win using that strategy that including buying SKYY? Think again!
- Iterated 14y ago
  Have you looked into CRSP (Center for Research in Security Prices)? I know they have all that data, I'm not sure how much it costs though. Probably not profitable for the average retail trader.
- fawce 14y ago
  You're speaking truth. We (quantopian) deal with all of those headaches, and test the algos with fully adjusted data. Splits, symbol changes, mergers, divestitures, dead companies, dividends - they're all covered.
- spitfire 14y ago
  tickdata.com has split adjusted, survivor bias free data. It isn't cheap however. You'll still need to go elsewhere for the SEC filings though.
- kal00ma 14y ago
  I've been working on a similar strategy after having read Nejat's book: http://www.amazon.com/Investment-Intelligence-Insider-Trading-Seyhun/dp/0262692341 http://www.amazon.com/Investment-Intelligence-Insider-Tradin... The plan is to derive trading signals from insider purchase data while taking into account the insider's relative risk-aversion (estimated from age, salary, sex). At this point I'm just trying to recreate Nejat's results. Data-quality seems to be an issue (stock splits aren't recorded in the yahoo data). If you would like to collaborate or trade ideas message kal00ma on reddit.
stevewilhelm 14y ago
When the broad market is rising by over 10% annually, it is very difficult to come up with a trading strategy that looses money. For example, buying SPY and holding it for the same period would have outperformed your algorithm.
- steven2012 14y ago
  Sorry, but saying that "it is very difficult to come up with a trading strategy that loses money" means you really have no credible experience with running trading algorithms that use real money.
  - hyperbovine 14y ago
    Do your care to address his point? If the S&P 500 genuinely outperforms "your" (his? someone's) algorithm, said algorithm is a priori unimpressive.
polskibus 14y ago
Serious question: does this meet "Show HN" criteria? I mean I value sharing the algorithm, but I thought that Show HN is reserved for entire projects (ie. sites, saas platforms, etc.), not using ones platform to put up a description of algorithm and some numeric data. I'm not trying to troll, just wanted to know how the community understands "Show HNs"? In this case it can be seen as more of a Quantopian show off (which is interesting service, but had already been showcased) than the algorithm or project itself?
- polskibus 14y ago
  Why the downvotes? I explicitly said I just want an answer about how the community sees "Show HNs", not attacking anyone. Is that really that offensive and unconstructive? How are we suppose to improve on quality of this environment if one cannot ask about community guidelines?
jstauth 14y ago
While I'd love to take all the credit (blame?), the reformed academic in my feels compelled to admit that the idea to look for predictive value in stock loan data is not original to me. The finance literature has some fascinating articles on this dating back as far as the late 80s (look for Desai 2002, J of Finance, Asquith 2005 J Fin Econ, or most recently http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1570451 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1570451). The intuition behind this signal as a market inefficiency, or 'anomaly' is that the market sees short sellers as informed investors, the so called 'smart money', and there is a herding effect to follow their trades which generates abnormal returns. The same logic can be applied to disclosed insider trades or institutional holdings filings made public via the SEC's EDGAR database. Fawce's slick implementation of a 'Days to Cover' signal is a great way to highlight the power of aiming new tools like Quantopian at freely available public data stores (which exist expressly to increase market transparency). And sure, it doesn't go the whole way for you on execution details like borrow costs, liquidity etc. but those aspects tend to be unique to each trader.
vellum 14y ago
You should put in some kind of protection for a max drawdown loss, like if you lose x%, you exit. Sometimes your algorithm messes up, or market conditions are bad. http://www.businessinsider.com/hedge-funds-smashed-worst-quarter-since-2008-collapse-2011-10 http://www.businessinsider.com/hedge-funds-smashed-worst-qua... Long short equity funds did poorly in 2008 financial crisis, and also in 2011, when there was high volatility.
- fawce 14y ago
  It would be cool to do that with this signal, if the algo was buying/selling on another signal. Maybe use the short interest signal as a gate on momentum investing for example.
ad 14y ago
Very interesting stuff. "The Benchmark" is the SP500 I'm guessing? I couldn't find the answer after clicking around for a bit, sorry if I'm dumb. You might list the reference security in the chart, or do something like "SPY (benchmark)" in the key.

89 points by fawce 14y ago

Can somebody explain to me why, if this really works, you would publish it in a blogpost? Shouldn't you be hunting down investments of $X to turn $1.093X?

jhowell 14y ago
IMHO, when developing a trading strategy it helps to document and share your strategy with others as you'll come to better understand it from the questions and observations others make. No one strategy can or will be successful forever. Many algorithms stop performing when market conditions (lasting hours, days, weeks, months) change. Having a deep understanding of your algorithm and what makes it successful for any given period of time can better help you make adjustments when needed. Lastly, this may be where the algo started but not necessarily what they will run in production. It's much more likely to no longer be discussed at this point. Perhaps similar to ideas are worthless, execution is everything for startups. edit: typo
- codex 14y ago
  If an algorithm stops performing after hours or days, it's likely you haven't discovered anything, but are simply seeing the effects of random noise on your hundreds, thousands, or millions of signal possibilities.
  - jhowell 14y ago
    One example that comes to my mind could be an algo closely related to the price of another security or index of what have you. At times, this algo could be highly correlative and at other times less so. I agree with you about random noise. Ultimately I'm just looking for something to make me feel like I'm taking an "informed position." You never really know what's going to happen.
- chatmasta 14y ago
  Have their been any studies on the accuracy of backtesting data in predicting actual returns? Comments like yours always appear on these articles... "You can backtest, but you never know what's going to happen!" Well, obviously. But at what point is testing against 12 years of backdata not enough? Is it a matter of understanding exactly what your algorithm depends on and watching for those conditions to fail?
  - jhowell 14y ago
    To me, backtests are constructive but like you said, not predictive and perhaps questionably meaningful. Any good strategy using technical analysis will be able to identify a trend in any market. What differs are entries and exits. These factors can be influenced by current events making them tough to model into an algorithm especially with one with a shorter timeframe.
prodigal_erik 14y ago
Like selling pickaxes during a gold rush, Quantopian is probably better off encouraging hobbyists to start speculating via their platform.
- pathdependent 14y ago
  Yes. Which seems like a better investment: dedication to a platform that certainly will attract speculators and is underserved in general or dedication to a trading algorithm in an arena overpopulated by very smart people competing in a zero sum game -- against you? Both provide the potential for scalable, massive returns, yet in the quant realm, selling pickaxes is a much safer bet and is equally -- if not more -- lucrative. (In the contextually correct parlance, the sharpe ratio is much higher for selling Quantopian.)
steven2012 14y ago
I'd be hard-pressed to call this algo as "working". If you look at the PnL, it makes most of its profit from Sept 15 2012 to Nov 15 2012. The rest of the time, it's either drawn down or chopping back and forth. 2 months of gains over a one year period sounds more like noise to me than anything else.

tokenadult 14y ago

My comment in the last thread opened with a post from this source: Past performance does not guarantee future results" is still the operative principle here. Data-mining discovers patterns, but it doesn't lead to deep insight into causes, and markets are perturbed by many events that you don't put into your training algorithm. "The market can remain irrational longer than you can remain solvent" is still important investment advice. You can never build a trading signal just by scraping historical data, unless you like losing your shirt. Can you tell I'm reading Antifragile: Things That Gain from Disorder just now? I'm very sensitive to errors in statistical thinking today.

fchollet 14y ago
You can do it; you just need to only evaluate your algorithm on data it hasn't been trained on. The same as with any machine learning problem really. Though it is indeed dismaying how many programmers dabbling into ML tend to do so without any scientific rigor... Whether the stockmarket can or cannot be predicted on the short term based on its past is another question... but I've been gathering some convincing evidence that it cannot (as in, its variations have no intrinsic structure; though it can still be predicted based on various external factors).
- fawce 14y ago
  what's an example of an external factor? maybe I can find it on quandl and backtest it.
  - fchollet 14y ago
    The stockmarket is primarily driven by the news, both on the ultra-short term (minute scale) and at the scale of a few days. So, scraping the web and performing semantic analysis of investor sentiment can be a good way to get an edge in the game. Though in order to be really effective you'd need to do it before Wall Street traders' reactions adjust the prices, ie. get access to Bloomberg's B-Pipe [1], do real time semantic analysis and place orders with ultra low latency. Which quite a few trading firms are already doing... Other factors include P/E ratio (only accurate in the longer term), div yield, recent growth... [1] http://www.bloomberg.com/enterprise/enterprise_products/data_optimization/data_feeds/ http://www.bloomberg.com/enterprise/enterprise_products/data...

niggler 14y ago

Did anyone actually try this with real money? Does the model include transaction costs and market impact effect?

fawce 14y ago
If you click on the code and search for commissions, you'll see how those costs are taken into account. The big missing thing is the market for borrowing the stock to do the short side of the trade. No money has traded on my version no. But, I understand that asset management firms have licensed the more sophisticated one Jess wrote at TR, so I would think they use it with real money. From what I understand, firms look at numerous signals like this, and then make investments based on a combination of the signals.

dkhenry 14y ago

I have for two years now been playing around with Algorithmic trading as a hobby and I am amazed by people who think wave riders or simple mathmatical transforms will get them profits in the market. I have found that the best method is still a good mix of modeling and trader input. I don't think a model exists that you can just turn on and have it print you money. So attempts like this to make one of those really are a waste of time. Your systems should be tuned to listen to you and then take what input you have and do what you cannot ( make decision in sub-second windows )

megaframe 14y ago
Completely agree, been doing the same for last 3 years using a combination of machine learning techniques which without some human input are only at best as successful as putting money into a savings account, or in most cases would loose money.
fawce 14y ago
what do you mean by trader input?
- cinquemb 14y ago
  news events? info from discussions on trading/economic/product related forums? (insider info? dont tell the SEC :P)
vecter 14y ago
I don't know where you're getting your data from, but I know of at least one high frequency algorithmic trading firm that make ~$1B a year using mathematical models. The models aren't simple, but they're entirely automated and they behave exactly the opposite of how you describe them: you turn them on and they print unbelievable gobs of money.
- aortega 14y ago
  Anybody else think this is like, inherently bad? I mean making money from nothing, producing nothing, doing no service to anybody. The only way you could possible get that billion without doing nothing is to take it from other people, essentially stealing it. Why is this legal?

ikea_meatballs 14y ago

Back testing is a real bitch. I've been building my own app for back testing recently, my specific interest being how published insider buys (SEC Form 4 transactions) affect the prices of stocks in the short near and long term. You can get dividend data and stock splits easily enough from some public feeds. But where do you get a database of ticker changes, bankruptcy events, and spin-offs, especially on the OTC markets? You can't unless you're willing to shell out a lot of money. Back testing properly is probably out of the cost range of the individual investor. Some examples: * Lehhman's ticker changes on the way down * GM going bankrupt and then coming back from the dead! * Skye International used to trade under SKYY (at 0.35c/share), but now SKYY tracks a cloud SaaS ETF 20.60/share). Think you got a big win using that strategy that including buying SKYY? Think again!

Iterated 14y ago
Have you looked into CRSP (Center for Research in Security Prices)? I know they have all that data, I'm not sure how much it costs though. Probably not profitable for the average retail trader.
fawce 14y ago
You're speaking truth. We (quantopian) deal with all of those headaches, and test the algos with fully adjusted data. Splits, symbol changes, mergers, divestitures, dead companies, dividends - they're all covered.
spitfire 14y ago
tickdata.com has split adjusted, survivor bias free data. It isn't cheap however. You'll still need to go elsewhere for the SEC filings though.
kal00ma 14y ago
I've been working on a similar strategy after having read Nejat's book: http://www.amazon.com/Investment-Intelligence-Insider-Trading-Seyhun/dp/0262692341 http://www.amazon.com/Investment-Intelligence-Insider-Tradin... The plan is to derive trading signals from insider purchase data while taking into account the insider's relative risk-aversion (estimated from age, salary, sex). At this point I'm just trying to recreate Nejat's results. Data-quality seems to be an issue (stock splits aren't recorded in the yahoo data). If you would like to collaborate or trade ideas message kal00ma on reddit.

stevewilhelm 14y ago

When the broad market is rising by over 10% annually, it is very difficult to come up with a trading strategy that looses money. For example, buying SPY and holding it for the same period would have outperformed your algorithm.

steven2012 14y ago
Sorry, but saying that "it is very difficult to come up with a trading strategy that loses money" means you really have no credible experience with running trading algorithms that use real money.
- hyperbovine 14y ago
  Do your care to address his point? If the S&P 500 genuinely outperforms "your" (his? someone's) algorithm, said algorithm is a priori unimpressive.

polskibus 14y ago

Serious question: does this meet "Show HN" criteria? I mean I value sharing the algorithm, but I thought that Show HN is reserved for entire projects (ie. sites, saas platforms, etc.), not using ones platform to put up a description of algorithm and some numeric data. I'm not trying to troll, just wanted to know how the community understands "Show HNs"? In this case it can be seen as more of a Quantopian show off (which is interesting service, but had already been showcased) than the algorithm or project itself?

polskibus 14y ago
Why the downvotes? I explicitly said I just want an answer about how the community sees "Show HNs", not attacking anyone. Is that really that offensive and unconstructive? How are we suppose to improve on quality of this environment if one cannot ask about community guidelines?

jstauth 14y ago

While I'd love to take all the credit (blame?), the reformed academic in my feels compelled to admit that the idea to look for predictive value in stock loan data is not original to me. The finance literature has some fascinating articles on this dating back as far as the late 80s (look for Desai 2002, J of Finance, Asquith 2005 J Fin Econ, or most recently http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1570451 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1570451). The intuition behind this signal as a market inefficiency, or 'anomaly' is that the market sees short sellers as informed investors, the so called 'smart money', and there is a herding effect to follow their trades which generates abnormal returns. The same logic can be applied to disclosed insider trades or institutional holdings filings made public via the SEC's EDGAR database. Fawce's slick implementation of a 'Days to Cover' signal is a great way to highlight the power of aiming new tools like Quantopian at freely available public data stores (which exist expressly to increase market transparency). And sure, it doesn't go the whole way for you on execution details like borrow costs, liquidity etc. but those aspects tend to be unique to each trader.

vellum 14y ago

You should put in some kind of protection for a max drawdown loss, like if you lose x%, you exit. Sometimes your algorithm messes up, or market conditions are bad. http://www.businessinsider.com/hedge-funds-smashed-worst-quarter-since-2008-collapse-2011-10 http://www.businessinsider.com/hedge-funds-smashed-worst-qua... Long short equity funds did poorly in 2008 financial crisis, and also in 2011, when there was high volatility.

fawce 14y ago
It would be cool to do that with this signal, if the algo was buying/selling on another signal. Maybe use the short interest signal as a gate on momentum investing for example.

ad 14y ago

Very interesting stuff. "The Benchmark" is the SP500 I'm guessing? I couldn't find the answer after clicking around for a bit, sorry if I'm dumb. You might list the reference security in the chart, or do something like "SPY (benchmark)" in the key.