5 ms·

The problem with the paper is not overfit. They claim to have run their simulation with out of band ("live") data. The actual problem with the paper is that we

by minimax 12y ago

The problem with the paper is not overfit. They claim to have run their simulation with out of band ("live") data. The actual problem with the paper is that we have no idea if their simulator is any good, which means that their result (89% return in 50 days) could be totally bogus. In other words, we don't know if the actual bitcoin exchange would fill their orders at the same prices (if at all) as their simulator does. A decent simulator for a high-frequecy strategy like this is not trivial because you have to incorporate all the exchange behavior (documented or not) into your simulator, and then you have to validate your results by comparing your simulated results to the results of some actual trading. The fact that they spent none of the paper on the details of the simulator makes me extremely skeptical.

latj 12y ago
Imagine if someone learned how to legally print money and their first action was to tell the world-- then I would be extremely skeptical.
lrm242 12y ago
People so often overlook the role that execution plays in trading. As time scales shrink the impact of execution increases, and indeed many HFT strategies are not profitable without top tier execution (both in terms of fee schedules and technology). This is also one of the places that most academics fail when analysing a trading strategy. They make typical "assume a frictionless surface" types of assumptions that break down in a real market. To your point, high-frequency simulation is damn hard and it is highly likely that they failed here and have tainted their results with bad simulations. Most of these papers would be well served to avoid dollars and cents and simply analyse the statistical qualities of their signal to their target over time. That is the first step in this business, anyway. Simulations only happen once the signal has been through the wringer.
- hft_throwaway 12y ago
  I see no mention of the spread, costs or even ability to put on a short position, exchange lag variability (huge issue when simulating even on modern exchanges, let alone fly-by-night bitcoin markets). Additionally, it is very easy to find trend-following signals that "work" but break down when conditions change very quickly. That seems to make up most of the prediction, along with an order book imbalance signal that might be more stable. I think a better approach would be looking at order book features and lags vs. other markets. The costs are high on BTC markets so it would be pretty tough to overcome those though. Anyone with experience to do this is probably doing it somewhere more lucrative. I think BTC markets only trade a few million USD a day.
  - reduce 12y ago
    About 35 million USD in trades on the public bitcoin-paired markets today. About 40 million USD today in total volume for all pairs on these exchanges. http://www.cryptocoincharts.info/coins/info http://www.cryptocoincharts.info/coins/info
    - hft_throwaway 12y ago
      More than when I last looked, but to put that in perspective, you could trade $40mm in a few price ticks in major currency pairs and CME avg daily turnover alone is > $100bn: http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Record-Trading-in-its-Global-FX-Complex-Transacted-239-billion-in-Notional-Volume-Across-FX-Futures-and-Options?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+mediaroom%2FCMsF+%28CME+Group+Media+Room%29 http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Reco... Public spot FX markets and OTC trading do huge volumes too. Some BTC exchanges charge like 60 bps per trade. Finding a signal to overcome that cost and do enough trading to make it worthwhile would be quite difficult.
deleted 12y ago
[deleted]
rational-future 12y ago
These guys did it right. You first build a simulator and publish a paper and then join a hedge fund for $$$$$$ and actually try the algo.
Dylan16807 12y ago
I remember exchange delays on mtgox reaching into the minutes! This algorithm breaks with mere seconds of delay.
DavidSJ 12y ago
Even with live data as a validation set (and ignoring your point about execution friction), it's inappropriate to naively infer expected future return from total return during the validation run, since asset price fluctuations don't follow a normal distribution. Very high volatility events happen far more commonly than a Bell curve predicts, hence a few lucky or unlucky outliers can skew your total. The Sharpe ratio which they used to measure risk exposure also dangerously assumes a normal distribution.
downandout 12y ago
>we don't know if the actual bitcoin exchange would fill their orders at the same prices This is the problem with many simulated trading algorithm "success" stories. No one actually knows the effect that the simulated trades would have on the real market, or how others trading against the algorithm would react to these trades. The numbers may be somewhat close to reality if the simulated trades are very small in relation to the trading volume for the asset. However, Bitcoin markets are small to begin with, which probably makes the results in this paper unreliable.
zaptheimpaler 12y ago
From the article: >Specifically, every two seconds they predicted the average price movement over the following 10 seconds. If the price movement was higher than a certain threshold, they bought a Bitcoin; if it was lower than the opposite threshold, they sold one; and if it was in-between, they did nothing. If they are indeed predicting the price at X+10 seconds at second X, they have more than enough time to act on that info without having to do HFT.
- hft_throwaway 12y ago
  That assumes there is no bias to their signal returns over time, which is extremely unlikely. 90% of the moves that go their way could happen in the first second, for example. Many HFTs hold positions for seconds or minutes, and predict prices out over similar time horizons, but they'll lose their best winners to competitors if they aren't fast and adept at executing.

by minimax 12y ago

latj 12y ago

Imagine if someone learned how to legally print money and their first action was to tell the world-- then I would be extremely skeptical.

lrm242 12y ago

People so often overlook the role that execution plays in trading. As time scales shrink the impact of execution increases, and indeed many HFT strategies are not profitable without top tier execution (both in terms of fee schedules and technology). This is also one of the places that most academics fail when analysing a trading strategy. They make typical "assume a frictionless surface" types of assumptions that break down in a real market. To your point, high-frequency simulation is damn hard and it is highly likely that they failed here and have tainted their results with bad simulations. Most of these papers would be well served to avoid dollars and cents and simply analyse the statistical qualities of their signal to their target over time. That is the first step in this business, anyway. Simulations only happen once the signal has been through the wringer.

hft_throwaway 12y ago
I see no mention of the spread, costs or even ability to put on a short position, exchange lag variability (huge issue when simulating even on modern exchanges, let alone fly-by-night bitcoin markets). Additionally, it is very easy to find trend-following signals that "work" but break down when conditions change very quickly. That seems to make up most of the prediction, along with an order book imbalance signal that might be more stable. I think a better approach would be looking at order book features and lags vs. other markets. The costs are high on BTC markets so it would be pretty tough to overcome those though. Anyone with experience to do this is probably doing it somewhere more lucrative. I think BTC markets only trade a few million USD a day.
- reduce 12y ago
  About 35 million USD in trades on the public bitcoin-paired markets today. About 40 million USD today in total volume for all pairs on these exchanges. http://www.cryptocoincharts.info/coins/info http://www.cryptocoincharts.info/coins/info
  - hft_throwaway 12y ago
    More than when I last looked, but to put that in perspective, you could trade $40mm in a few price ticks in major currency pairs and CME avg daily turnover alone is > $100bn: http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Record-Trading-in-its-Global-FX-Complex-Transacted-239-billion-in-Notional-Volume-Across-FX-Futures-and-Options?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+mediaroom%2FCMsF+%28CME+Group+Media+Room%29 http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Reco... Public spot FX markets and OTC trading do huge volumes too. Some BTC exchanges charge like 60 bps per trade. Finding a signal to overcome that cost and do enough trading to make it worthwhile would be quite difficult.

deleted 12y ago

[deleted]

rational-future 12y ago

These guys did it right. You first build a simulator and publish a paper and then join a hedge fund for $$$$$$ and actually try the algo.

Dylan16807 12y ago

I remember exchange delays on mtgox reaching into the minutes! This algorithm breaks with mere seconds of delay.

DavidSJ 12y ago

Even with live data as a validation set (and ignoring your point about execution friction), it's inappropriate to naively infer expected future return from total return during the validation run, since asset price fluctuations don't follow a normal distribution. Very high volatility events happen far more commonly than a Bell curve predicts, hence a few lucky or unlucky outliers can skew your total. The Sharpe ratio which they used to measure risk exposure also dangerously assumes a normal distribution.

downandout 12y ago

>we don't know if the actual bitcoin exchange would fill their orders at the same prices This is the problem with many simulated trading algorithm "success" stories. No one actually knows the effect that the simulated trades would have on the real market, or how others trading against the algorithm would react to these trades. The numbers may be somewhat close to reality if the simulated trades are very small in relation to the trading volume for the asset. However, Bitcoin markets are small to begin with, which probably makes the results in this paper unreliable.

zaptheimpaler 12y ago

From the article: >Specifically, every two seconds they predicted the average price movement over the following 10 seconds. If the price movement was higher than a certain threshold, they bought a Bitcoin; if it was lower than the opposite threshold, they sold one; and if it was in-between, they did nothing. If they are indeed predicting the price at X+10 seconds at second X, they have more than enough time to act on that info without having to do HFT.

hft_throwaway 12y ago
That assumes there is no bias to their signal returns over time, which is extremely unlikely. 90% of the moves that go their way could happen in the first second, for example. Many HFTs hold positions for seconds or minutes, and predict prices out over similar time horizons, but they'll lose their best winners to competitors if they aren't fast and adept at executing.