5 ms·

Isn't it pretty well known in the finance world that using stale public information to predict the market is a fool's errand? Unless you have some kind of spec

Isn't it pretty well known in the finance world that using stale public information to predict the market is a fool's errand? Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out), trying to predict the market with historical data does worse than "Just give me some monkeys, darts and a dart board".

minimaxir 6y ago
Granted, a typical Kaggle metagame-that-is-technically-against-the-rules is to use data from outside the dataset, which is one of the reasons winners have to be validated.
- usmannk 6y ago
  Generally this is allowed if you publish the data you're bringing in. They even create a sponsored thread for it in most competitions.
- justjonathan 6y ago
  From: kaggle.com/c/jane-street-market-prediction/overview/code-requirements "Freely & publicly available external data is allowed, including pre-trained models"
cultus 6y ago
I'm sure they've got more/better data in production. There seems to be some arbitrage that can be shaved off the edges for players with innovative enough strategies and good and timely enough data.
logicslave 6y ago
Theres actually still money to be made in small scale strategies. Sophisticated funds are running billions. They cant focus on strategies that only work for 100-500k. This is where big returns can be made. Even warren buffet will say, if he was only managing 1 million, he would get 100% a year returns.
- xapata 6y ago
  That doesn't make sense unless there are very few viable small scale strategies, at which point they'd probably be difficult to identify. Your assertion might have been true before computers were able to help someone manage many strategies simultaneously.
  - logicslave 6y ago
    No it makes a ton of sense. 100k is too small to have a researcher focus on full time. His compensation is probably 500k or more. Then add in fees, infra, cost of regulations, etc and small time strategies arent developed. Making money in the market is completely overrated from a difficulty perspective, the hard part is managing billions.
    - xapata 6y ago
      You're saying it'd require a salary of $500k or more to find someone competent enough to manage an investment opportunity worth $100k annually? That implies the investment opportunities are actually quite complicated. Obviously, opportunities exist for investment gains at all magnitudes. The question was whether they're easy enough for the average Jolanda. If Jolanda would require $500k annual salary to do the work, then, that easy opportunity doesn't really exist at all. Alternately, if Jolanda doesn't need to make this opportunity her full-time job, then the investment company can pay her $500k to manage many of these small opportunities. After all, there must be many opportunities. Otherwise only 1 of the HN readers would be able to exploit it. In which case, again, it doesn't really exist at all.
      - logicslave 6y ago
        Honestly, you seem like youre trying to reason this out with zero experience in the field. You cant just "run statistics to identify strategies". I'm not going to go into it further because you know nothing
        6y ago
smabie 6y ago
no not really. you can use public information to give you an edge. And I say this as a person who trades and develops models at a vety successful market making firm. Alternative data no one else can get easily certainly has tremendous value though. Of course, predicting one or two seconds into the future (my primary concern) is easier than days or years, so there's that.
Tinyyy 6y ago
That’s not necessarily true.
2-tpg 6y ago
Using purely historical price data it is harrowingly difficult. There are 130 anonymized features, so that's unlikely to be only price data. It could include information on the order book, correlated assets, fundamentals, vectorized/embedded text, etc. Besides, I bet you can train monkeys to do (slightly) better than blindfolded random throwing. Even with public data (replace satellite images with Youtube mentions, or number of links moving into a company website) it is very possible to do better than average guessing on quite a lot of assets (especially smaller and newer markets). Most hedge funds, even with specialized expensive non-public data, are not magical unicorns. Their quants really may just run a gradient boosting machine and leave it at that. Some hedge funds even prefer linear methods, because this lowers risk through lower variance. Such models can be beaten by experienced Kagglers for sure. For one, I did.
- Traster 6y ago
  One thing we need to be clear about is that you're not aiming to be better than average. You're aiming to make a profit. There are probably hundreds of thousands of day traders, there are probably <100 market makers and tradingfirms (far less than that for a some specific products) and you'll probably find 99% of the day traders aren't making systematic profits. There are lots of strategies that are much better than average and still worse than putting your cash in a bank.
  - 2-tpg 6y ago
    You can aim for both. If you just aim for profit, then you can get lucky with just average, or even random, betting. If you find a weighted coinflip (which is not impossible), provided by how many times you can flip that coin, you will see steady systematic profits. Of course, majority of day traders are getting owned by the big players, and they would do better doing more reasoned and long-term investments. Most day traders are not even using predictive models though.
    - Traster 6y ago
      On that point - it's pretty clear that this Kaggle competition is highly likely to result in a decent number of submissions that make more money through luck, than other make through strategy.
blhack 6y ago
>Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out) Planet labs will sell you all of that data, in case people reading along here are curious.
hogFeast 6y ago
None of that information is non-public (you can find cargo ship data online for free), none of it is particularly valuable (you are looking for information, it is hard to know how much information is in cargo ship movement...it depends), and most non-quant hedge funds have been doing stuff like this for decades (i.e. hiring people to stand outside a retailer's stores and count customers)...most of this stuff is less useful than people think (again, you need information, data with intent). Also, most of this stuff isn't in the price. Lots of people are collecting new data, it is definitely becoming more widespread but the actual synthesis is tricky (most people who are quants do not understand fundamentals, and most fundamental analysis don't understand data...most firms are swirling in a perfect storm of ignorance).

minimaxir 6y ago

Granted, a typical Kaggle metagame-that-is-technically-against-the-rules is to use data from outside the dataset, which is one of the reasons winners have to be validated.

usmannk 6y ago
Generally this is allowed if you publish the data you're bringing in. They even create a sponsored thread for it in most competitions.
justjonathan 6y ago
From: kaggle.com/c/jane-street-market-prediction/overview/code-requirements "Freely & publicly available external data is allowed, including pre-trained models"

cultus 6y ago

I'm sure they've got more/better data in production. There seems to be some arbitrage that can be shaved off the edges for players with innovative enough strategies and good and timely enough data.

logicslave 6y ago

Theres actually still money to be made in small scale strategies. Sophisticated funds are running billions. They cant focus on strategies that only work for 100-500k. This is where big returns can be made. Even warren buffet will say, if he was only managing 1 million, he would get 100% a year returns.

xapata 6y ago
That doesn't make sense unless there are very few viable small scale strategies, at which point they'd probably be difficult to identify. Your assertion might have been true before computers were able to help someone manage many strategies simultaneously.
- logicslave 6y ago
  No it makes a ton of sense. 100k is too small to have a researcher focus on full time. His compensation is probably 500k or more. Then add in fees, infra, cost of regulations, etc and small time strategies arent developed. Making money in the market is completely overrated from a difficulty perspective, the hard part is managing billions.
  - xapata 6y ago
    You're saying it'd require a salary of $500k or more to find someone competent enough to manage an investment opportunity worth $100k annually? That implies the investment opportunities are actually quite complicated. Obviously, opportunities exist for investment gains at all magnitudes. The question was whether they're easy enough for the average Jolanda. If Jolanda would require $500k annual salary to do the work, then, that easy opportunity doesn't really exist at all. Alternately, if Jolanda doesn't need to make this opportunity her full-time job, then the investment company can pay her $500k to manage many of these small opportunities. After all, there must be many opportunities. Otherwise only 1 of the HN readers would be able to exploit it. In which case, again, it doesn't really exist at all.
    - logicslave 6y ago
      Honestly, you seem like youre trying to reason this out with zero experience in the field. You cant just "run statistics to identify strategies". I'm not going to go into it further because you know nothing
      - 6y ago

smabie 6y ago

no not really. you can use public information to give you an edge. And I say this as a person who trades and develops models at a vety successful market making firm. Alternative data no one else can get easily certainly has tremendous value though. Of course, predicting one or two seconds into the future (my primary concern) is easier than days or years, so there's that.

Tinyyy 6y ago

That’s not necessarily true.

2-tpg 6y ago

Using purely historical price data it is harrowingly difficult. There are 130 anonymized features, so that's unlikely to be only price data. It could include information on the order book, correlated assets, fundamentals, vectorized/embedded text, etc. Besides, I bet you can train monkeys to do (slightly) better than blindfolded random throwing. Even with public data (replace satellite images with Youtube mentions, or number of links moving into a company website) it is very possible to do better than average guessing on quite a lot of assets (especially smaller and newer markets). Most hedge funds, even with specialized expensive non-public data, are not magical unicorns. Their quants really may just run a gradient boosting machine and leave it at that. Some hedge funds even prefer linear methods, because this lowers risk through lower variance. Such models can be beaten by experienced Kagglers for sure. For one, I did.

Traster 6y ago
One thing we need to be clear about is that you're not aiming to be better than average. You're aiming to make a profit. There are probably hundreds of thousands of day traders, there are probably <100 market makers and tradingfirms (far less than that for a some specific products) and you'll probably find 99% of the day traders aren't making systematic profits. There are lots of strategies that are much better than average and still worse than putting your cash in a bank.
- 2-tpg 6y ago
  You can aim for both. If you just aim for profit, then you can get lucky with just average, or even random, betting. If you find a weighted coinflip (which is not impossible), provided by how many times you can flip that coin, you will see steady systematic profits. Of course, majority of day traders are getting owned by the big players, and they would do better doing more reasoned and long-term investments. Most day traders are not even using predictive models though.
  - Traster 6y ago
    On that point - it's pretty clear that this Kaggle competition is highly likely to result in a decent number of submissions that make more money through luck, than other make through strategy.

blhack 6y ago

>Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out) Planet labs will sell you all of that data, in case people reading along here are curious.

hogFeast 6y ago

None of that information is non-public (you can find cargo ship data online for free), none of it is particularly valuable (you are looking for information, it is hard to know how much information is in cargo ship movement...it depends), and most non-quant hedge funds have been doing stuff like this for decades (i.e. hiring people to stand outside a retailer's stores and count customers)...most of this stuff is less useful than people think (again, you need information, data with intent). Also, most of this stuff isn't in the price. Lots of people are collecting new data, it is definitely becoming more widespread but the actual synthesis is tricky (most people who are quants do not understand fundamentals, and most fundamental analysis don't understand data...most firms are swirling in a perfect storm of ignorance).