3 ms·
Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that t
by kqr 1y ago
Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital.
Although I lack the maths to determine it numerically (depends on volatility etc.), it looks to me as though all six are overbetting and would be ruined in the long run. It would have been interesting to compare against a constant fraction portfolio that maintains 1/6 in each asset, as closely as possible while optimising for fees. (Or even better, Cover's universal portfolio, seeded with joint returns from the recent past.)
I couldn't resist starting to look into it. With no costs and no leverage, the hourly rebalanced portfolio just barely outperforms 4/6 coins in the period: https://i.xkqr.org/cfportfolio-vs-6.png https://i.xkqr.org/cfportfolio-vs-6.png. I suspect costs would eat up many of the benefits of rebalancing at this timescale.
This is not too surprising, given the similiarity of coin returns. The mean pairwise correlation is 0.8, the lowest is 0.68. Not particularly good for diversification returns. https://i.xkqr.org/coinscatter.png https://i.xkqr.org/coinscatter.png
> difficulty executing against self-authored plans as state evolves
This is indeed also what I've found trying to make LLMs play text adventures. Even when given a fair bit of help in the prompt, they lose track of the overall goal and find some niche corner to explore very patiently, but ultimately fruitlessly.
- falcor84 1y agoAgreed, and I'd also love to see a baseline of human performance here, both of experienced quant traders and of fresh grads who know the theory but never did this sort of trading and aren't familiar with the crypto futures market.
- spaceman_2020 1y agoAs someone who trades crypto semi-professionally, this was one of the toughest trading periods I've ever seen and included a massive liquidation event on 10th of October that wiped out over $20B in capital. Any trader who broke even in this period likely outperformed. I know some very, very good traders who got wiped out on leverage on 10th of October when stop losses didn't trigger and prices plummetted to 2021 levels (still no clarity why). BTC also performed abysmally during this period with a sustained chop down from $126k to $90k.
- kqr 1y agoNote that 10th of October is before the trading period in this experiment. If anything, autoregression over shorter timescales would suggest entering after 10th of October being a good idea!
- spaceman_2020 11mo agoJust noticed that it was after October 10th. I'll still say that the trading period after October 10th has been brutally choppy. Only now do we have a clear direction (down) where you can at least short with some confidence
- fragmede 1y ago> find some niche corner to explore very patiently, but ultimately fruitlessly. What, so they're better at my hobbies than me? Someone give Claude a 3d printer!
- michaelcampbell 11mo agoLLM's know the WORDS of "The market can remain irrational longer than you can remain solvent", but not the meaning.
- ponector 11mo ago>> That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital LLM indeed can replace average human being.
- pants2 11mo agoWell, if you can get a model to consistently lose money like that, then you just trade the opposite of what it says and you're guaranteed money!
- kqr 11mo agoThanks to the magic of compounding, inverting overbetting also leads to overbetting. Especially once costs are accounted for.