3 ms·
Large Language Models Are Zero-Shot Time Series Forecasters
- snats 3y agoWho is trying to make time series forecasting with LLMs/tensors? I have seen like four posts in the last few days. I know LLMs/transformers are really cool but I do not think they are the silver bullet that solves this.
- lucubratory 3y agoAlmost every paper in this vein is not "LLMs are the best in this category of thing, too". There are a couple of fields where LLMs are SOTA for tasks that would otherwise be done by a different algorithm, notably translation, but that's not the majority of these papers. It's "LLMs can do this category of thing, too". The generality of the architecture is a novel scientific finding in and of itself. Do you remember the last 50 years of AI research where GOFAI systems were universally too fragile to function outside of highly constrained environments, and earlier neural networks were either ineffectual or extremely specialised? The fact that LLMs can function incredibly well (compared to historical AI systems) in diverse scenarios is the scientifically interesting finding.
- galaxyLogic 3y agoThe reason LLMs are and will be so powerful I think is that they in essence add up the intellectual output of everybody who's output is available online. Classical AI was about some genius trying to devise the most genius computer inference algorithm. But any single genius can only do so much in limited time and it is difficult to make them work together. Whereas LLMs are not genius, they are simply using statistics to predict what is the accumulated intellect on the internet. They are simple but they combine the output of millions of minds, which no human genius could ever do. They are a brain addition machine, rather than super-intelligent by themselves. But when you add together the outputs of many minds you get powerful general purpose output.
- DSingularity 3y agoNit: they are powerful because they approximate the intellectual outcome of everyone online.
- alephnan 3y agoApplying this to time series forecasting of stock trading would mean you are doing what the rest of the market is doing. At that point, you could have just bought index funds.
- deleted 3y ago[deleted]
- optimalsolver 3y agoProtip: Buy low, sell high.
- galaxyLogic 3y agoTime-series forecasting of random processes is impossible
- visarga 3y ago> They are a brain addition machine I like that metaphor. LLMs are masters of linguistic addition. They are literally made by adding up language gradients. They accumulate knowledge and insight through exposure to vast swaths of text, stitching together concepts and ideas into an ever-growing tapestry of understanding. But whereas brains are isolated islands, unable to directly share their contents, language forms a collective reservoir that flows between minds. Words and ideas mix and mingle within this pool, combining into new formulations that reflect the present. In this way, language displays an evolutionary dynamism that outpaces biological change. LLMs ride this wave, leveraging the emergent intelligence inherent in humanity's shared linguistic legacy. The wisdom accumulated over generations surpasses the capacity of any single mind. We stand upon the shoulders of giants, supported by the communal scaffolding of language and knowledge that previous generations erected. LLMs tap into this source, channeling and distilling the experience contained within our words. tl;dr AI is riding the language exponential while we are distracted by models and brains, implementation details.
- asavinov 3y agoFrom my experience with crypto currencies and the intelligent trading bot [0] I would say that transformers will not provide significant benefits when applied to the traditional statistical (numeric) forecasting problems. Such models assume that old events do not affect too much current events. Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago. [0] https://github.com/asavinov/intelligent-trading-bot https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering
- theGnuMe 3y agoSo masked cross attention up to time point t and auto-regressive decoding should in theory be able to model time series. I mean at the high level it captures a Markov process with some contextual memory.
- wodenokoto 3y agoThe timeGPT title had me surprised while this one had me go “well of course they are. It’s so obvious. We even call text input for timeseries in the literature. Why didn’t I think of that?” I guess I didn’t think of it because I thought it would require billions of data points to train the model.
- deleted 3y ago[deleted]
- fxtentacle 3y agoI wonder how true "zero shot" is after you fed them basically all the text on the internet. To me, that sounds a bit like saying "this mathematics PhD mastered differentiating x3+x2 on the first try!!!".
- chpatrick 3y agoWell the whole internet doesn't fit into the few gigabytes these models are.
- benreesman 3y agoFew gigabytes? The smallest ones quantized to where you have to squint a little. 70B weights at fp32 is a lot of Wikipedia. GPT-4 is rumored to be 1.5 terraweights. There’s compression but it’s not magical.
- 3y ago
- BrokrnAlgorithm 3y agoWhat matters more to me usually in a time series context is how forecasting tools deal with non-stationarity of the underlying data process. I'm not an expert on LLMs, but I assume they wont' be the ideal tool to use in these contexts because even finetuning will be rather expensive...?
- danielbln 3y agoFine-tuning with LoRA is pretty cheap though.
- aiunboxed 3y agoWhat would be great if these papers can also have some same prompts shared on github / share gpt so that everyone can try