5 ms·
Who is trying to make time series forecasting with LLMs/tensors? I have seen like four posts in the last few days. I know LLMs/transformers are really cool but
by snats 3y ago
Who is trying to make time series forecasting with LLMs/tensors? I have seen like four posts in the last few days. I know LLMs/transformers are really cool but I do not think they are the silver bullet that solves this.
- lucubratory 3y agoAlmost every paper in this vein is not "LLMs are the best in this category of thing, too". There are a couple of fields where LLMs are SOTA for tasks that would otherwise be done by a different algorithm, notably translation, but that's not the majority of these papers. It's "LLMs can do this category of thing, too". The generality of the architecture is a novel scientific finding in and of itself. Do you remember the last 50 years of AI research where GOFAI systems were universally too fragile to function outside of highly constrained environments, and earlier neural networks were either ineffectual or extremely specialised? The fact that LLMs can function incredibly well (compared to historical AI systems) in diverse scenarios is the scientifically interesting finding.
- galaxyLogic 3y agoThe reason LLMs are and will be so powerful I think is that they in essence add up the intellectual output of everybody who's output is available online. Classical AI was about some genius trying to devise the most genius computer inference algorithm. But any single genius can only do so much in limited time and it is difficult to make them work together. Whereas LLMs are not genius, they are simply using statistics to predict what is the accumulated intellect on the internet. They are simple but they combine the output of millions of minds, which no human genius could ever do. They are a brain addition machine, rather than super-intelligent by themselves. But when you add together the outputs of many minds you get powerful general purpose output.
- DSingularity 3y agoNit: they are powerful because they approximate the intellectual outcome of everyone online.
- alephnan 3y agoApplying this to time series forecasting of stock trading would mean you are doing what the rest of the market is doing. At that point, you could have just bought index funds.
- deleted 3y ago[deleted]
- optimalsolver 3y agoProtip: Buy low, sell high.
- galaxyLogic 3y agoTime-series forecasting of random processes is impossible
- visarga 3y ago> They are a brain addition machine I like that metaphor. LLMs are masters of linguistic addition. They are literally made by adding up language gradients. They accumulate knowledge and insight through exposure to vast swaths of text, stitching together concepts and ideas into an ever-growing tapestry of understanding. But whereas brains are isolated islands, unable to directly share their contents, language forms a collective reservoir that flows between minds. Words and ideas mix and mingle within this pool, combining into new formulations that reflect the present. In this way, language displays an evolutionary dynamism that outpaces biological change. LLMs ride this wave, leveraging the emergent intelligence inherent in humanity's shared linguistic legacy. The wisdom accumulated over generations surpasses the capacity of any single mind. We stand upon the shoulders of giants, supported by the communal scaffolding of language and knowledge that previous generations erected. LLMs tap into this source, channeling and distilling the experience contained within our words. tl;dr AI is riding the language exponential while we are distracted by models and brains, implementation details.
- cicce19 3y agoThis is a really elegantly worded perspective
- layla5alive 3y agoThey likely watched wheel of time recently...
- noduerme 3y agoCool, so they'll be able to project the future time-series of the S&P 500 as well as the average of all the analysts in the world combined? What startup can I buy part of that's most adept at selling that bullshit to investors as SaS to most rapidly separate gamblers from their cash? /s
- czbond 3y agoOne comment that made me think on HN a while back is something akin to "The average IQ is between 85 and 115, therefore AI will be akin to friends with IQ's between 85 and 115 giving you advice". So not that useful?
- Retric 3y agoLower IQ people are likely under represented online. Accurate information is also over represented because you can be wrong in far more ways than you can be correct. In other words the wisdom of the crowd is in part the 10% of people that know X all match each other while the other 90% is all over the place.
- swid 3y agoIf the AI can distinguish between good and bad reasoning, it can select training material that has a higher average than than the corpus. If it can output data that covers the range, it can generate its own data to bootstrap higher functioning models (like alpha go playing against itself to achieve superhuman go playing skills). I also don’t think you need to remove the dumb comments from the training material anyway. The AI could be able to simulate either someone of 80 IQ or 180 IQ, producing output consistent with both personas.
- galaxyLogic 3y agoIt's not so much about IQ but the amount of knowledge in the system. And ignorant person could have a very high IQ still be very ignorant. LLMs may not be "smart" but they know a lot.
- asavinov 3y agoFrom my experience with crypto currencies and the intelligent trading bot [0] I would say that transformers will not provide significant benefits when applied to the traditional statistical (numeric) forecasting problems. Such models assume that old events do not affect too much current events. Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago. [0] https://github.com/asavinov/intelligent-trading-bot https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering
- theGnuMe 3y agoSo masked cross attention up to time point t and auto-regressive decoding should in theory be able to model time series. I mean at the high level it captures a Markov process with some contextual memory.