6 ms·
A decoder-only foundation model for time-series forecasting
- djoldman 3y agoFYI: lately there has been very interesting univariate time series forecasting work mostly by Albert Gu using state space models. https://arxiv.org/abs/2111.00396 https://arxiv.org/abs/2111.00396 https://arxiv.org/pdf/2111.00396.pdf https://arxiv.org/pdf/2111.00396.pdf
- daxfohl 3y agoAre there any multimodal models where you can annotate your time series with events in natural language, and have it identify other occurrences of similar events? Or where you can ask it to do a forecast assuming some new event, also in natural language? I feel like this is what I'd want out of a DL time series NN. For just prediction it seems like overkill.
- jerpint 3y agoYou could probably annotate a graph , export it as an image and ask GPT4V for its best guesses, I suspect it’ll work decently well for simpler scenarios
- daxfohl 3y agoI'm surprised the transformer model scales here. In NLP the amount of self attention you need is like a few paragraphs maybe, but with time series you'd need to capture millions of data points in the attention blocks in order to correlate with historical trends and anomalous events like Black Fridays or seizures, etc, right? Otherwise what's the point of using transformers?
- jsenn 3y agoCheck the article. They have a learned preprocessing step that translates time slices containing multiple data points into tokens, so the transformer is actually predicting larger chunks of time rather than individual time points.
- daxfohl 3y agoOh, right. I missed the point of that step on first read.
- mistrial9 3y agodecoder-only models from a giant corp. which only they can change. uh.. then a read-only API is installed into Android stack without options? Where is this going?
- GeoffNN 3y agoThey say they can optionally use some time-dependent covariates (derived from the datetime) -- did anyone figure out if/how they are actually used in this model from the paper?
- thatcherc 3y agoSounds like a cool model. I'd love to try it but it seems like they're not releasing it (yet?). I've really gotten spoiled with the recent language models where I can download and run any new model or fine-tune I hear about. It's gotten to the point where I don't feel a model is very relevant unless I can run it locally. Hopefully a local version of this becomes available because I have plenty of time series data I'd like to run through it!
- deleted 3y ago[deleted]
- gardnr 3y agoWhat time series data do you have that you'd like to run through it?
- ctrw 3y agoMine of your business. That's the point of open source.
- mordechai9000 3y agoStock market data. But maybe the advantage disappears if other traders have the same model. Perhaps that's why they haven't released it.
- bethekind 3y agoI've tried using TimeGPT to predict the SPDR future price and it couldn't make any useful predictions Maybe this one is better I still think humans have better pattern recognition in the stock market than neural nets...at least for now
- adastra22 3y agoHumans haven’t been in the loop for about two decades now. High frequency quants just use faster, lower latency models than AI stuff.
- glial 3y agoThis is cool, but I'm amused that CatBoost still beats it.
- shiandow 3y agoThe fact that the mean absolute error doesn't even change all that much from ARIMA suggests to me that there's no magic here. Though mean absolute error is a bit of a weird measure. They're basically testing how closely a model predicts the median.
- naijaboiler 3y agoI’m not sure that’s what MAE measures. What’s the median of a time series? What does that even mean?
- shiandow 3y agoThe MAE is minimized by the median. I share your confusion as for why one would use it for a time series. Clearly they had some reason to pick it over others, but I couldn't tell you why. Especially since ARIMA effectively minimizes RMS error.
- naijaboiler 3y agoBet minute feature engineering + glm beats it comfortably
- old_bayes 3y agoFraction of the cost, way more interpretable. Still, I suppose we gotta keep an eye on this kind of work in case there's a tipping point.
- snats 3y agoA little bit of context. Basically, all of tabular deep learning has been stuck and SOTA has been tree based algorithms like Catboost and XGBoost. This seems like a big step forward towards getting a generalizable deep learning model besides this "tree models"
- mistrial9 3y agoCatboost and XGBoost do not need to be replaced
- spyder 3y agoYea, and from this paper it looks like the tree model (Catboost supervised) is still beating their (zero-shot) performance and they don't show their supervised version yet, but they write at the end that they plan to do it in a future work. Will be interesting to see.
- ipsum2 3y agoThis isn't for tabular data though? Time series transformers have been around for a few years now, see Transformers in Time Series: A Survey https://arxiv.org/abs/2202.07125 https://arxiv.org/abs/2202.07125 and Are Transformers Effective for Time Series Forecasting? https://arxiv.org/abs/2205.13504 https://arxiv.org/abs/2205.13504
- j7ake 3y agoIt's interesting how they say that their model has high "zero-shot" performance, even though it was trained on millions of examples. To me, zero-shot would mean a model whose parameters was never tuned through training examples at all...
- p1esk 3y agoZero shot means a model trained one one dataset, or for one task, performs well on another dataset, or on another task - without any finetuning.
- adastra22 3y agoThanks for the clarification, because that is NOT what zero shot used to mean in AI.
- whiplash451 3y agoWhat did it use to mean?
- adastra22 3y agoBasically the ability to perform a task on the first try, without any feedback. E.g. a robot that is trained to pick up blocks being told to shoot a basketball though a hoop, and making a reasonable attempt at doing so on the first try (doesn't have to be successful, but has to in some way resemble throwing a ball through a hoop rather than picking up blocks). I can see how the machine learning definition is similar, but the whole idea of retraining and fine-tuning is foreign to a lot of subfields in GOFAI.
- optimalsolver 3y agoSo a model trained on one task, performing reasonably well on an related task?
- neodypsis 3y ago> "Synthetic data helps with the basics. Meaningful synthetic time-series data can be generated using statistical models or physical simulations. These basic temporal patterns can teach the model the grammar of time series forecasting." Can someone elaborate on what a grammar means in the context of time series forecasting?
- WanderPanda 3y agoThey probably mean the manifold of the universe
- mnoronha 3y agoah yes, the manifold of the universe (god) :)
- alextheparrot 3y agoYou could probably consider learning a sign wave to be a “grammar” related to periodic variations. Grammar in this context feels like “What are the core conceptual heuristics that help guide towards faster understanding”
- remontoire 3y agoProbably drawing an analogy to how causal pretrained models go through stages of understanding language, words -> grammar -> meaning. Gwen mentions this experience when training character level RNNs. https://gwern.net/scaling-hypothesis#why-does-pretraining-work https://gwern.net/scaling-hypothesis#why-does-pretraining-wo...
- horacemorace 3y agoTotally. Also basic temporal scale or cyclic properties. It’s kind of mind blowing that the shape of most recorded human patterns is reducible in this way.
- nerdponx 3y agoIt's a bit of an anthropomorphism ation. I don't believe it has any formal meaning here. The idea is that there are certain kinds of underlying signals and patterns which are common to a wide range of time series data. So if a model is able to learn those signals and patterns, it can look at any time series and, with enough historical data, predict future data, without actually updating any model weights. Those signals constitute a "grammar" of sorts.
- frogamel 3y agoThe research in this space is very conflicting about what methods actually work. In the graph on the page, the ETS model (basically just a weighted moving average) outperforms multiple, recent deep learning models. But the papers for those models claim they outperform ETS and other basic methods by quite a bit. You can find recent papers from researchers about how their new transformers model is the best and SOTA, papers which claim transformers is garbage for time series and claim their own MLP variant is SOTA, other papers which claim deep learning in general underperforms compared to xgboost/lightgbm, etc. Realistically I think time series is incredibly diverse, and results are going to be highly dependent on which dataset was cherry-picked for benchmarking. IMO this is why the idea of a time series foundation model is fundamentally flawed - transfer learning is the reason why foundation models work in language models, but most time series are overwhelmingly noise and don't provide enough context to figure out what information is actually transferrable between different time series.
- jimmySixDOF 3y agoAlso a big difference between applications so anomaly detection has a wider set of working solutions than prediction if such a thing is even possible
- nerdponx 3y agoI feel the same way for the most part. However I cam imagine a kind of meta-learning foundation model that basically has a huge internal library of micro-features, and when you put a sequence into it, it matches those features against the sequence and builds up a low-noise summary of the data that it can use to make predictions. That's of course heavily anthropomorphized, but it seems potentially in-scope for a transformer model. The real problem with time series data is that you can't predict the future. Images and text are relatively homogeneous and exist within a kind of restricted space. "Time series" in general however could be just about anything, and there's not as much reason to believe that something like a "grammar of time series" even exists beyond what we already can do with STL etc.
- two_in_one 3y ago> incredibly diverse, and results are going to be highly dependent on which dataset was cherry-picked for benchmarking This naturally comes to multi-model solution under one umbrella. Sort of MoE, with selector (router, classifier) and specialized experts. If there is something which can't be handled by existing experts then train another one.
- hackerlight 3y agoAre they ever going to release the weights?
- ashvardanian 3y agoI like tabular ML, but very skeptical about this research direction. Language models trained on a broad corpus make sense, as the language is similar across different domains. Time-series, however, are extremely different... Stock prices, heart-rates, brain-waves, digital signals... Patterns learned on a broad dataset should only introduce uninterpretable noise.
- Smith42 3y agoIf you are interested in this also check out EarthPT, which is also a time series decoding transformer (and has the code and weights released under the MIT licence): https://arxiv.org/abs/2309.07207 https://arxiv.org/abs/2309.07207
- alexey-salmin 3y agoI'm not sure a zero-shot foundational model for time series makes sense at all: if you only look at the numbers it's the "continue the sequence" game, nowhere to apply your "foundational knowledge" really. A one-shot model that takes a prompt like "this sequence is a voltage of a PV-cell every hour" will have a chance though.
- hackerlight 3y ago> A one-shot model that takes a prompt like "this sequence is a voltage of a PV-cell every hour" will have a chance though. Such information will somewhat be embedded in the network (a new voltage graph will look similar to the voltage graphs it's seen before) but it should be better to make it explicit.
- mistrial9 3y agoyou are right, but being right is not enough! For a lot of real-world cases, the automation is worth it, despite information problems. to be clear, I share these concerns