3 ms·
Aren't LLM's already zero shot time series predictors? Predicting next token and forecasting seem like the exact same problem. I will admit some small tweaks in
by Difwif 3y ago
Aren't LLM's already zero shot time series predictors? Predicting next token and forecasting seem like the exact same problem. I will admit some small tweaks in tokenization could help but it seems like we're just pretraining on a different dataset.
One idea I was interested in was after reading the paper on introducing pause tokens[1] was a multimodal architecture that generalizes everything to parallel time series streams of tokens in different modalities. Pause tokens make even more sense in that setup.
1. https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226
-Arm chair ML practitioner
- beernet 3y agoThat is technically true, however, they are two very different problems from a domain/business perspective
- bob1029 3y agoI agree - You could frame LLMs this way. Tokens over "time" where time just happens to be represented by discrete, sequential memory. Each token could encode a specific amplitude for the signal. You could literally just have tokens [0,1,...,MAX_AMPLITUDE] and map your input signal to this range. In the most extreme case, you could have 2 tokens - zero and one. This is the scheme used in DSD audio. The only tradeoff is that you need way more samples per unit time to represent the same amount of information, but there are probably some elegant perf hacks for having only 2 states to represent per sample. There are probably a lot of variations on the theme where you can "resample" the input sequences to different token rate vs bits per token arrangements.