3 ms·

To me, this basically says "LLMs aren't pre-trained on enough 1D timeseries data" - there's a classic technique in time series analysis where you just do a wave

To me, this basically says "LLMs aren't pre-trained on enough 1D timeseries data" - there's a classic technique in time series analysis where you just do a wavelet or FFT on the time series and feed it into a convnet as an image, leveraging the massive pre-training on, e.g. ImageNet. This "shouldn't" be the best way to do it, since a giant network should learn a better internal representation than something static like FFT or a wavelet transform. But there's no 1D equivalent of ImageNet so it still often works better than a 1D ConvNet trained from scratch. Same applies here. An LLM trained on tons of time series should be able to create its own internal representation that's much more effective than looking at a static plot, since plots can't represent patterns at all scales (indeed, a human plotting to explore data will zoom in, zoom out, transform the timeseries, etc.). But since LLMs don't have enough 1D timeseries pretraining, the plot-as-image technique leverages the massive amount of image pre-training.

quantadev 2y ago
For training AI MLPs to predict time-series data that's known to have sinusoidal behaviors (which might lead to 'reasoning' like it did in LLMs) I bet it's more efficient to first curve-fit the data onto continuous data points, and then convert to frequency domain (like you said, FFT), and then do all the training using just "Frequency Domain" datasets. So then the way the AI would "predict" (run inference) would be by spitting out Frequency Domain predictions, which have to be converted back to 'time domain' to get the 'real output'. I'm sure the audio-processing AI systems out there are doing something like this already so it would be interesting to try to leverage that stuff by sending it "audio" that's actually just arbitrary time-series data rather than PCM of sound waves.
alach11 2y ago
There are some recent foundation models pre-trained on time-series data. For example TimesFM from Google. Of course, it's not directly built for classification, and it's meant for univariate datasets, so it would take some work to adapt it to these problem domains.

quantadev 2y ago

For training AI MLPs to predict time-series data that's known to have sinusoidal behaviors (which might lead to 'reasoning' like it did in LLMs) I bet it's more efficient to first curve-fit the data onto continuous data points, and then convert to frequency domain (like you said, FFT), and then do all the training using just "Frequency Domain" datasets. So then the way the AI would "predict" (run inference) would be by spitting out Frequency Domain predictions, which have to be converted back to 'time domain' to get the 'real output'. I'm sure the audio-processing AI systems out there are doing something like this already so it would be interesting to try to leverage that stuff by sending it "audio" that's actually just arbitrary time-series data rather than PCM of sound waves.

alach11 2y ago

There are some recent foundation models pre-trained on time-series data. For example TimesFM from Google. Of course, it's not directly built for classification, and it's meant for univariate datasets, so it would take some work to adapt it to these problem domains.