3 ms·
I'm surprised the transformer model scales here. In NLP the amount of self attention you need is like a few paragraphs maybe, but with time series you'd need to
by daxfohl 3y ago
I'm surprised the transformer model scales here. In NLP the amount of self attention you need is like a few paragraphs maybe, but with time series you'd need to capture millions of data points in the attention blocks in order to correlate with historical trends and anomalous events like Black Fridays or seizures, etc, right? Otherwise what's the point of using transformers?
- jsenn 3y agoCheck the article. They have a learned preprocessing step that translates time slices containing multiple data points into tokens, so the transformer is actually predicting larger chunks of time rather than individual time points.
- daxfohl 3y agoOh, right. I missed the point of that step on first read.