3 ms·
The point of work like this is to characterize in more detail the kinds of operations transformers are capable of representing. It's in the same spirit as the e
by blt 3y ago
The point of work like this is to characterize in more detail the kinds of operations transformers are capable of representing. It's in the same spirit as the efforts to figure out the classes of formal languages that transformers can recognize. Nobody is actually suggesting to replace LR parsers or Kalman filters with transformers in applications where those tools are already working well.
Think of what would happen if the result were negative. Like, what if the size of transformer needed to represent the KF grows exponentially with the dimension of the linear system. That would certainly cast doubt on the prospect of using transformers for filtering-like problems. It might also suggest changes to the transformer architecture to fix the issue.
Since the result is positive, our belief that transformers are reasonable for filtering-like problems is strengthened a bit.
- techbro92 3y ago[flagged]
- behnamoh 3y agoThis.
- bluish29 3y agoHave you looked into the authors? None of them are grad students
- terminous 3y agoFirst author is a postdoc, second author is director of the institute and probably gets last author even if they did no work. The same criticism applies, even moreso for a postdoc than a grad student. They're probably under huge publish or perish pressure to get a faculty job soon. I'm so glad I left that racket.
- fragmede 3y ago> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- gautamcgoel 3y agoAuthor here: couldn't have put it better myself! Yes, the result isn't super surprising, but it helps strengthen the case for using Transformers for signal processing and control. Also gives practical guidance on how such filters should be constructed (for example, how the embeddings should be chosen).