2 ms·
Did anyone ever attempt a recursive architecture? So you take the first window or logical separatation (chapter, paragraph) and let the model summarize it into
by choeger 3y ago
Did anyone ever attempt a recursive architecture?
So you take the first window or logical separatation (chapter, paragraph) and let the model summarize it into one or two sentences. Then you repeat that with the next window (and that derived sentence as context) and create a new logical separatation out out of a fixed number of sentences. Rinse and repeat until the result fits into your window.
I have a hunch that this is somewhat how the brain works when reading.
- brrrrrm 3y agoHierarchical attention gets you close https://arxiv.org/abs/2210.05529 https://arxiv.org/abs/2210.05529