4 ms·

HMT: Hierarchical Memory Transformer for Long Context Language Processing

87 points by jasondavies 2y ago

cs702 2y ago
Code: https://github.com/OswaldHe/HMT-pytorch https://github.com/OswaldHe/HMT-pytorch This looks really interesting. I've added the paper to my reading list and look forward to playing with the code. I'm curious to see what kinds of improvements we can get by agumenting Transformers and other generative sequence models with this and other mechanisms implementing hierarchical memory.[a] Shouldn't the authors cite the work by Jeff Hawkins et al at Numenta? Hawkins has been proposing AI models with hierarchical temporal memory for a long time.[b] I can't help but wonder if there is a way, somehow, to incorporate his work and ideas in Transformers and other generative sequence models. We sure live in interesting times! --- [a] In the past, I've experimented with mechanisms that add memory to Transformers, but never with hierarchy. [b] https://en.wikipedia.org/wiki/Hierarchical_temporal_memory https://en.wikipedia.org/wiki/Hierarchical_temporal_memory
- anthonyskipper 2y ago
  I thought Hawkins's book "On Intelligence" was amazing. It's a bit wild how close things have followed to the direction he laid out.
  - rawrawrawrr 2y ago
    That's kinda hilarious, because I think the book was exactly wrong in its predictions. This can be evidenced by the continuous failures of his AI company, Numenta.
    - anthonyskipper 2y ago
      Well both things can be true at the same time. I agree Numenta missed the boat, but it doesn't mean that book wasn't prescient. Numenta just didn't get there first, wasn't a fast follower, and blew a huge lead in AI. It may still end up the HTM is one of the final state solution,s but they are so far from ever being able to capitalize on it that it is unlikely to even matter if they invented the concept.
- 3abiton 2y ago
  Does it relate to applications in time series? How does the hierarchy play a role in Transformers?
  - cs702 2y ago
    Not really. But questions about the work should be asked to the authors. I've only skimmed the paper.

87 points by jasondavies 2y ago

Code: https://github.com/OswaldHe/HMT-pytorch https://github.com/OswaldHe/HMT-pytorch This looks really interesting. I've added the paper to my reading list and look forward to playing with the code. I'm curious to see what kinds of improvements we can get by agumenting Transformers and other generative sequence models with this and other mechanisms implementing hierarchical memory.[a] Shouldn't the authors cite the work by Jeff Hawkins et al at Numenta? Hawkins has been proposing AI models with hierarchical temporal memory for a long time.[b] I can't help but wonder if there is a way, somehow, to incorporate his work and ideas in Transformers and other generative sequence models. We sure live in interesting times! --- [a] In the past, I've experimented with mechanisms that add memory to Transformers, but never with hierarchy. [b] https://en.wikipedia.org/wiki/Hierarchical_temporal_memory https://en.wikipedia.org/wiki/Hierarchical_temporal_memory

anthonyskipper 2y ago
I thought Hawkins's book "On Intelligence" was amazing. It's a bit wild how close things have followed to the direction he laid out.
- rawrawrawrr 2y ago
  That's kinda hilarious, because I think the book was exactly wrong in its predictions. This can be evidenced by the continuous failures of his AI company, Numenta.
  - anthonyskipper 2y ago
    Well both things can be true at the same time. I agree Numenta missed the boat, but it doesn't mean that book wasn't prescient. Numenta just didn't get there first, wasn't a fast follower, and blew a huge lead in AI. It may still end up the HTM is one of the final state solution,s but they are so far from ever being able to capitalize on it that it is unlikely to even matter if they invented the concept.
3abiton 2y ago
Does it relate to applications in time series? How does the hierarchy play a role in Transformers?
- cs702 2y ago
  Not really. But questions about the work should be asked to the authors. I've only skimmed the paper.