michael_lutz

4 ms·

1. 1.▲
   by michael_lutz 1y ago

   That's a really interesting question, and it's one I'd love to answer in a future work. This blog mostly focuses on characterizing context depth limits.
2. 2.▲
   More Layers Unlock 2^N Transformer Context Depth with Divide and Conquer(ml-mike.com)

   5 points by michael_lutz 1y ago|3 comments
3. 3.▲
   by michael_lutz 1y ago

   Context windows are now 1M+ tokens, but context depth is limited. Often, the answer is hidden behind layers of linked information, but an attention block can only resolve one link at a time. We trained a tiny 5 layer model that beats GPT-4.