3 ms·
People at Microsoft Research certainly do. But the bet is that, with scale, these limitations won't matter in practice. For example Transformers can't recogniz
by canjobear 2y ago
People at Microsoft Research certainly do. But the bet is that, with scale, these limitations won't matter in practice. For example Transformers can't recognize well-nested brackets (like {[()]}) to infinite depth; the depth that Transformers can recognize is limited by how many self-attention layers they have. But in practice, you rarely need much depth.
- toxik 2y agoNotably, humans also cannot track this to infinite depth. Somehow we know this and reach for algorithmic solutions (like pen and paper).
- logicchains 2y agoInterestingly LLMs also do better with pen and paper reasoning: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923 .