3 ms·
Was really cool to see yous use Engrams to cut down compute! Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you
by minimaltom 2mo ago
Was really cool to see yous use Engrams to cut down compute!
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
- HenryNdubuaku 2mo agoYes, we ablated Engrams rigorously and found that it returned world knowledge like FFN without without compute expenditure.
- minimaltom 2mo agoWhat about mHC? I'm surprised it helped with such a small compute budget.
- HenryNdubuaku 2mo ago[flagged]