3 ms·
Do we know if any of these techniques are actually used in the so-called "frontier" models?
by mrtesthah 1y ago
Do we know if any of these techniques are actually used in the so-called "frontier" models?
- vinithavn01 1y agoThe model names are mentioned under each type of attention mechanism
- gchadwick 1y agoWho knows what the closed source models use but certainly going by what's happening in open models all the big changes and corresponding gains in capability are in training techniques not model architecture. Things like GQA and MLA as discussed in this article are important techniques for getting better scaling but are relatively minor tweak vs the evolution in training techniques. I suspect closed models aren't doing anything too radically different from what's presented here.
- zackangelo 1y agoNot quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.