3 ms·
FYI Mistral at launch just dropped the weights without any model architecture mentioned. Most of the OSS models follow the same architecture which is Llama +-
by ismailmaj 20d ago
FYI Mistral at launch just dropped the weights without any model architecture mentioned.
Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work.
- anuj0456 20d agoyes, most of them are similar. but implementation of GQA, MLA, mHC, Sliding Window changes the implementation drastically because of which the overall model effeciency changes.
- stymaar 20d ago> Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work. It used to be the case until last year, but now almost every Chinese model come with their own linear attention mechanism.