2 ms·
There are 32 layers (numbered from 0), which is... almost how many the model has. Actually it's got 33, but maybe one of them is before or after the separate ex
by sp332 3y ago
There are 32 layers (numbered from 0), which is... almost how many the model has. Actually it's got 33, but maybe one of them is before or after the separate experts?
- ajtejankar 3y agoThe base model has 32 layers and there is a single linear layer for language modeling (going from embeddings to the vocabulary) that gets applied at the very end.