3 ms·

I don't understand why there are 31 layers in this tool. I don't understand why we only see 4 charts (I would expect to see 8, one for each 7B mistral tiny mod

• antirez 3y ago
  Can't visualize the graphs well from mobile but the many layers are there because MoE models select a different expert at each layer, not just at each token.
• sp332 3y ago
  There are 32 layers (numbered from 0), which is... almost how many the model has. Actually it's got 33, but maybe one of them is before or after the separate experts?
  • ajtejankar 3y ago
    The base model has 32 layers and there is a single linear layer for language modeling (going from embeddings to the vocabulary) that gets applied at the very end.
• ajtejankar 3y ago
  The plots show 2 dimensional projection of the 8 dimensional feature vector of each paragraph. So, x and y axis are linear combination of 8 different experts. Ideally, all of this should be in a single plot but there was a lot of overlap between different sub-categories and it was hard to see. So, I thought of separating them by their broad categories. Also, there are 32 layers in the model, each has 8 different experts, and 2 of them picked for each token.