4 ms·

from scratch: https://research.myshell.ai/jetmoe https://research.myshell.ai/jetmoe

• kaibee 3y ago
  That's for an 8B model.
  • cptcobalt 3y ago
    This is over trivializing it, but there isn't much more inherent complexity in training an 8B or larger model other than more money, more compute, more data, more time. Overall, the principles are similar.
    • lostmsu 2y ago
      Assuming linear growth to number of parameters that's 7.5 figures instead of 6 for 8x22B model.