4 ms·

for finetuning or parameter training from scratch?

• kiney 3y ago
  from scratch: https://research.myshell.ai/jetmoe https://research.myshell.ai/jetmoe
  • kaibee 3y ago
    That's for an 8B model.
    • cptcobalt 3y ago
      This is over trivializing it, but there isn't much more inherent complexity in training an 8B or larger model other than more money, more compute, more data, more time. Overall, the principles are similar.
      • lostmsu 3y ago
        Assuming linear growth to number of parameters that's 7.5 figures instead of 6 for 8x22B model.