3 ms·

This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the abili

• adamsmith143 5y ago
  >This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. I think I support this in principle but it seems like the scaling curves keep going so it's easier to just make larger models with more data. >Looking forward to a group like Eluther or Hugging Face releasing a version of this Both of those groups have access to dozens if not hundreds of Cloud GPUs, I'd hardly call them small. It would be impossible to replicate these models as say an independent researcher or even in an academic research group outside of maybe Stanford/Berkeley/MIT/etc. and I'd even doubt their ability to replicate models like this based purely on Cost alone.
  • ianbutler 5y ago
    Small is relative -- and to Google, Facebook and Microsoft they're positively tiny. Perfect is the enemy of good or some such and I think this is a move in the right direction even if I can't personally train this on my 3090.