3 ms·

300M params will train in a Google Colab most likely, or for a few bucks on vast.ai, or a few hours on a fairly decent card. So that is super small!

• baq 3y ago
  By 'fairly decent' do you mean a civilian 4090 or one of the weapons-grade DC cards?
  • buildbot 3y ago
    In my experience you’d need several weapons grade DC nodes (unless you are fine-tuning with LoRa, totally different game then!)
  • quickthrower2 3y ago
    3080, 4090, etc.
• buildbot 3y ago
  Errr 20M is a few hours to train 10K iters on 8x V100s for what I’ve been doing. 300M is in the range of 4-8 nodes of that scale.
  • quickthrower2 3y ago
    I'll check my wandb account when I can to see what I got (it was months ago). At the moment I can't log in "An application error occurred. Click to refresh the page." :-o I was using a 3090 though - I remember that because I was using vast.ai and a cheapskate! That card has serious bang for buck.
    • quickthrower2 3y ago
      Just a thought: Are people confusing million and billion?
      • buildbot 3y ago
        Or the library I’ve been using for this work is … not very memory efficient possibly? I’ve been referring to millions at least intentionally