4 ms·

Makes sense to me, although if you're using it professionally as an individual, then I would also seriously consider a last-gen A6000 (about $4K) - despite adva

by lhl 3y ago

Makes sense to me, although if you're using it professionally as an individual, then I would also seriously consider a last-gen A6000 (about $4K) - despite advances in quants, I'd say 48GB is a better minimum bar atm especially if you want to use long context, run multiple model types, do local fine tuning, etc.

MezzoDelCammin 3y ago
yeah, that's the thing. The AI/ML is something I'd like to play with / learn / experiment on, but it's only tangential to my current paying job at this point in time (I'm basically a freelancer). "Expense" in my context means that I'll deduce about 50% off of the invoice come taxes time. I'm OK paying $400 of my own money for an AMD card, I'd think twice, but probably accept coughing up $800 for a 4090, but $2k is definitely over the budget at this stage. Thanks for sharing the spreadsheet with Your benchmarks though. You're going into my "HN commenters to watch" list ;-)
- lhl 3y ago
  I'm not sure where you're getting your pricing from but in the US a new 4090 is at least $1600, a 7900 XTX is about $1000 (a 20GB 7900 XT is $750-800). Used 3090s are usually about $700. If you're just dipping your toe in I'd recommend using Google Colab or cloud rentals like Runpod/Vast.ai which will be more cost effective if you're not running 24/7 or want to have access immediately on tap.
  - MezzoDelCammin 3y ago
    Ah, sorry, I was talking about the money I would pay after the expense deduction of 50%. So, 4090 at $1600 would cost me about $800 of my own money in the end. That being said, I'm writing from the EU, so all these numbers are really just ballparks where I take 1EUR==1USD for the sake of simplicity. Thanks for the rental tips though. I'll take a look. It's indeed more effective in the beginning...
behnamoh 3y ago
Is it not possible to do finetuning on a 24GB 4090? I'm thinking about the best bang for the buck below $2k and the options are 2x3090 or 1x4090.
- brucethemoose2 3y ago
  > Is it not possible to do finetuning on a 24GB 4090? Not for 70B. You can finetune 7B or 13B.
  - Tostino 3y ago
    You can't FT 70b with 48gb of vram (2x 3090) either.

by lhl 3y ago

MezzoDelCammin 3y ago

yeah, that's the thing. The AI/ML is something I'd like to play with / learn / experiment on, but it's only tangential to my current paying job at this point in time (I'm basically a freelancer). "Expense" in my context means that I'll deduce about 50% off of the invoice come taxes time. I'm OK paying $400 of my own money for an AMD card, I'd think twice, but probably accept coughing up $800 for a 4090, but $2k is definitely over the budget at this stage. Thanks for sharing the spreadsheet with Your benchmarks though. You're going into my "HN commenters to watch" list ;-)

lhl 3y ago
I'm not sure where you're getting your pricing from but in the US a new 4090 is at least $1600, a 7900 XTX is about $1000 (a 20GB 7900 XT is $750-800). Used 3090s are usually about $700. If you're just dipping your toe in I'd recommend using Google Colab or cloud rentals like Runpod/Vast.ai which will be more cost effective if you're not running 24/7 or want to have access immediately on tap.
- MezzoDelCammin 3y ago
  Ah, sorry, I was talking about the money I would pay after the expense deduction of 50%. So, 4090 at $1600 would cost me about $800 of my own money in the end. That being said, I'm writing from the EU, so all these numbers are really just ballparks where I take 1EUR==1USD for the sake of simplicity. Thanks for the rental tips though. I'll take a look. It's indeed more effective in the beginning...

behnamoh 3y ago

Is it not possible to do finetuning on a 24GB 4090? I'm thinking about the best bang for the buck below $2k and the options are 2x3090 or 1x4090.

brucethemoose2 3y ago
> Is it not possible to do finetuning on a 24GB 4090? Not for 70B. You can finetune 7B or 13B.
- Tostino 3y ago
  You can't FT 70b with 48gb of vram (2x 3090) either.