4 ms·

yeah, a used 3090 would be the most likely candidate if my next build was something for personal use only. Thing is, I'm probably going to expense it. Which lim

yeah, a used 3090 would be the most likely candidate if my next build was something for personal use only. Thing is, I'm probably going to expense it. Which limits me to new components. And I'm not in too much of a rush (weeks / short months). Can afford to wait for a month / two to see how the ROCm support pans out on the AMD side. 60% performance for <50% price is a compelling argument. If the ML support is decent... Though the 4090 is somewhat expensive, I might end up taking it anyway just for the simple reason that it's the only 24GB card with CUDA.

lhl 3y ago
Makes sense to me, although if you're using it professionally as an individual, then I would also seriously consider a last-gen A6000 (about $4K) - despite advances in quants, I'd say 48GB is a better minimum bar atm especially if you want to use long context, run multiple model types, do local fine tuning, etc.
- MezzoDelCammin 3y ago
  yeah, that's the thing. The AI/ML is something I'd like to play with / learn / experiment on, but it's only tangential to my current paying job at this point in time (I'm basically a freelancer). "Expense" in my context means that I'll deduce about 50% off of the invoice come taxes time. I'm OK paying $400 of my own money for an AMD card, I'd think twice, but probably accept coughing up $800 for a 4090, but $2k is definitely over the budget at this stage. Thanks for sharing the spreadsheet with Your benchmarks though. You're going into my "HN commenters to watch" list ;-)
  - lhl 3y ago
    I'm not sure where you're getting your pricing from but in the US a new 4090 is at least $1600, a 7900 XTX is about $1000 (a 20GB 7900 XT is $750-800). Used 3090s are usually about $700. If you're just dipping your toe in I'd recommend using Google Colab or cloud rentals like Runpod/Vast.ai which will be more cost effective if you're not running 24/7 or want to have access immediately on tap.
    - MezzoDelCammin 3y ago
      Ah, sorry, I was talking about the money I would pay after the expense deduction of 50%. So, 4090 at $1600 would cost me about $800 of my own money in the end. That being said, I'm writing from the EU, so all these numbers are really just ballparks where I take 1EUR==1USD for the sake of simplicity. Thanks for the rental tips though. I'll take a look. It's indeed more effective in the beginning...
- behnamoh 3y ago
  Is it not possible to do finetuning on a 24GB 4090? I'm thinking about the best bang for the buck below $2k and the options are 2x3090 or 1x4090.
  - brucethemoose2 3y ago
    > Is it not possible to do finetuning on a 24GB 4090? Not for 70B. You can finetune 7B or 13B.
    - Tostino 3y ago
      You can't FT 70b with 48gb of vram (2x 3090) either.

lhl 3y ago

Makes sense to me, although if you're using it professionally as an individual, then I would also seriously consider a last-gen A6000 (about $4K) - despite advances in quants, I'd say 48GB is a better minimum bar atm especially if you want to use long context, run multiple model types, do local fine tuning, etc.

MezzoDelCammin 3y ago
yeah, that's the thing. The AI/ML is something I'd like to play with / learn / experiment on, but it's only tangential to my current paying job at this point in time (I'm basically a freelancer). "Expense" in my context means that I'll deduce about 50% off of the invoice come taxes time. I'm OK paying $400 of my own money for an AMD card, I'd think twice, but probably accept coughing up $800 for a 4090, but $2k is definitely over the budget at this stage. Thanks for sharing the spreadsheet with Your benchmarks though. You're going into my "HN commenters to watch" list ;-)
- lhl 3y ago
  I'm not sure where you're getting your pricing from but in the US a new 4090 is at least $1600, a 7900 XTX is about $1000 (a 20GB 7900 XT is $750-800). Used 3090s are usually about $700. If you're just dipping your toe in I'd recommend using Google Colab or cloud rentals like Runpod/Vast.ai which will be more cost effective if you're not running 24/7 or want to have access immediately on tap.
  - MezzoDelCammin 3y ago
    Ah, sorry, I was talking about the money I would pay after the expense deduction of 50%. So, 4090 at $1600 would cost me about $800 of my own money in the end. That being said, I'm writing from the EU, so all these numbers are really just ballparks where I take 1EUR==1USD for the sake of simplicity. Thanks for the rental tips though. I'll take a look. It's indeed more effective in the beginning...
behnamoh 3y ago
Is it not possible to do finetuning on a 24GB 4090? I'm thinking about the best bang for the buck below $2k and the options are 2x3090 or 1x4090.
- brucethemoose2 3y ago
  > Is it not possible to do finetuning on a 24GB 4090? Not for 70B. You can finetune 7B or 13B.
  - Tostino 3y ago
    You can't FT 70b with 48gb of vram (2x 3090) either.