5 ms·

I'm eyeing Tinybox as a deep learning rig. https://tinygrad.org/ https://tinygrad.org/ https://twitter.com/tinygrad/status/1760988080754856210 https://twi

I'm eyeing Tinybox as a deep learning rig. https://tinygrad.org/ https://tinygrad.org/ https://twitter.com/__tinygrad__/status/1760988080754856210 https://twitter.com/__tinygrad__/status/1760988080754856210

Smith42 3y ago
$15k!
- KeplerBoy 3y ago
  Which is not unreasonable for that amount of hardware. You have to ask yourself if you want to drop that kind of money on consumer GPUs, which launched late 2022. But then again, with that kind of money you are stuck with consumer GPUs either way, unless you want to buy Ada workstation cards for 6k each and those are just 4090s with p2p memory enabled. Hardly worth the premium, if you don't absolutely have to have that.
  - cyanydeez 3y ago
    I believe the ada workstation cards are typically 1slot cards which means you could build a 4gpu server from normal cases. most of the 4090 cards are 2-3 slot cards
    - KeplerBoy 3y ago
      The beefy workstation cards are 2 slots, but yeah the 4090 cards are usually 3.something slots, which is ridiculous. The few dual slot ones are water cooled.
      - cyanydeez 3y ago
        the work station cards also run on 300 watts and looks like the 4090 goes to 450. so you are getting a better practical card for the price if you are making a mining type rig, then yeah, the extra price is wasting money. but if you wanted to build a normal machine, the workstation cards are the most reasonable choice for anything more than 2 gpus
        kkielhofner 3y ago
        I find it challenging to get my 4090s to consume more than 300 watts. There are also a lot of articles, benchmarks, etc around showing you can dramatically limit power while reducing perf by insignificant mounts (single digit %).
    - justsomehnguy 3y ago
      > which means you could build a 4gpu server from normal cases. Only if you are already live near an airport and you are accustomed to the sounds of the lift off and flying away.
treprinum 3y ago
Sure, if you want to waste your time on getting stuff working on AMD instead of spending it on actual model training...
- kkielhofner 3y ago
  This. People complain about the "Nvidia tax". I don't like monopolies and I fully support the efforts of AMD, Intel, Apple, anyone to chip away at this. That said as-is with ROCm you will: - Absolutely burn hours/days/weeks getting many (most?) things to work at all. If you get it working you need to essentially "freeze" the configuration because an upgrade means do it all over again. - In the event you get it to work at all you'll realize performance is nowhere near the hardware specs. - Throw up your hands and go back to CUDA. Between what it takes to get ROCm to work and the performance issues the Nvidia tax becomes a dividend nearly instantly once you factor in human time, less-than-optimal performance, and opportunity cost. Nvidia says roughly 30% of their costs are on software. That's what you need to do to deliver something that's actually usable in the real world. With the "Nvidia tax" they're also reaping the benefit of the ~15 years they've been sinking resources into CUDA.
  - dkjaudyeqooe 3y ago
    Wow! It's incredible how Nvidia has created the dark voodoo magic, and how only they can deliver the strong juju for AI. How are the so incredibly smart and powerful!? I wonder if it has anything to do with the strategy they used in 3D graphics, where game developers ended up writing for NV drivers in order to maximise performance, and Nvidia abused their market position and used every trick in the book to make AMD cards run poorly. People complained about AMD driver quality, but the actual problem was that they were not NV drivers and AMD couldn't defeat their software moat. So here we are again, this time with AI. You'd think we'd learnt our lesson but instead people are fooled yet again and instead of understanding the importance of diversity and competition in the lifeblood of their art, myopia and amnesia is the order of the day. Tinygrad are doing god's work and I won't be giving Nvidia a single fucking cent of my money until the software is hardware neutral and there is real competition.
tutfbhuf 3y ago
This is the new startup from George Hotz. I would like him to succeed, but I'm not so optimistic about their chances of selling a $15k box that is most likely less than $10k in parts. Most people would do much better by buying a second-hand 3090 or similar and connecting them into a rig.
- segmondy 3y ago
  Not necessarily, I'm not sure about AMD GPUs, but he tweeted that AMD supports linking all 6 together. If that's the case, then 6 of those XTX should crush 6 3090's. For us techies we definitely will decide to build vs buy. However businesses would definitely decide to buy vs build.
  - tutfbhuf 3y ago
    I think businesses are more likley to rent and techies are more likely to build. So he is betting on a niche, techies who wants to buy a 15k device or companies who do not want to rent. Also companies who are willing to go the AMD GPU route instead of NVIDIA GPU, which as much better tooling and much more experts on the job market.
- downrightmike 3y ago
  He keeps hopping from thing to thing
- varelse 3y ago
  [dead]
lostmsu 3y ago
As I mentioned in comments to this post on twitter, you can beat this with a pretty regular $6000 2x4090 system on compute (but with less total VRAM).

I'm eyeing Tinybox as a deep learning rig. https://tinygrad.org/ https://tinygrad.org/ https://twitter.com/__tinygrad__/status/1760988080754856210 https://twi

Smith42 3y ago

$15k!

KeplerBoy 3y ago
Which is not unreasonable for that amount of hardware. You have to ask yourself if you want to drop that kind of money on consumer GPUs, which launched late 2022. But then again, with that kind of money you are stuck with consumer GPUs either way, unless you want to buy Ada workstation cards for 6k each and those are just 4090s with p2p memory enabled. Hardly worth the premium, if you don't absolutely have to have that.
- cyanydeez 3y ago
  I believe the ada workstation cards are typically 1slot cards which means you could build a 4gpu server from normal cases. most of the 4090 cards are 2-3 slot cards
  - KeplerBoy 3y ago
    The beefy workstation cards are 2 slots, but yeah the 4090 cards are usually 3.something slots, which is ridiculous. The few dual slot ones are water cooled.
    - cyanydeez 3y ago
      the work station cards also run on 300 watts and looks like the 4090 goes to 450. so you are getting a better practical card for the price if you are making a mining type rig, then yeah, the extra price is wasting money. but if you wanted to build a normal machine, the workstation cards are the most reasonable choice for anything more than 2 gpus
      - kkielhofner 3y ago
        I find it challenging to get my 4090s to consume more than 300 watts. There are also a lot of articles, benchmarks, etc around showing you can dramatically limit power while reducing perf by insignificant mounts (single digit %).
  - justsomehnguy 3y ago
    > which means you could build a 4gpu server from normal cases. Only if you are already live near an airport and you are accustomed to the sounds of the lift off and flying away.

treprinum 3y ago

Sure, if you want to waste your time on getting stuff working on AMD instead of spending it on actual model training...

kkielhofner 3y ago
This. People complain about the "Nvidia tax". I don't like monopolies and I fully support the efforts of AMD, Intel, Apple, anyone to chip away at this. That said as-is with ROCm you will: - Absolutely burn hours/days/weeks getting many (most?) things to work at all. If you get it working you need to essentially "freeze" the configuration because an upgrade means do it all over again. - In the event you get it to work at all you'll realize performance is nowhere near the hardware specs. - Throw up your hands and go back to CUDA. Between what it takes to get ROCm to work and the performance issues the Nvidia tax becomes a dividend nearly instantly once you factor in human time, less-than-optimal performance, and opportunity cost. Nvidia says roughly 30% of their costs are on software. That's what you need to do to deliver something that's actually usable in the real world. With the "Nvidia tax" they're also reaping the benefit of the ~15 years they've been sinking resources into CUDA.
- dkjaudyeqooe 3y ago
  Wow! It's incredible how Nvidia has created the dark voodoo magic, and how only they can deliver the strong juju for AI. How are the so incredibly smart and powerful!? I wonder if it has anything to do with the strategy they used in 3D graphics, where game developers ended up writing for NV drivers in order to maximise performance, and Nvidia abused their market position and used every trick in the book to make AMD cards run poorly. People complained about AMD driver quality, but the actual problem was that they were not NV drivers and AMD couldn't defeat their software moat. So here we are again, this time with AI. You'd think we'd learnt our lesson but instead people are fooled yet again and instead of understanding the importance of diversity and competition in the lifeblood of their art, myopia and amnesia is the order of the day. Tinygrad are doing god's work and I won't be giving Nvidia a single fucking cent of my money until the software is hardware neutral and there is real competition.

tutfbhuf 3y ago

This is the new startup from George Hotz. I would like him to succeed, but I'm not so optimistic about their chances of selling a $15k box that is most likely less than $10k in parts. Most people would do much better by buying a second-hand 3090 or similar and connecting them into a rig.

segmondy 3y ago
Not necessarily, I'm not sure about AMD GPUs, but he tweeted that AMD supports linking all 6 together. If that's the case, then 6 of those XTX should crush 6 3090's. For us techies we definitely will decide to build vs buy. However businesses would definitely decide to buy vs build.
- tutfbhuf 3y ago
  I think businesses are more likley to rent and techies are more likely to build. So he is betting on a niche, techies who wants to buy a 15k device or companies who do not want to rent. Also companies who are willing to go the AMD GPU route instead of NVIDIA GPU, which as much better tooling and much more experts on the job market.
downrightmike 3y ago
He keeps hopping from thing to thing
varelse 3y ago
[dead]

lostmsu 3y ago

As I mentioned in comments to this post on twitter, you can beat this with a pretty regular $6000 2x4090 system on compute (but with less total VRAM).