4 ms·

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

by mongol 3y ago

ajb117 3y ago
I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive. Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud. I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
- jmrm 3y ago
  AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/ https://coral.ai/products/accelerator/
  - worldsayshi 3y ago
    Very cool! Although seems to have no memory to speak of so many use cases like LLM goes away because of that I guess?
    - muxamilian 3y ago
      It has 8 MB of memory. It also supports live streaming the neural network to the chip, although that is slower than when it is cached in the memory.
  - kmeisthax 3y ago
    The problem is that this is an "inferencing" accelerator - i.e. it can only execute pretrained models. You cannot train a model on one of these, you need a training accelerator. And pretty much all of those are either NVidia GPUs or cloud-only offerings.
fulafel 3y ago
Apple and Google consumer hardware have specialized ML compute features. https://www.tomsguide.com/news/google-pixel-7s-most-critical-feature-what-tensor-g2-brings-to-the-table https://www.tomsguide.com/news/google-pixel-7s-most-critical... https://www.macobserver.com/tips/deep-dive/what-is-apple-neural-engine/ https://www.macobserver.com/tips/deep-dive/what-is-apple-neu...
lexicality 3y ago
Depending on your definition of affordable, the windows dev kit 2023 makes a big deal out of their NPU but you'll have to deal with windows 11 to access it unfortunately
wyldfire 3y ago
Qualcomm has an SDK [1] where you can run software on a DSP/NSP simulator. [1] https://developer.qualcomm.com/software/hexagon-dsp-sdk https://developer.qualcomm.com/software/hexagon-dsp-sdk

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

by mongol 3y ago

What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive. Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud. I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

jmrm 3y ago
AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/ https://coral.ai/products/accelerator/
- worldsayshi 3y ago
  Very cool! Although seems to have no memory to speak of so many use cases like LLM goes away because of that I guess?
  - muxamilian 3y ago
    It has 8 MB of memory. It also supports live streaming the neural network to the chip, although that is slower than when it is cached in the memory.
- kmeisthax 3y ago
  The problem is that this is an "inferencing" accelerator - i.e. it can only execute pretrained models. You cannot train a model on one of these, you need a training accelerator. And pretty much all of those are either NVidia GPUs or cloud-only offerings.

fulafel 3y ago

Apple and Google consumer hardware have specialized ML compute features. https://www.tomsguide.com/news/google-pixel-7s-most-critical-feature-what-tensor-g2-brings-to-the-table https://www.tomsguide.com/news/google-pixel-7s-most-critical... https://www.macobserver.com/tips/deep-dive/what-is-apple-neural-engine/ https://www.macobserver.com/tips/deep-dive/what-is-apple-neu...

lexicality 3y ago

Depending on your definition of affordable, the windows dev kit 2023 makes a big deal out of their NPU but you'll have to deal with windows 11 to access it unfortunately

wyldfire 3y ago

Qualcomm has an SDK [1] where you can run software on a DSP/NSP simulator. [1] https://developer.qualcomm.com/software/hexagon-dsp-sdk https://developer.qualcomm.com/software/hexagon-dsp-sdk