3 ms·

That sucks. I wanted to save up for a couple years and get some hardware for home, but I guess the "AI" space moves so fast you barely get a couple months

wmf 3y ago
Save up for Tenstorrent instead.
- LoganDark 3y ago
  I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.) Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn't seem promising?
  - sipjca 3y ago
    fwiw as a consumer I have a tenstorrent card in my machine
    - latchkey 3y ago
      that is great. please hit me up privately, i'd love to chat with you about it.
  - ojn 3y ago
    Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time. (It was all Python up until ~6 months ago) EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.
    - latchkey 3y ago
      Even talking to people there, my experience is that they are super nice!
    - LoganDark 3y ago
      > EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone. I actually don't mind having to contact, I only mind if they won't want to sell to me due to being a non-bulk order. > Most of the low-level pieces are in Rust That's awesome!
    - anony_moose 3y ago
      I figured I'd add a bit of clarification here. Tenstorrent has a couple SW stacks. The lowest level programming (Metalium) model is written in C++ and goes down to the register level. Higher level entrypoints (ttlib, ttnn) are written on top of that in python. I think it's the smi tooling and other systems software that might be written in rust.
SuchAnonMuchWow 3y ago
You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars. Their hardware was never for people at home, but for cloud providers.
- LoganDark 3y ago
  That doesn't sound right. Their public demo ran on 568 LPUs because they had Mixtral-8x7B and LLaMA-70B (45B and 70B respectively). IIRC their cards each have slightly over 200MB of SRAM so this almost exactly checks out. A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)

wmf 3y ago

Save up for Tenstorrent instead.

LoganDark 3y ago
I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.) Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn't seem promising?
- sipjca 3y ago
  fwiw as a consumer I have a tenstorrent card in my machine
  - latchkey 3y ago
    that is great. please hit me up privately, i'd love to chat with you about it.
- ojn 3y ago
  Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time. (It was all Python up until ~6 months ago) EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.
  - latchkey 3y ago
    Even talking to people there, my experience is that they are super nice!
  - LoganDark 3y ago
    > EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone. I actually don't mind having to contact, I only mind if they won't want to sell to me due to being a non-bulk order. > Most of the low-level pieces are in Rust That's awesome!
  - anony_moose 3y ago
    I figured I'd add a bit of clarification here. Tenstorrent has a couple SW stacks. The lowest level programming (Metalium) model is written in C++ and goes down to the register level. Higher level entrypoints (ttlib, ttnn) are written on top of that in python. I think it's the smi tooling and other systems software that might be written in rust.

SuchAnonMuchWow 3y ago

You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars. Their hardware was never for people at home, but for cloud providers.

LoganDark 3y ago
That doesn't sound right. Their public demo ran on 568 LPUs because they had Mixtral-8x7B and LLaMA-70B (45B and 70B respectively). IIRC their cards each have slightly over 200MB of SRAM so this almost exactly checks out. A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)