5 ms·

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box.

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?

kkielhofner 3y ago
I've been bamboozled by the Jetson series over at least three generations on a variety of platforms (TX1, Nano, AGX, Orin Nano). They have their "special place" for certain applications but the software is a mess (old driver and CUDA versions, Jetpack is still based on Ubuntu 20.04), the ARM cores are (very) weak relative to ARM flagships, and the performance/price ratio makes no sense unless you really need the form factor and energy efficiency. Oh yeah and the SD card storage is typically frustratingly slow and often unreliable. A $500 Jetson Nano devkit with 8GB of shared RAM has roughly 10% of the performance of even ancient cards like the GTX 1070 (8GB VRAM alone) that you can throw in a random used x86_64 tower or whatever for $300 all-in. For the extra couple of hundred dollars difference you can get a more recent GPU with higher compute capability, extra storage, more system RAM, whatever. Significantly higher power usage and larger form factor but with power optimization, scaling, etc this approach makes very little difference in practice for occasional inference workloads for "typical home use". I have this configuration idling (with models loaded) at 20 watts, scaling up to around 100-150w for however long given inference loads take to execute, and then scaling back down.
thehumanmeat 3y ago
See tinybox https://tinygrad.org/ https://tinygrad.org/
- yunohn 3y ago
  15,000$…
  - JimtheCoder 3y ago
    Organization run by Geohot and no actual product delivered yet...
  - dsalzman 3y ago
    The apple lisa was 25k when adjusted for inflation…prices are for early adopters
    - turnsout 3y ago
      Yes, and the Lisa failed, because there just weren’t enough adopters at that price. I struggle to see any sizable market for the tinybox, but I wish them good luck.
      - Damogran6 3y ago
        Lessons learned on the Lisa were applied to the Mac for less money to sell to more people. I suspect the same thing will happen with their VR/AR/XR headset.
        turnsout 3y ago
        They only got a second crack at it because they were floating on sales of the Apple II. If the Lisa had been their first product, we'd be saying "Apple who?" The Vision is a whole different universe… with their cash and position they could (and may) take 10 cracks at it.
        Damogran6 3y ago
        Doesn't make the tactic any more or less relevant...let the people with money and motivation become the first real-world usability testers, then optimize for the rest of the population.
  - dharma1 3y ago
    You can build your own and you don’t need 6 GPUs unless you’re training
    - yunohn 3y ago
      I believe you missed the beginning of this thread? We’re specifically discussing pre-built solutions.
  - deleted 3y ago
    [deleted]
  - cstejerean 3y ago
    $15k for a box with 144 GB GPU RAM is not bad, but I'm not clear on how they're going to run that from a single 1600W PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd need 2x 1600W PSUs and two separate 15amp circuits to run such a thing at home (in the US).
    - elorant 3y ago
      You can undervolt gpus without loosing that much performance.
      - jrockway 3y ago
        But realistically, the 2x PSU case was a common crypto mining setup and yeah, you have your electrician install two circuits right next to each other. Or use a 240V PSU.
        cstejerean 3y ago
        That's exactly what I imagine would need to happen, I'm just surprised that the specs are claiming all of that with a single 1600W PSU. It's the kind of thing that doesn't inspire a lot of confidence that someone has thought through the details of what they are claiming to sell as part of this pre-order.
        kkielhofner 3y ago
        Nitpick - almost all modern power supplies are auto-ranging between 100-240V. In 240V locales at 80% constant load on a 15amp circuit this is 2880 watts. In the US you would (ideally) drop a 240V circuit from the panel and make sure you have sufficient supply amperage from the utility provider.
      - kkielhofner 3y ago
        With Nvidia this is (typically) setting the power limit[0]. Even with the default power limits of my RTX 4090s (480 watts)I don't think I've ever seen an ML workload get close to that and as the referenced article demonstrates you can more-or-less set your power limit to somewhere around 75% of max without losing much if any performance depending on workload. It doesn't take much testing to come up with the ideal power limit for your given workload(s). [0] - https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit-vs-performance-2296/ https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit...
dharma1 3y ago
For inference? Mobile phones (unless you need it always on). Kind of works already with 7B weights, will keep getting better
chad1n 3y ago
I was thinking that powerful used phones will be extremely valuable in the next years, since they are fairly cheap and more powerful than these devices.
- NavinF 3y ago
  Not gonna happen. Perf/watt on phones is terrible compared to consumer GPUs
- Damogran6 3y ago
  Vendors have been pushing hard for trade-in value when new phones are purchased to keep perfectly fine phones out of the ecosystem.
brucethemoose2 3y ago
Jetson is basically like Apple. Theoretically its good, but the models with enough RAM are too expensive. Smartphones aside, little Ryzen 6000 boxes would be OK. Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.
- joshvm 3y ago
  You can get an Orin with 8GB. Or the 32/64GB AGX module for 1.3/2.3k (3k for the dev kit). Not cheap but plenty of ram and 60W power target maximum. You could maybe run something on the 4GB Jetson Nano? But very slow: https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_pointless_experiment_jetson_nano_2gb_running/ https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_poi...
  - brucethemoose2 3y ago
    IIRC 32GB of shared RAM is not enough for llama 70B, and 8GB is just barely enough for llama 7B... So yeah, that value proposition is not good at all. A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC, just barely) do llama 70B for far less money and a similar TDP.

kkielhofner 3y ago

I've been bamboozled by the Jetson series over at least three generations on a variety of platforms (TX1, Nano, AGX, Orin Nano). They have their "special place" for certain applications but the software is a mess (old driver and CUDA versions, Jetpack is still based on Ubuntu 20.04), the ARM cores are (very) weak relative to ARM flagships, and the performance/price ratio makes no sense unless you really need the form factor and energy efficiency. Oh yeah and the SD card storage is typically frustratingly slow and often unreliable. A $500 Jetson Nano devkit with 8GB of shared RAM has roughly 10% of the performance of even ancient cards like the GTX 1070 (8GB VRAM alone) that you can throw in a random used x86_64 tower or whatever for $300 all-in. For the extra couple of hundred dollars difference you can get a more recent GPU with higher compute capability, extra storage, more system RAM, whatever. Significantly higher power usage and larger form factor but with power optimization, scaling, etc this approach makes very little difference in practice for occasional inference workloads for "typical home use". I have this configuration idling (with models loaded) at 20 watts, scaling up to around 100-150w for however long given inference loads take to execute, and then scaling back down.

thehumanmeat 3y ago

See tinybox https://tinygrad.org/ https://tinygrad.org/

yunohn 3y ago
15,000$…
- JimtheCoder 3y ago
  Organization run by Geohot and no actual product delivered yet...
- dsalzman 3y ago
  The apple lisa was 25k when adjusted for inflation…prices are for early adopters
  - turnsout 3y ago
    Yes, and the Lisa failed, because there just weren’t enough adopters at that price. I struggle to see any sizable market for the tinybox, but I wish them good luck.
    - Damogran6 3y ago
      Lessons learned on the Lisa were applied to the Mac for less money to sell to more people. I suspect the same thing will happen with their VR/AR/XR headset.
      - turnsout 3y ago
        They only got a second crack at it because they were floating on sales of the Apple II. If the Lisa had been their first product, we'd be saying "Apple who?" The Vision is a whole different universe… with their cash and position they could (and may) take 10 cracks at it.
        Damogran6 3y ago
        Doesn't make the tactic any more or less relevant...let the people with money and motivation become the first real-world usability testers, then optimize for the rest of the population.
- dharma1 3y ago
  You can build your own and you don’t need 6 GPUs unless you’re training
  - yunohn 3y ago
    I believe you missed the beginning of this thread? We’re specifically discussing pre-built solutions.
- deleted 3y ago
  [deleted]
- cstejerean 3y ago
  $15k for a box with 144 GB GPU RAM is not bad, but I'm not clear on how they're going to run that from a single 1600W PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd need 2x 1600W PSUs and two separate 15amp circuits to run such a thing at home (in the US).
  - elorant 3y ago
    You can undervolt gpus without loosing that much performance.
    - jrockway 3y ago
      But realistically, the 2x PSU case was a common crypto mining setup and yeah, you have your electrician install two circuits right next to each other. Or use a 240V PSU.
      - cstejerean 3y ago
        That's exactly what I imagine would need to happen, I'm just surprised that the specs are claiming all of that with a single 1600W PSU. It's the kind of thing that doesn't inspire a lot of confidence that someone has thought through the details of what they are claiming to sell as part of this pre-order.
      - kkielhofner 3y ago
        Nitpick - almost all modern power supplies are auto-ranging between 100-240V. In 240V locales at 80% constant load on a 15amp circuit this is 2880 watts. In the US you would (ideally) drop a 240V circuit from the panel and make sure you have sufficient supply amperage from the utility provider.
    - kkielhofner 3y ago
      With Nvidia this is (typically) setting the power limit[0]. Even with the default power limits of my RTX 4090s (480 watts)I don't think I've ever seen an ML workload get close to that and as the referenced article demonstrates you can more-or-less set your power limit to somewhere around 75% of max without losing much if any performance depending on workload. It doesn't take much testing to come up with the ideal power limit for your given workload(s). [0] - https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit-vs-performance-2296/ https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit...

dharma1 3y ago

For inference? Mobile phones (unless you need it always on). Kind of works already with 7B weights, will keep getting better

chad1n 3y ago

I was thinking that powerful used phones will be extremely valuable in the next years, since they are fairly cheap and more powerful than these devices.

NavinF 3y ago
Not gonna happen. Perf/watt on phones is terrible compared to consumer GPUs
Damogran6 3y ago
Vendors have been pushing hard for trade-in value when new phones are purchased to keep perfectly fine phones out of the ecosystem.

brucethemoose2 3y ago

Jetson is basically like Apple. Theoretically its good, but the models with enough RAM are too expensive. Smartphones aside, little Ryzen 6000 boxes would be OK. Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.

joshvm 3y ago
You can get an Orin with 8GB. Or the 32/64GB AGX module for 1.3/2.3k (3k for the dev kit). Not cheap but plenty of ram and 60W power target maximum. You could maybe run something on the 4GB Jetson Nano? But very slow: https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_pointless_experiment_jetson_nano_2gb_running/ https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_poi...
- brucethemoose2 3y ago
  IIRC 32GB of shared RAM is not enough for llama 70B, and 8GB is just barely enough for llama 7B... So yeah, that value proposition is not good at all. A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC, just barely) do llama 70B for far less money and a similar TDP.