5 ms·

Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.

by KeplerBoy 3y ago

robterrell 3y ago
Have you tried this yourself? Curious to know how well this works for an LLM home lab.
- kkielhofner 3y ago
  I’ve worked with Jetson going back to the TK1 and I highly recommend you do not do this. Nvidia has significant dominance in the AI space because of their work on software and the overall platform. With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power. The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere. Just don’t do it.
  - dchichkov 3y ago
    This is a bit surprising to hear. Current Jetpack 6 is Ubuntu 22.04 - this is the current Ubuntu LTS release. There's nothing ancient about it, no? I'm pretty sure, if I go and check versions of CUDA, PyTorch, Tensorflow - it'd be also relatively recent. I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com From what I've seen, mainstream LLM libraries like VLLM, llamacpp that use CUDA under the hood tend to work out-of-the-box. And there are tutorials available: https://www.jetson-ai-lab.com/tutorial_text-generation.html https://www.jetson-ai-lab.com/tutorial_text-generation.html. I think that TensorFlow/Pytorch are also well maintained, although I've not checked recently.
    - KeplerBoy 3y ago
      Yes, it's all rather recent in my experience. You get CUDA 12 and the newest Pytorch.
      - kkielhofner 3y ago
        For now. Check back in a couple of years.
    - kkielhofner 3y ago
      I think this perspective comes from a lack of historical experience and hands-on experience overall. Nvidia more broadly has very impressive support for their GPUs. If you look at the support lifecycles for their Jetson hardware over time it's significantly worse. I encourage you to look at what support lifecycles have looked like, with the most "egregious" example being dropping of support for the Jetson Nano in from what I recall was within a couple of years. Another consideration - Jetson is optimized for power efficiency/form-factor and on a per $ basis CUDA performance is terrible. The power efficiency and form-factor come at significant cost. See this discussion from one of my projects[0]. I evaluated the use of WIS on an Orin Nano that I have and it was nearly 10x slower than a GTX 1070 which is seven years old and is still supported by the latest drivers and CUDA 12 on whatever OS you want. Nvidia knows what they're doing in terms of productization and the Jetson line should not be seen as some kind of secret hack/unlock for getting CUDA performance with gobs of RAM. In the case of LLMs I wouldn't be surprised at all if CPU beats it and at that point pickup 256GB of RAM or whatever for equivalent cost. In the end what do I care what people use, I'm offering the perspective and experience of someone who has actually used the Jetson line for many years and frequently struggled with all of these issues and more. [0] - https://github.com/toverainc/willow-inference-server/discussions/112 https://github.com/toverainc/willow-inference-server/discuss...
      - dchichkov 3y ago
        I've used Jetson for a few projects as a hobby. Made an I2S Sodar array with a TX2. And some robotics projects with a Jetson AGX Xavier that I got to evaluate and then to work on. And a few both, professional and toy projects with versions of Jetson Nano kit and Xavier. But this was between 2017 and 2021 or so. About a year back, I took that very early version of AGX Xavier, that got released years ago. It wasn't even the version that was officially released. Yet I was able to refresh it to newer Ubuntu without any issues. Wheels are often not pre-built for aarch64, yes. If you want to compile directly on Nano, disk performance is very important. Sometimes you get I/O bound. Orin Nano being that slow in [0], it looks like you've been trying it in Aug 2023. It maybe worth re-evaluating on the latest Jetpack, it had transitioned to CUDA 12.2, TensorRT 8.6, cuDNN 8.9. I would expect that recent popularity of ASR/TTS pipelines and LLMs was not completely missed by Jetpack maintainers (there are some tutorials here - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com ). And recently released JetPack could be optimized a lot more for these workflows. And your project is very cool! I'd suggest sharing it and your performance numbers (!) with the maintainers of: https://developer.nvidia.com/embedded/community/jetson-projects https://developer.nvidia.com/embedded/community/jetson-proje...
- qrios 3y ago
  According to this article [1] it looks like there is no complex preparation needs to run the inference on a Jetson system. Should work with Mixtral too. [1] https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jetson-and-textgeneration-web-ui-96b070 https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jets...
- KeplerBoy 3y ago
  I haven't tried it for LLMs yet, i use it for real time RF processing, but I actually have one of them on my desk and they are fun little devices. Maybe I will try to get a 32 GB+ LLM running one of those days.
fsiefken 3y ago
I wonder what would be the cheapest way to run an LLM, with the latest Ryzen integrated graphics and 64G Ram or the Jetson AGX Orin 64. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...
- KeplerBoy 3y ago
  The Ryzen is a lot cheaper, but most likely also a fair bit slower. You'd be looking at a 200$ CPU, 200$ Motherboard + 200$ of ddr5 ram. Throw in a case, nvme drive and power supply and you're still below $1k and those numbers are quite generous estimates, you could do it a lot cheaper by going AM4 with DDR4 ram.

Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.

robterrell 3y ago

Have you tried this yourself? Curious to know how well this works for an LLM home lab.

kkielhofner 3y ago
I’ve worked with Jetson going back to the TK1 and I highly recommend you do not do this. Nvidia has significant dominance in the AI space because of their work on software and the overall platform. With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power. The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere. Just don’t do it.
- dchichkov 3y ago
  This is a bit surprising to hear. Current Jetpack 6 is Ubuntu 22.04 - this is the current Ubuntu LTS release. There's nothing ancient about it, no? I'm pretty sure, if I go and check versions of CUDA, PyTorch, Tensorflow - it'd be also relatively recent. I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com From what I've seen, mainstream LLM libraries like VLLM, llamacpp that use CUDA under the hood tend to work out-of-the-box. And there are tutorials available: https://www.jetson-ai-lab.com/tutorial_text-generation.html https://www.jetson-ai-lab.com/tutorial_text-generation.html. I think that TensorFlow/Pytorch are also well maintained, although I've not checked recently.
  - KeplerBoy 3y ago
    Yes, it's all rather recent in my experience. You get CUDA 12 and the newest Pytorch.
    - kkielhofner 3y ago
      For now. Check back in a couple of years.
  - kkielhofner 3y ago
    I think this perspective comes from a lack of historical experience and hands-on experience overall. Nvidia more broadly has very impressive support for their GPUs. If you look at the support lifecycles for their Jetson hardware over time it's significantly worse. I encourage you to look at what support lifecycles have looked like, with the most "egregious" example being dropping of support for the Jetson Nano in from what I recall was within a couple of years. Another consideration - Jetson is optimized for power efficiency/form-factor and on a per $ basis CUDA performance is terrible. The power efficiency and form-factor come at significant cost. See this discussion from one of my projects[0]. I evaluated the use of WIS on an Orin Nano that I have and it was nearly 10x slower than a GTX 1070 which is seven years old and is still supported by the latest drivers and CUDA 12 on whatever OS you want. Nvidia knows what they're doing in terms of productization and the Jetson line should not be seen as some kind of secret hack/unlock for getting CUDA performance with gobs of RAM. In the case of LLMs I wouldn't be surprised at all if CPU beats it and at that point pickup 256GB of RAM or whatever for equivalent cost. In the end what do I care what people use, I'm offering the perspective and experience of someone who has actually used the Jetson line for many years and frequently struggled with all of these issues and more. [0] - https://github.com/toverainc/willow-inference-server/discussions/112 https://github.com/toverainc/willow-inference-server/discuss...
    - dchichkov 3y ago
      I've used Jetson for a few projects as a hobby. Made an I2S Sodar array with a TX2. And some robotics projects with a Jetson AGX Xavier that I got to evaluate and then to work on. And a few both, professional and toy projects with versions of Jetson Nano kit and Xavier. But this was between 2017 and 2021 or so. About a year back, I took that very early version of AGX Xavier, that got released years ago. It wasn't even the version that was officially released. Yet I was able to refresh it to newer Ubuntu without any issues. Wheels are often not pre-built for aarch64, yes. If you want to compile directly on Nano, disk performance is very important. Sometimes you get I/O bound. Orin Nano being that slow in [0], it looks like you've been trying it in Aug 2023. It maybe worth re-evaluating on the latest Jetpack, it had transitioned to CUDA 12.2, TensorRT 8.6, cuDNN 8.9. I would expect that recent popularity of ASR/TTS pipelines and LLMs was not completely missed by Jetpack maintainers (there are some tutorials here - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com ). And recently released JetPack could be optimized a lot more for these workflows. And your project is very cool! I'd suggest sharing it and your performance numbers (!) with the maintainers of: https://developer.nvidia.com/embedded/community/jetson-projects https://developer.nvidia.com/embedded/community/jetson-proje...
qrios 3y ago
According to this article [1] it looks like there is no complex preparation needs to run the inference on a Jetson system. Should work with Mixtral too. [1] https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jetson-and-textgeneration-web-ui-96b070 https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jets...
KeplerBoy 3y ago
I haven't tried it for LLMs yet, i use it for real time RF processing, but I actually have one of them on my desk and they are fun little devices. Maybe I will try to get a 32 GB+ LLM running one of those days.

fsiefken 3y ago

I wonder what would be the cheapest way to run an LLM, with the latest Ryzen integrated graphics and 64G Ram or the Jetson AGX Orin 64. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...

KeplerBoy 3y ago
The Ryzen is a lot cheaper, but most likely also a fair bit slower. You'd be looking at a 200$ CPU, 200$ Motherboard + 200$ of ddr5 ram. Throw in a case, nvme drive and power supply and you're still below $1k and those numbers are quite generous estimates, you could do it a lot cheaper by going AM4 with DDR4 ram.