4 ms·

You can use a Coral USB Accelerator, doesn't use more than 10W.

by phito 2y ago

nicholasjarnold 2y ago
Can confirm. The Coral inference accelerator is quite performant with very low power draw. Once I figured out some passthrough and config issues I was able to run Frigate in an LXC container on Proxmox using Coral USB for inference. It's been working reliably 24/7 for months now.
Eisenstein 2y ago
You can see here: res = rest(ollama, { "model": "llava", "prompt": genprompt(box.name), "images": [box.export()], "stream": False }) They are calling the ollama API to run Llava. Llava is a combo of an LLM base model and + vision projector (clip or ViT), and is usually around 4 - 8GB. Since every token generated needs access to all of the model weights, you would have to send 4 - 8 GB through USB with the Coral. Even at a generous 10gbit/s that is 8GB / 1.25GB = 6.4seconds per token. A 150 (short paragraph) generation would be 16minutes.
- phito 2y ago
  Hm yeah sure, I didn't think of the llm part. I don't think it's really useful tbh.
hcfman 2y ago
Yeah. But it’s likely it’s an 8-bit quantised, likely very small model with a small number of parameters. Which translates into poor recall and lots of false positives. How many parameters is the model you are using with hailo? And what’s the quantisation and what model is it actually ?
- phito 2y ago
  Honestly I have no idea what you are asking about. It's just dedicated hardware to a yolo-like object detection model
  - Eisenstein 2y ago
    They are asking about LLMs. There is a confusion it seems -- you are thinking of the object detection model (YOLO) which runs perfectly fine in (near) real time with a Coral or other NPU. The parent is referring the Llava part, which is a full-fledged language model with a vision projector glued onto to it for vision capability. Large language models are generally quantized (converted from full precision float values to less precise floats or ints for instance F16, Q8, Q4) because they would otherwise be extremely large and slow and require a ton of RAM (the model has to access the entire weights for every token generated, so if you don't have a gigantic amount of VRAM you would be pushing many tens of gigabytes of model weights through the system bus slowly).
    - janalsncm 2y ago
      Recall and false positives are classification metrics which relates to the YOLO part.

You can use a Coral USB Accelerator, doesn't use more than 10W.

by phito 2y ago

You can use a Coral USB Accelerator, doesn't use more than 10W.

Can confirm. The Coral inference accelerator is quite performant with very low power draw. Once I figured out some passthrough and config issues I was able to run Frigate in an LXC container on Proxmox using Coral USB for inference. It's been working reliably 24/7 for months now.

Eisenstein 2y ago

You can see here: res = rest(ollama, { "model": "llava", "prompt": genprompt(box.name), "images": [box.export()], "stream": False }) They are calling the ollama API to run Llava. Llava is a combo of an LLM base model and + vision projector (clip or ViT), and is usually around 4 - 8GB. Since every token generated needs access to all of the model weights, you would have to send 4 - 8 GB through USB with the Coral. Even at a generous 10gbit/s that is 8GB / 1.25GB = 6.4seconds per token. A 150 (short paragraph) generation would be 16minutes.

phito 2y ago
Hm yeah sure, I didn't think of the llm part. I don't think it's really useful tbh.

hcfman 2y ago

Yeah. But it’s likely it’s an 8-bit quantised, likely very small model with a small number of parameters. Which translates into poor recall and lots of false positives. How many parameters is the model you are using with hailo? And what’s the quantisation and what model is it actually ?

phito 2y ago
Honestly I have no idea what you are asking about. It's just dedicated hardware to a yolo-like object detection model
- Eisenstein 2y ago
  They are asking about LLMs. There is a confusion it seems -- you are thinking of the object detection model (YOLO) which runs perfectly fine in (near) real time with a Coral or other NPU. The parent is referring the Llava part, which is a full-fledged language model with a vision projector glued onto to it for vision capability. Large language models are generally quantized (converted from full precision float values to less precise floats or ints for instance F16, Q8, Q4) because they would otherwise be extremely large and slow and require a ton of RAM (the model has to access the entire weights for every token generated, so if you don't have a gigantic amount of VRAM you would be pushing many tens of gigabytes of model weights through the system bus slowly).
  - janalsncm 2y ago
    Recall and false positives are classification metrics which relates to the YOLO part.