8 ms·

Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it

by Tsarp 13d ago

Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it makes sense to run something like a 8-27B on local network and get the work done. Are there perhaps some industrial or agri use cases?

HenryNdubuaku 13d ago
Fair, we gotta do a better job at explaining this properly! So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.
- yorwba 13d ago
  A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?
  - stymaar 13d ago
    At my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface). And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).
    - HenryNdubuaku 13d ago
      Thanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.
      - stymaar 13d ago
        We definitely intend to try fine-tuning, don't worry we're not going to dismiss needle just because the base model's performance is too low ;).
    - int_19h 13d ago
      You can embed much bigger models into web apps with wasm and WebGL or WebGPU. I have a web app running a 0.6B embedding model client-side.
      - stymaar 13d ago
        WebGPU is a non-starter for production usage since the support is too limited (No Firefox support, no Linux Support, no Apple x86 support, no MacOS <26). But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.
- ash_091 13d ago
  So it will run on those devices, but when/why would you do that? All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.
- KennyBlanken 13d ago
  You've explained WHAT it could run ON. You haven't shown any examples of its intended use, from input to result. I have no idea what is expected of me to be entering into that home automation example, and as everyone has handily demonstrated, it's dogshit at what people actually try to feed it so clearly im not the only one.

by Tsarp 13d ago

HenryNdubuaku 13d ago

Fair, we gotta do a better job at explaining this properly! So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.

yorwba 13d ago
A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?
- stymaar 13d ago
  At my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface). And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).
  - HenryNdubuaku 13d ago
    Thanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.
    - stymaar 13d ago
      We definitely intend to try fine-tuning, don't worry we're not going to dismiss needle just because the base model's performance is too low ;).
  - int_19h 13d ago
    You can embed much bigger models into web apps with wasm and WebGL or WebGPU. I have a web app running a 0.6B embedding model client-side.
    - stymaar 13d ago
      WebGPU is a non-starter for production usage since the support is too limited (No Firefox support, no Linux Support, no Apple x86 support, no MacOS <26). But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.
ash_091 13d ago
So it will run on those devices, but when/why would you do that? All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.
KennyBlanken 13d ago
You've explained WHAT it could run ON. You haven't shown any examples of its intended use, from input to result. I have no idea what is expected of me to be entering into that home automation example, and as everyone has handily demonstrated, it's dogshit at what people actually try to feed it so clearly im not the only one.