3 ms·
A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?
by yorwba 10d ago
A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?
- stymaar 10d agoAt my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface). And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).
- HenryNdubuaku 10d agoThanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.
- stymaar 9d agoWe definitely intend to try fine-tuning, don't worry we're not going to dismiss needle just because the base model's performance is too low ;).
- int_19h 9d agoYou can embed much bigger models into web apps with wasm and WebGL or WebGPU. I have a web app running a 0.6B embedding model client-side.
- stymaar 9d agoWebGPU is a non-starter for production usage since the support is too limited (No Firefox support, no Linux Support, no Apple x86 support, no MacOS <26). But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.