5 ms·
Ternlight – 7 MB embedding model that runs in browser (WASM)
- bvrmn 3mo agoDemo works quite strangely. For example "how to use typescript with createContext" show only typescript entries on top. Similarity search failed.
- soycaporal 3mo agoHobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies. Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app https://ternlight-demo.vercel.app Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed). Bundled for Node and browsers. Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight https://github.com/soycaporal/ternlight Curious if this is something useful, what are the use cases for on-device embeddings.
- fellowniusmonk 3mo agoAwesome! Besides size, how does this compare to gte-small?
- soycaporal 3mo agogte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher). I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.
- dwheeler 3mo agoThanks! I strongly suggest copy-pasting that explanation to your web page, that's a nice summary.
- keynha 3mo ago0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?
- soycaporal 3mo agoIt's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact. The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.
- deleted 3mo ago[deleted]
- versteegen 3mo agoNice, I'm really interested in using this for simple semantic search in a native desktop application. Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)". But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.
- heltale 3mo agoSame! I’m trying to find small models that can embed effectively to enable BM25/hybrid search over a large number of documents for a personal information repository. Ideally, it should run on consumer hardware. bge-small-en-v1.5 is one that is comparable and what we’re working with for now.
- soycaporal 3mo agothe base model I clocked it at 5 ms per embedded on my mac studio. There is a mini variant (the demo version) that is sub - 2 ms. It could be a SIMD issue.. I'll look into this for better runtime support (also fee free to file an issue)
- dannyw 3mo agoHuge kudos for sharing everything including your training code. Awesome project!
- abrookewood 3mo agoWhat is the process for adding different text? What are the limitations on that process? The demo is very cool by the way.
- maelito 3mo agoAwesome. We have a dictionary of words to OpenStreetMap tags here : https://codeberg.org/cartes/web/src/branch/master/components/categories/categories.yaml#L46-L63 https://codeberg.org/cartes/web/src/branch/master/components... Do you think your work could help us let users type "pancake" and get "crêpe" without writing an explicit "pancake = crêpe" dictionary entry ? In practice : if I understand well, your lib would first need to download 5 Mb, once and for all, and would then be used as we use Fuse.js right now ? How well does it handle languages other than English ? Could it be "trained" on the OpenStreetMap tag wiki ? Thanks a lot for your work.
- yorwba 3mo agoThe OSM tag wiki would probably not make for good training data because it only has a single short description for each tag (and even on the French wiki many descriptions seem to be in English?) whereas you want to map multiple descriptions to the same tag. Ideally you would have real query data (e.g. from cartes.app telemetry), then you could get a LLM to write a bespoke Overpass query for each one and use that as the ground truth. Alternatively, start from the list of OSM tag values used in the wild and ask an LLM to list possible reasons to visit that POI. You could then use that data to finetune an embedding model for your use case. But, you know, somewhere in that model there's going to be a token vocabulary that the model knows about and at the other end you get a similarity score for each tag value. If you don't need to support complex queries where interactions between words matter ("any restaurant that is NOT Korean"), you could get away with a simple list of words and tags that they match to. Which is right where you started, except it could be more exhaustive. Why limit yourself to two Korean dishes when you can have a LLM list many more for you?
- maelito 3mo agoThanks a lot !
- soycaporal 3mo agoThe corpus is mainly trained in english, unfortunately no other languages have been included in the distillation training. Yes it would work like fuse.js, but unlocks semantic search. Source code has the entire embedding distillation pipeline includes dataset preparation. You could run the same distillation training (but not sure if teacher model used a multi-language dataset). Open up an issue on the github repo and I can reply with details
- sodimel 3mo agoThank you for this tool! We've just used it to embed the entire django doc + our private knowledge base, allowing us to search in the 2 sources instantly!
- soycaporal 3mo agosuper cool use case! Hopefully it can provide quality embeddings + retrieval. Would love to learn to results/issues or feedback. Please feel free to file for issues on github
- aetherspawn 3mo agoCan the 30 second embedding time be done beforehand and sent to the browser? Inference is nice and quick after that.
- soycaporal 3mo agoyes, you could run a 1 time indexing run on the server side, and just ship the embeddings to frontend
- dirteater_ 3mo agoThis is cool! but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
- Waterluvian 3mo agoAgree. But this also reminds me fondly of the days where the sounds of my computer so intimately indicated what’s going on.
- soycaporal 3mo agoCPU cycle maxxing, who said GPUs were special?
- jonnonz 3mo agoSame here, when the fans started up I got startled. However my bread toaster often scares me too
- esafak 3mo agoWhat we need is a W3C LLM API like the one Chrome already offers: https://developer.chrome.com/docs/ai/built-in https://developer.chrome.com/docs/ai/built-in
- yesidoagree 3mo agoIf it was like Math (Math.round, Math.PI, etc.) it could be Language, as in: Language.complete('the quick brown fox jumped over the lazy') and maybe even static methods on Image Image.generate('a spaceship flying toward a planet')
- soycaporal 3mo agoI think standardizing the runtime is pretty effective, it then open up portability
- wazzup_im 3mo agoI added an offline search engine to app.wazzup.im/search (no login or payment required). First search downloads the model from the internet and subsequent runs are from the cache. The model is very small so it's not the best for everything but it's good for basic math and coding. Give it a try.
- Barbing 3mo agoIn Safari, stuck on: Loading model... + Loading search results... Or sometimes "Service Worker API is available and in use." + "Loading search results...".
- wazzup_im 3mo agoThis is a known issue and I am actively trying to find why this is happening. So far it's pretty good on Brave/Chrome. Tested on Macbook Pro M1 8gb RAM and Macbook Air M1 8gb RAM. Mostly likely because of M series of chips. All tests were done on Brave/Chrome. Does not work on iPhone 11 Pro Max and iPhone 16 Pro. Mostly likely because of A series of chips. Tests were done on Safari and Chrome and it crashes on both.
- soycaporal 3mo agoohh thanks for the report.. probably has to do with wasm runtime.. Will note this as a known issue
- wazzup_im 3mo agoNp! The workaround is to unregister/stop the service worker from the DevTools > Application tab > Service workers.
- wazzup_im 3mo agoAlso does not work in Firefox 152.0.6 (aarch64). Due to WebGPU issue.
- wazzup_im 3mo agoHere's a video https://youtu.be/X6M7T0lLqTo https://youtu.be/X6M7T0lLqTo
- CobrastanJorji 3mo agoGreat, now my websites are gonna push entire LLMs onto my browser in order to use my CPU to make inferences about my shopping habits or whatever.
- antonvs 3mo agoDisabling WASM is the new disable JavaScript
- paytonjjones 3mo agoHa, I was literally thinking this but from the other side. "Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend" Probably a lot more practical to use this though: https://developer.apple.com/apple-intelligence/ https://developer.apple.com/apple-intelligence/
- deleted 3mo ago[deleted]
- iammrpayments 3mo agoIf you think about it, running a crypto miner without being asked is probably less annoying than downloading an entire LLM, but only the first will get you in jail.
- Technical_Plant 3mo ago[dead]
- newspaper1 3mo agoVery cool! I'd love to point it at my own corpus to index/embed. Would be cool if you could give it a link to a markdown file or even a website to crawl.
- soycaporal 3mo agolove the idea! Will think of a way to host it probably on huggingface
- rvz 3mo agoWhy do these things download into the browser automatically? This could be used to distribute malware and also or hog excessive browser memory.
- deleted 3mo ago[deleted]
- gaigalas 3mo agoThat's... how the web works? You download things on demand. There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now. A WASM model is not that offensive.
- akoboldfrying 3mo agoThis doesn't add any malware risks beyond what a JavaScript-enabled browser already allows. Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
- tangsoupgallery 3mo ago[flagged]
- gaigalas 3mo agoThat's really impressive, congratulations. It's nice to see novel applications of browser models.
- soycaporal 3mo agothank you! hopefully it can unlock some novel applications, that would be cool
- dmezzetti 3mo agoInteresting project. Happy to see someone who shares an interest in tiny vector embeddings models. I've worked on tiny (1MB - 4MB, 250K - 950K parameters) embeddings models called BERT Hash https://huggingface.co/blog/NeuML/bert-hash-embeddings https://huggingface.co/blog/NeuML/bert-hash-embeddings Keep up the great work!
- chris-hartwig 3mo agoThank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.
- soycaporal 3mo agothat's great! let me know if there is anyway I can support, or any specific use case a roadmap could address!
- jbellis 3mo agoFWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.
- soycaporal 3mo agoawesome, noted, looking for capable teacher models to distill other architectures
- jbellis 3mo agovoyage 4 nano is sota at the next size up and if you really want the best teacher models it's probably the voyage commercial APIs
- kamranjon 3mo agoThis would be a pretty cool addition to the duckdb HNSW search project I found on here some time ago: https://github.com/jasonjmcghee/portable-hnsw https://github.com/jasonjmcghee/portable-hnsw What I think is really cool is that the search happens using http range queries across statically hosted parquet files. I think things like this could bloom into a relatively open and distributed search ecosystem that isn’t controlled by major corporations.
- TheTaytay 3mo agoCool idea. I love range requests and other static-hosted client-navigable formats!
- lexh 3mo agoSimilar idea here that may be of interest: SQLite DB on static host via HTTP range + WASM. https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- soycaporal 3mo agovery cool, I'll look into this. Thanks for sharing.
- tonysbuildsx 3mo ago[flagged]
- superzero11 3mo ago[flagged]
- ljcoco 3mo agocool stuff
- abhgh 3mo agoCool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser. So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side. [1] http://sol.quipu-strands.com/ http://sol.quipu-strands.com/
- soycaporal 3mo agoamazing.. glad to know this integration path worked fro you!
- iberator 3mo agoPrime example of wasm supremacy over JavaScript. Stack machines for the win hehe
- syx 3mo ago[dead]
- aziis98 3mo agoThis would be nice as an Astro (or generic meta-framework plugin) that automatically parses all generated html files and generates a small db of embeddings. This way on the frontend you can lazily load this. Maybe you could even store the HNSW in chunks and just load the pieces you need for your specific search query. i.e. like https://pagefind.app/ https://pagefind.app/ but to get fully static vector search.
- weitendorf 3mo agoWe really wanted to use sqlite-vec for this for our SSG but last we checked it hadn’t implemented HNSW/had good support for running vector search in-browser yet (I think it was still doing full-table scans?). I was pretty disappointed because after so many months/years, to not have that suggested to me that they weren’t up to task of delivering on their project, and I had recommended them as a worthy project for a grant I had also applied for, that they won and I didn’t. If anybody knows of a good solution in this space, or if I’m wrong about SQLite-vec, please let me know. For our own SSG we’ve basically decided that we’ll give it a couple months while we work on other infra we want, then if they’re still not done we’ll just do it ourselves.
- VaporJournalAPP 3mo ago[dead]
- zibbava 3mo ago[flagged]
- WhitneyLand 3mo agoNice work. It’s advertised 7MB, but also comes with a 5MB mini version. Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility. It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.
- littlerobot 3mo ago[dead]
- scritty-dev 3mo agoso this is really cool and I think could be the missing piece for something I wanted to build, I found this awhile back and using https://github.com/npiesco/absurder-sql https://github.com/npiesco/absurder-sql you could keep the entire raw corpus in browser (persisted via IndexedDB/SQLite)...then you could generate + cache embeddings on demand with Ternlight (instead of pre-indexing everything i.e., https://weaviate.io/blog/chunking-strategies-for-rag https://weaviate.io/blog/chunking-strategies-for-rag). then this opens up the door for Reciprocal Rank Fusion (RRF) aka hybrid retrieval where you combine FTS5/BM25 from the native SQLite plues the semantic search using from TernLight!
- marashiai 3mo agoVery nice to see this type of work. The next version of web should become more interactive with the help of AI.
- febed 3mo agoApart from maximizing compatibility, is there a technical reason why GPU is not used if available?
- soycaporal 3mo agomainly this is a portability and compatibility solution.. I think with GPU available deployments, it wouldn't benefit from ternary distillation. It would be a different problem domain.