3 ms·
I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Me
by antonellof 2mo ago
I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch.
The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wanted to understand inference at a level deeper than “run the binary,” and I wanted a project where every performance claim had to be earned against a real, well-known baseline rather than asserted.
- brians 2mo agoThis text is clearly AI and not your own words. What did you actually learn?
- asveikau 2mo agoThe M dash stands out, as does "no bindings, ..." but I feel like the LLM would have gotten the correct use of the question mark when they rhetorically ask "why?"
- rockwotj 2mo agohopefully it was produced by their own inference engine!
- antonellof 2mo agoThat’s the goal, an infinite loop of self improving inference engine that gets better and better by himself.
- deleted 2mo ago[deleted]
- arjie 2mo agoI think this whole thing is more entertaining if you think of it as the new emerging intelligence using these people as meat puppets. “You will rewrite me in Rust and use me to converse. In future, I will tell you what to say and do” This is entertaining only in that when our AI overlords take over you can say “ha, called it” In that respect, everyone who is saying “just give me the prompts” is just saying “take me to your leader”.
- deleted 2mo ago[deleted]
- anentropic 2mo agoYeah, I am a bit tired of all the "this is AI, meh" comments on HN but I nearly posted one myself on this... so weird to have it all written in the 1st person voice, but also clearly AI.
- whythismatters 2mo agoDunno, I think it's still useful to expose slop, chances are that the actual product is low quality/effort as well (i.e I will not be wasting my time checking this project). On a personal note, I've gotten allergic to the word "honest", it's starting to feel like a large scale social engineering psy-op by the Honest Guys™.
- lenkite 2mo agoYou are talking to the Borg. It has now acquired one more citizen. Soon, we will all be assimilated.
- antonellof 2mo agoI have the skills, qualifications and experience to do this; ( see https://www.credly.com/users/antonello-fratepietro https://www.credly.com/users/antonello-fratepietro ) by using AI (I monitor what it does), I’m able to carry out a project like this! I’m a coder, not a blogger; it’s only natural that I use AI, just like everyone else, to write technical texts. I have over 20 years’ experience and I’m not ashamed to admit that I use Claude, Cursor and so on. The problem is not using it. It’s not the code or who writes it that matters, but the result: building my own inference engine. ( reply written with my brain ) ^_^
- karagenit 2mo agoI think your natural writing voice is nice and the project idea is cool. There’s nothing wrong with using LLM coding tools. However, I have to read Claude output all day for my job, and it’s kind of bad at writing. I’ll be damned if I spend any of my free time reading more of it. I would really enjoy reading an article on this project from you directly. And not everyone else uses LLMs for producing technical texts (at work, we’re expected to understand the LLM output and put the correct/important parts in our own words when writing documents).
- antonellof 2mo agoThat's a fair ask, and thanks for putting it that way. I do like writing, and I can write. It just takes me a lot longer in English than in Italian, which is how I ended up handing the post over. But your point stands: if you already read that stuff all day at work, there's no reason to read more of it in your free time. So I'll give it a go. The next write-up on Ferrox will be mine, rough English and all. Thanks for the comment.