5 ms·
Fair warning, I have found local models and frontier models to be very bad at the specifics when it comes to cars. Small differences like month and year model
by ehnto 1mo ago
Fair warning, I have found local models and frontier models to be very bad at the specifics when it comes to cars.
Small differences like month and year model can impact oil capacity, oil weight and things like that, the details that matter quite a bit.
I found frontier models couldn't get things like what engine was in a 1994 Nissan Skyline, one of the more infamous and talked about cars on internet forums for decades, with dedicated fan databases that would have been scraped.
Questions like "what air filter do I need for my 1994 Suzuki Swift?" are hit and miss.
- javier123454321 1mo agoThis seems to be trained(? or referencing) on the specific cars' owner's manual.
- deleted 1mo ago[deleted]
- efskap 1mo agoYeah referencing is the way to go, as even finetuning probably captures style more than concrete facts. I know with large context windows we don't really RAG anymore, but for owner's manual lookup with a smaller model it seems ideal. Something every LLM user ends up learning is that they're far better used as search and summarization tools than as knowledge databases in themselves.
- HPsquared 1mo agoThe hard part is finding the correct owners manual. Also in many cases the owners manual is generic and covers many options and trim levels. Edit: this applies to service manuals too
- Kim_Bruning 1mo agoA model's internal knowledge is great! It's useful as initial priors to speed up the REAL search.
- whalesalad 1mo agoJust a few hours ago I gave ChatGPT my window sticker and the installation manual for a new suspension setup. I asked for new hardware that would typically be replaced during this install, like torque-to-yield bolts and fasteners. I also asked for new oil filters. I got a comprehensive grid of the exact part numbers needed in a nice dense table. sol 5.6 high is my daily driver.
- ssl-3 1mo ago> sol 5.6 high is my daily driver. Mine, too. Did you fact-check the output? I've found the bot to be particularly bad at stuff like producing accurate lists of parts to replace during automotive projects.
- whalesalad 1mo agoSo far so good, hasn’t failed me yet. It’s done a stellar job chasing down parts for my cub cadet lawn mower too. Sorted out mid year model revisions and everything. I just gave it the sticker under the seat.
- ssl-3 1mo agoIt makes stuff up about my car all the time, even when provided with factory and/or aftermarket service manuals to peruse in a custom GPT. That car is a ridiculously-common Honda Odyssey with an engine that is even more-common. I do sincerely wish you good luck with your suspension project.
- ygouzerh 1mo agoFor prompts that needs fact-checking, I like these days to use Perplexity directly instead these days. It's way faster than the default websearch tool + give a link to the reference directly.
- HPsquared 1mo agoYou really need something smart enough to browse through the parts etc suppliers and find links.
- alexandra_au 1mo agoIt's nothing that tool calling/feeding it the correct information can't solve
- embedding-shape 1mo agoEven so, in this case, author is using UD-Q3_K_S dynamic weights for Qwen3.6-35B-A3B, it will be dumb. Even the BF16 weights do stupid stuff like missing to confirm all parameters are defined when doing "rm -rf directory/$id", so it ends up deleting more than expected, I can't imagine the Q3 are actually useful for anything serious, even with tool calling or what not.
- petruspennanen 1mo agoI've been very impressed by it's intelligence and lack of hallucinations. The dynamic Q3 is a good balance between accuracy and size keeping the 35B just below 16 GB. It is not supposed to know everything, it is your car. It actively disengages from off topic chatter (too slow for that anyway), better spend that time feeling the car. It keeps itself grounded on sensor input. One principle per wheel. assert only what you can sense, claim only what is verified, label anything interim loudly, and report failure plainly with no silver lining. Everything above those four patches is just suspension.
- embedding-shape 1mo ago> . The dynamic Q3 is a good balance between accuracy and size keeping I'm having a hard time understanding how you find any sort of accuracy in Q3, when I use it with BF16 and it's hardly usable due to drastic hallucinations and inability for system prompt following. But, if it works for you, that's pretty good! Guess I'm jealous :)
- petruspennanen 1mo agoHmm we must be doing something different, I think the official accuracy measurement for the Q3 is more than 90%. Maybe some bug affecting your test?