julianlam

6 ms·

61.
by julianlam 5mo ago
Try llama.cpp and Qwen3.6-35B-A3B Good balance of intelligence and speed.
62.
by julianlam 5mo ago
May I ask why the M instead of XL? Obviously bigger != better but I don't know what the differences are.
63.
by julianlam 5mo ago
Gemma 4 and Qwen 3.6 were when my local inference experiments graduated from toy challenges with much hand holding to actually full day back and forth with good ability to utilise tool calls to discover how things are glued together. I&#x27
64.
by julianlam 5mo ago
> for one year snort
65.
by julianlam 5mo ago
LLMs are bad at deterministic output. Full stop.
66.
by julianlam 5mo ago
> those aren't going to enable much future growth. What is with this obsessive need for "growth".
67.
by julianlam 5mo ago
Arguably, some of the things HN readers ask for can be capably completed by a local open weight model for free.
68.
by julianlam 5mo ago
Not by much, and moving goalposts makes for a bad comparison. Local open weight models are already more powerful than frontier models from only a year back. If you believe what you read here, the gap is closing fast.
69.
by julianlam 5mo ago
Indeed, that's what I do. I inspect the diff, though if it's an indentation change the entire block will be marked changed. Still not an excuse to not read every line of course... Unit tests give me the confidence that at least th
70.
by julianlam 5mo ago
> After I implemented that feature, a page from one of my favourite websites refused to load in the console... the third URL returns an HTTP 404 error page. The website uses the query string to determine which one of its several font c
71.
by julianlam 5mo ago
I always thought it was a little weird that LLMs aren't sophisticated enough to surgically edit files as needed. For example, if there is a code block that needs to be wrapped within another function call, it'll rewrite the entire
72.
by julianlam 5mo ago
Reader mode also works well
73.
by julianlam 5mo ago
> This started breaking down after ~30 files. Sometimes it would miss a file. Sometimes it would triple-test a bundle of files and take 10 minutes instead of 3. An error in one file would convince it it needs to re-test four previous fil
74.
by julianlam 6mo ago
So then these models could be used by llama.cpp today with the -md switch? Interesting, must try tomorrow.
75.
by julianlam 6mo ago
Does this mean there will be new Gemma 4 models released with MTP, or are they already available in existing models + quants?
76.
by julianlam 6mo ago
Really excited to try this once it is merged into llama.cpp. Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing. Have tried draft models to limited success (the small
77.
by julianlam 6mo ago
We're all busy doing work instead of incessantly commenting about our models?
78.
by julianlam 6mo ago
I only started playing around with local inference a couple weeks ago. Prior to that I was just using Gemini via web since it came with my Workspace subscription, but I did not want to be reliant on the cloud. Others will have a better idea
79.
by julianlam 6mo ago
It's so interesting to see the wild pendulum swings of LLM sentiment here. If one likes a model then it's capable of one-shotting entire apps. Otherwise it's "only suitable for the most trivial tasks". Never in betw
80.
by julianlam 6mo ago
Interesting that Gemma 4 didn't crack the top 10. I've been experimenting with the 26B-A4B model with some surprisingly good results (both in inference speed and code quality — 15 tok/s, flying along!), vs my last few experim
81.
by julianlam 6mo ago
Lo-fi channels used to show the artist and song names. These newer ones don't bother with credits, or have made up song titles. E.g. "funky chicken jam"
82.
by julianlam 6mo ago
Using an LLM to code without AC power is ... certainly a choice. On my framework 13 a couple exploratory prompts and a code writing step drained my battery to 85% in about 5 minutes.
83.
by julianlam 6mo ago
It's hardly a theory when the converse is plainly true. Look up similar jobs for academia, government, or NFP/Charities. They're (on paper) driven by their mission, not by profit, and the salaries match that goal.
84.
by julianlam 6mo ago
Thank you, I've been doing guided exploration of the various quantized models with the help of Gemini (which is highly ironic, but effective.) It does seem like 16GB is on the extreme lower end of being able to produce capable results,
85.
by julianlam 6mo ago
There were some niche use cases too. For awhile, the Ontario Health Insurance Program allowed physicians to submit their billing in person via zip drive. I remember depositing the disks for my parents (funny the things one remembers) This w
86.
by julianlam 6mo ago
Can you share more about what adaptations you made when using smaller models? I'm just starting my exploration of these small models for coding on my 16GB machine (yeah, puny...) and am running into issues where the solution may very w
87.
by julianlam 7mo ago
Before you start to feel smug about this and think you're above it. I've read three blog posts in as many days where their authors quietly reflect that Claude is so good that it has effectively hijacked their own decision making p
88.
by julianlam 7mo ago
It's baked in as deep as it can go. Use a different protocol.
89.
by julianlam 7mo ago
> Just a few years ago, AI essentially could not program at all. In the future, a given AI instance may “program better” than any single human in history. But for now, real programmers will always win. For how long? Do I get to feel smu
90.
by julianlam 7mo ago
Why would you expect it to? It probably predates these concepts. /s

More ›