jychang

20 ms·

271.
by jychang 1y ago
Speculative decoding doesn't change output tokens.
272.
by jychang 1y ago
You definitely want to use 4bit quants at minimum. https://arxiv.org/abs/2505.24832 LLMs usually have about 3.6 bits of data per parameter. You're losing a lot of information if quantized to 2 bits. 4 bit quants a
273.
by jychang 1y ago
How fast you run this model will strongly depend on if you have DDR4 or DDR5 ram. You will be mostly using 1 of your 3090s. The other one will be basically doing nothing. You CAN put the MoE weights on the 2nd 3090, but it's not going
274.
by jychang 1y ago
Yeah, this is the correct heuristic. Spend $15 or $100 for one or two measurements, *then* worry about cost to mitigate.
275.
by jychang 1y ago
They have similar memory bandwidth compared to the Mac Studio. You can run it off CPU at the same speed.
276.
by jychang 1y ago
If you're running it for personal use, you don't need to put all of it onto GPU vram. Cheap DDR5 ram is fine. You just need a GPU in the system to do compute for the prompt processing and to hold the common tensors that run for ev
277.
by jychang 1y ago
Qwen uses 16bit, Kimi and Deepseek uses FP8.
278.
by jychang 1y ago
Well, yeah, you're supposed to put in a GPU. It's a MoE model, the common tensors should be on the GPU, which also does prompt processing. The RAM is for the 400gb of experts.
279.
by jychang 1y ago
You can get similar performance on an Azure HX vm: https://learn.microsoft.com/en-us/azure/virtual-machines/siz...
280.
by jychang 1y ago
A million people is a very strong political force. A million determined voters can easily force laws to be made which forces youtube to be more efficient. I often think about how orthodoxical all humans are. We never think about different p
281.
by jychang 1y ago
Yeah. This is a really weird complaint to be honest. By that standard, python is not real programming because you're not managing your own memory. Is python considered AI now?
282.
by jychang 1y ago
I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real. If we some
283.
by jychang 1y ago
You want Tailscale for that.
284.
by jychang 1y ago
Ehhhh I strongly disagree with your statement. The biological system of taxonomy is really for the biology of the organism. We have other categories we use (as humans who are not biologists), even though we borrow organizational structure
285.
by jychang 1y ago
Oracle OCI? Their Ampere A1 free tier is pretty good. 4 core ARM and 24gb ram webserver for free.
286.
by jychang 1y ago
Yeah, that's exactly completely missing the point. A bayonet multiplies a person's power by 1.1x, a nuke multiplies it by more than 1,000,000x. Trying to be cute and lumping them together as "every technology is a force multi
287.
by jychang 1y ago
Well, duh. Same thing applies for "Technology X can be used for war". But anyone with a brain can see nukes are on a different level than bayonets. Claiming AI isn't unique in being a tool for evil isn't interesting, the
288.
by jychang 1y ago
Uh, "we" ? From someone who uses quotes „like this”? ... https://i.imgur.com/swpYbpv.png
289.
by jychang 1y ago
Nontrivial way to do it is dynamically scan the python 3.12 namespace, and add these warnings. Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?
290.
by jychang 1y ago
16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet. Right now, for a coding LLM on a Mac, th
291.
by jychang 1y ago
> Combined with VS Code’s ‘# %%’-demarcated notebook cells in .py files What do you mean by this?
292.
by jychang 1y ago
But they don't OFFER Openrouter a paid product... You cannot give roo/cline dollars and get openrouter api access.
293.
by jychang 1y ago
Roo/Cline doesn't offer Openrouter, markup or not.
294.
by jychang 1y ago
That's an absolutely terrible defense. Ignorance is not an excuse, try telling that to a police officer. And plus, certain people are held to a higher standard. It's not like I'm expecting a random person on the street to kno
295.
by jychang 1y ago
Strong disagree. Just because something is free and open source does not make it good. Call a spade a spade. Ollama is a piece of shit software that basically stole the work of llama.cpp, locks down their GGUFs files so it cannot be used by
296.
by jychang 1y ago
> Optional local AI tagging https://docs.linkwarden.app/self-hosting/ai-worker I took a look at this... and you use the Ollama API behind the scenes?? Why not use an OpenAI compatible endpoint like the rest of the i
297.
by jychang 1y ago
More like the analog version of doing a FTT and looking at only the high frequency parts vs the low frequency parts.
298.
by jychang 1y ago
Legally, they're allowed to modify and use GPL code internally without redistributing the source. The only mistake was publishing the source code to a public git repo without the LICENSE file, which may be a GPL violation. I say "
299.
by jychang 1y ago
8b params is a reasonable estimate. As a sanity check, we can look at scores for how it performs. On livebench, GPT-4o-mini scores 37.63, right next to Gemini 1.5 Flash 8B at 34.37, and above Qwen2.5 7B Instruct Turbo/Gemma 3 4B at 29.
300.
by jychang 1y ago
Do you think there are more bugs in Llama 4 at this time? Or have the bugs been patched, and the current version of llama.cpp + whatever the latest GGUF version is would be representative of the true performance of Llama 4? I see you'v

More ›