vibe42

4 ms·

31.
by vibe42 7mo ago
Higher-end gaming laptops are still decently priced and work well for local AI inference. And Linux runs better than ever on them; I'm running debian 13 with almost no driver issues. For $2k you can get 32 GB DDR5 RAM and 16 GB fast VR
32.
by vibe42 7mo ago
The main server runs 3x RTX PRO 6000 (288 GB VRAM combined), power limited to 280W each (can crank it up as temps are fine but about to add some more fans first as the cards are stacked). The second server is 2x Radeon RX 7900 XTX (48 GB VR
33.
by vibe42 7mo ago
Ethereum has a new site for PQ research: https://pq.ethereum.org/
34.
by vibe42 7mo ago
I've got a light version of this with local models; just one coding agent, one task takes 1-5 minutes. All local on constrained hardware helps; can't really run a ton of agents in parallel at good speeds. During each task I contex
35.
by vibe42 7mo ago
Will be pretty wild when mass migration of accounts begin. The analytics of thousands of accounts sending tokens to new accounts. Better use a VPN a migrate on an unusual hour in your time zone :D
36.
by vibe42 7mo ago
One thing to benchmark is if LLMs are better at solving complex problems if they're described in one language vs others. There's SWE-bench Multilingual for example, but translating a problem into multiple natural languages before
37.
by vibe42 7mo ago
Would recommend reading https://dnhkng.github.io/posts/rys/ and check his github code to reproduce the findings. Easy to way to start hacking LLMs; there's much of value there and a fun way to get into it be
38.
by vibe42 7mo ago
Decided on DDR4 for a new local AI server, works well as I try to keep models in VRAM anyway. For browsers and general apps, devs have blown up memory usage like crazy the past two decades.. there's so much low hanging fruit in optimiz
39.
by vibe42 7mo ago
Building my own home lab for local AI inference and general-purpose servers. Purpose is to learn more about hardware, Linux, networking, open source AI tools. Decided as a constraint to exclusively use local AI! This was fun in that the fi
40.
by vibe42 7mo ago
If everyone is running the same models, does this not favour white hat / defense? Since many exploits consists of several vulnerabilities used in a chain, if a LLM finds one in the middle and it's fixed, that can change a zero day
41.
by vibe42 7mo ago
The pace of development in llama.cpp is really high, could see an implementation being merged in 4-6 weeks.
42.
by vibe42 7mo ago
Just learned about it the other day from this thread from Feb, 2024: https://old.reddit.com/r/LocalLLaMA/comments/1aqrd7t/i_made_... Has some interesting github links.
43.
by vibe42 7mo ago
This is orthogonal to quantisation. Could have big impact on smaller models in the 4B-14B range where people often try specific quants and context sizes to fit into the VRAM of a laptop/desktop GPU.
44.
by vibe42 7mo ago
Perhaps not widely known but certainly known in LLM research. There was a bunch of these experiments done 2 years ago and what's interesting is that it still seems to work on the latest models. Though beware that the increased score on
45.
by vibe42 7mo ago
Pretty sweet hack as it's orthogonal to quantisation. And while it uses more compute, it doesn't require more VRAM. Maybe in the future circuits will become modular and composable like models are today?
46.
by vibe42 7mo ago
This is part of why running open models on hardware you control is valuable. They may trail SOTA by 6-12 months (really less for many use cases) but there's more reliability, control etc.
47.
by vibe42 7mo ago
Interesting to compare with deepseek mHC (Jan 2026), which also changes how residual connections work: https://arxiv.org/abs/2512.24880