kgeist

17 ms·

271.
by kgeist 1y ago
On my RTX 5090 with llama.cpp: gpt-oss 120B - 37 tok/sec (with CPU offloading, doesn't fit in the GPU entirely) Qwen3 32B - 65 tok/sec Qwen3 30B-A3B - 150 tok/sec (all at 4-bit)
272.
by kgeist 1y ago
>Qwen3 32B is 6 times slower than GPT OSS 120B. Only if 120B fits entirely in the GPU. Otherwise, for me, with a consumer GPU that only has 32 GB VRAM, gpt-oss 120B is actually 2 times slower than Qwen3 32B (37 tok/sec vs. 65 tok&#x
273.
by kgeist 1y ago
There's a bunch of phonetical differences: rus. Volk ~ ukr. Vovk rus. Aleksey ~ ukr. Oleksiy Different suffixes: rus. Tarasov ~ ukr. Tarasenko, Tarasiuk Ukrainian surnames often have no suffixes at all (more often than
274.
by kgeist 1y ago
Most people act on gut instincts first as well. Gut instinct = first semi-random sample from experience (= training data). That's where all the logical fallacies come from. Things like the bat and the ball problem, where 95% people giv
275.
by kgeist 1y ago
I asked Qwen3 32b (with reasoning) to count all 3 letter words in your post. Its answer was: There are 3 three-letter words in the given text: "Can" (first instance) "all" "Can" (second instance)
276.
by kgeist 1y ago
Counting letters is a known blindspot in LLMs because of how tokenization works in most LLMs - they don't see individual letters. I'm not sure it's a valid test to make any far-reaching conclusions about their intelligence. I
277.
by kgeist 1y ago
Did you enable reasoning? Qwen3 32b with reasoning enabled gave me the correct answer on the first attempt.
278.
by kgeist 1y ago
Qwen3 32b with reasoning (which I run locally) gives the correct answer. A pretty good model for its size. Pretty sure GPT5 with reasoning should be able to solve it, too. I guess the real problem here is that GPT5's router doesn'
279.
by kgeist 1y ago
HTTP can be 100% local without involving the web.
280.
by kgeist 1y ago
Unless you expose random ports on the local machine to the Internet, running apps on localhost is pretty safe. Llama-server's UI stores conversations in the browser's localStorage so it's not retrievable even if you expose yo
281.
by kgeist 1y ago
I'm not sure what you're talking about. Llama.cpp is an inference server which runs LLMs locally. It has a built-in web UI. You can't get more private than the inference server itself. I tried downloading your app, and it&#x2
282.
by kgeist 1y ago
>I challenge everyone to find another local GUI with that privacy Llama.cpp's built-in web UI.
283.
by kgeist 1y ago
We have a team account and my buddy has GPT-5 in the app but not on the website. At the same time, I have GPT-5 on the website, but in the app, I still only have GPT-4o. We're confused as hell, to say the least.
284.
by kgeist 1y ago
Yeah except I already could do the same with Qwen-coder-30b on my laptop a week ago.
285.
by kgeist 1y ago
The snake game they showcased - if you ask Qwen3-coder-30b to generate a snake game in JS - it generates the exact same layout, the exact same two buttons below, and the exact same text under the 2 buttons. It just regurgigates its trainin
286.
by kgeist 1y ago
GPT-5's knowledge cutoff is September 2024 so my first thought was they used GPT-4's pretrained base from 2024 and post-trained it additionally to squeeze those additional +5% on the benchmarks. And added the router.
287.
by kgeist 1y ago
Just a week ago I added Qwen3-Coder (the 30b one) to our corporate LLM server, enabled Artifacts in LibreChat, and demoed creating a snake clone in zero shot to coworkers. And now seeing the same exact thing from GPT5's live presentati
288.
by kgeist 1y ago
>I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs I have a bunch of scripts that use tool calling. Qwen-3-32B handles everything flawlessly at 60 tok/sec. Gpt-oss-120B breaks
289.
by kgeist 1y ago
By the time you decide to route to a particular node, conditions on that node might have already changed. So, from what I understand, there can be worst-case scenarios in usage patterns where the same nodes keep getting stressed due to repe
290.
by kgeist 1y ago
>Both China and Russia have claimed parts of other countries using the tactic of moving in their people to then use that as an excuse to annex or overtake those parts. >Crimea for Russia as an example, but this is also true for other
291.
by kgeist 1y ago
A friend of mine recently quit his job because the CEO insisted on personally reviewing and approving designs. The thing is, as a CEO, he was super busy with other CEO stuff, so my friend could only get small time slots, like 5 to 15 minute
292.
by kgeist 1y ago
Interesting, in Russian, it often ends with "Subtitles by %some_username%"
293.
by kgeist 1y ago
The idea isn't exactly novel, I read about it back in 2023 and implemented it in one of my bots. Back when open-source LLMs were still quite dumb, they'd often get stuck in repetitive loops after a while. Running multiple models i
294.
by kgeist 1y ago
We're using llama.cpp. We use all kinds of different models other than Qwen3, and vLLM startup when switching models is prohibitively slow (several times slower than llama.cpp, which is already 5 sec) From what I understand, vLLM is be
295.
by kgeist 1y ago
Model: Qwen3 32b GPU: RTX 5090 (no rops missing), 32 GB VRAM Quants: Unsloth Dynamic 2.0, it's 4-6 bits depending on the layer. RAM is 96 GB: more RAM makes a difference even if the model fits entirely in the GPU: filesystem pages cont
296.
by kgeist 1y ago
We run our own LLM server at the office for a month now, as an experiment (for privacy/infosec reasons), and a single RTX 5090 is enough to serve 50 people for occasional use. We run Qwen3 32b which in some benchmarks is equivalent to
297.
by kgeist 1y ago
It's more common than you'd think, even today. A lot of sites I recently explored leave SQL injections as is (you can see the typical MySQL errors) and rely on some kind of "security plugin" provided by a third-party for
298.
by kgeist 1y ago
The OP compares the current LLM hype to crypto, but I think it's more fair to compare it to the dotcom bubble. When a new, interesting technology appears, there's always a lot of hype around it - I think it's natural. People
299.
by kgeist 1y ago
We recently started using LLMs at our company, and the first job I had was to transcribe 20k customer calls and extract the following info: 1) what products we're usually compared to 2) what problems users have with our software 3) wha
300.
by kgeist 1y ago
As someone using a Russian keyboard, I still got my fair share of viruses back in the day, before I knew the basics of cybersecurity. I wonder how prevalent that actually is in the grand scheme of things, or if it's overblown in the ar

More ›