manmal

34 ms·

751.
by manmal 2y ago
No, batched inference can work very well. Depending on architecture, you can get 100x or even more tokens out of the system if you feed it multiple requests in parallel.
752.
by manmal 2y ago
The Q8 model will likely slow this down to 50%, probably not a very useful speed. The 6k setup will probably do 10-12t/s at Q4.
753.
by manmal 2y ago
Interesting, I didn’t like the quality of the output of the 14B models. Could be the quantization though, apparently some are a bit broken.
754.
by manmal 2y ago
I've used it today, with R1 as architect and Sonnet as editor model. So far, this works great. There's no need to use a reasoning model as editor IMO. Alex ( https://alexcodes.app ) also does this now btw.
755.
by manmal 2y ago
In case you are on a 32+GB Mac, you could try deepseek-r1-distill-qwen-32b-mlx in LM Studio. It’s just barely usable speed-wise, but gives useful results most of the time.
756.
by manmal 2y ago
Continue.dev is available for Jetbrains, though the plugin is not as good as the VSCode counterpart. You can plug in any openai compatible API. Under experimental settings, you can also define an applyCode model (and others) which you could
757.
by manmal 2y ago
Do you use Continue.dev or similar tools to load code into the context, or do you copypaste into their web chat?
758.
by manmal 2y ago
Can you elaborate which models you are using? I‘m running an R1 distilled Qwen coder with 32B Q4, and while it’s giving useful answers, it‘s quite slow on my M1 Max. Slow enough that I keep reaching for cloud models.
759.
by manmal 2y ago
If you search on r/localllama, there are people who have improvised builds with eg 8x GPUs. Takes multiple power supplies and server mainboards. And some let the GPUs sit openly on wooden racks - not sure that’s good for longevity? BTW
760.
by manmal 2y ago
24GB VRAM. Using multiple ones scales well because models can be split by layers, and run in a pipelined fashion.
761.
by manmal 2y ago
I've had an Ioniq 1st Gen for 3 years, and while there was some brake rusting, it wasn't enough to not pass inspection. You can fix that by doing some heavy braking with recuperation disabled.
762.
by manmal 2y ago
The bad build quality of at least the US built models is a meme by now. The Chinese and German built models are supposedly way better though. Not that I'd buy a Tesla at this point.
763.
by manmal 2y ago
One 3090 seems to be equivalent to one M3 Max at inference: https://www.reddit.com/r/LocalLLaMA/s/BaoKxHj8ww There are many such threads on Reddit. M4 Max is incrementally faster, maybe 20%. Even if you facto
764.
by manmal 2y ago
No they are lacking compute power to be great at inference.
765.
by manmal 2y ago
The money would be better invested in a 2-4 3090 x86 build, than in a Mac Studio. While the Macs have a fantastic performance-per-watt ratio, and have decent memory support (both bus width and memory size), they are not great at compute pow
766.
by manmal 2y ago
Do you mean o1-preview or the current o1? I rarely get anything really useful out of the current one ($20 subscription, not the 200 one). They seem to have seriously nerfed it.
767.
by manmal 2y ago
The Hannsnote 2 is another one, and (at least has been) available in the EU.
768.
by manmal 2y ago
Might be the temporal dithering that can’t be disabled.
769.
by manmal 2y ago
A medium sized potato has that amount, I doubt it’d be the coconut milk to blame. Also, coconut water != coconut milk
770.
by manmal 2y ago
That’s not the case actually, if you consider that 10k cars are mostly used ones. Those are usually test driven, and haggled over first. And often taken in to a dealership to check the internals.
771.
by manmal 2y ago
That one doesn’t discriminate by location I think?
772.
by manmal 2y ago
Thanks, added context.
773.
by manmal 2y ago
Naive question, but why not raise taxes in hazardous areas, and use that money for a state-run insurance?
774.
by manmal 2y ago
I‘d say, you cannot write _interesting_ code without bugs.
775.
by manmal 2y ago
Responding to the link you posted: Apparently, the future of programming is 100 hour weeks? Naive me was thinking we could work less and think more with these new tools at our disposal.
776.
by manmal 2y ago
> The company now must assume the employee has been uploading confidential material to OpenAI too. If you think that’s not already the case for most of your codebase, you might be in for a rough awakening.
777.
by manmal 2y ago
Relatedly, asking LLMs what happens in a TV episode, or a series in general, I usually get very low quality and mostly flat out wrong answers. That baffles me, as I thought there are multiple well structured synopses for any TV series in th
778.
by manmal 2y ago
People buy 100k cars online nowadays, why wouldn’t a great online presence also work?
779.
by manmal 2y ago
5 years old, but here goes: https://www.test.de/Eiweisspulver-im-Test-1617462-1618800/
780.
by manmal 2y ago
Having talked with T1 diabetics about such systems, the issue is that you don’t want to deplete glycogen stores just for basal management. Because your life might depend on their levels in case of a severe hypo.

More ›