Gracana

7 ms·

121.
by Gracana 10mo ago
You can probably bet that the dumbest possible interpretation of someone's argument is not what they had in mind.
122.
by Gracana 10mo ago
Check out "--n-cpu-moe" in llama.cpp if you're not familiar. That allows you to force a certain number of experts to be kept in system memory while everything else (including context cache and the parts of the model that ever
123.
by Gracana 10mo ago
This is not a web framework, but it's pretty close to what you've imagined: https://github.com/JirkaKlimes/jit-implementation
124.
by Gracana 10mo ago
I bought one when they were pretty new and I had issues with rocm (iirc I was getting kernel oopses due to GPU OOMs) when running LLMs. It worked mostly fine with ComfyUI unless I tried to do especially esoteric stuff. From what I've h
125.
by Gracana 11mo ago
Because it’s not that serious.
126.
by Gracana 11mo ago
That's really cool. What's it like to use in terms of performance and software compatibility?
127.
by Gracana 11mo ago
Maybe you’re right. My experience is with radios, where it’s possible that high frequency noise is conducted into the RF section rather than into the audio amplifier. I know that in one case, both my transmitted signal and received audio ou
128.
by Gracana 11mo ago
If you're a bit handy, you can assemble a line filter using a part like this https://enerdoor.com/products/fin27/ for a heck of a lot cheaper than you can buy a filtered power strip. You may also be able to s
129.
by Gracana 11mo ago
I did that for years and I recommend it as well. Pure linux desktop + console for games is a nice combo and a good separation of functions. Of course... at this point I am back to having a PC with a beastly GPU and I boot Windows for games
130.
by Gracana 11mo ago
Full article with more pictures and a video at the end: https://dl.acm.org/doi/10.1145/3745900.3746080 It does look cute. I could imagine something like this being at children's museum or something.
131.
by Gracana 11mo ago
You can do it slowly with ik_llama.cpp, lots of RAM, and one good GPU. Also regular llama.cpp, but the ik fork has some enhancements that make this sort of thing more tolerable.
132.
by Gracana 11mo ago
Three power supplies, an old server, a grocery cart and a box fan, and every 3090 you and your friends can get your hands on.
133.
by Gracana 11mo ago
That is literally the thing the parent poster wants to avoid by running open models. [edit] I was a little unfair -- lack of access to training data is a bit of an issue (perhaps moreso for analysis than for for actual use, considering what
134.
by Gracana 11mo ago
You could set up your own proxy. It doesn't have to be anything complicated, just a VPS with nginx forwarding requests to your servers on a VPN.
135.
by Gracana 11mo ago
You'd have to be doing something where the unified memory is specifically necessary, and it's okay that it's slow. If all you want is to run large LLMs slowly, you can do that with split CPU/GPU inference using a norma
136.
by Gracana 11mo ago
It's too bad they went to Windows later on. Having the Unix environment available makes the older stuff really flexible.
137.
by Gracana 1y ago
Ohhh, thanks, that's really good to know. I'll have to give that one a shot.
138.
by Gracana 1y ago
Yeah, by "structured" I mean how it wants to do ChatGPT-style output with headings and emoji and lists and stuff. And the punchy style of K2 0905 as shown in the fiction example in the linked article is what I really dislike. K2 T
139.
by Gracana 1y ago
Hi Simon. I have a Xeon W5-3435X with a 768GB of DDR5 across 8 channels, iirc it's running at 5800MT/s. It also has 7x A4000s, water cooled to pack them into a desktop case. Very much a compromise build, and I wouldn't recomm
140.
by Gracana 1y ago
Interesting, I have the opposite impression. I want to like it because it's the biggest model I can run at home, but its punchy style and insistence on heavily structured output scream "tryhard AI." I was really hoping that t
141.
by Gracana 1y ago
If you want to do it at home, ik_llama.cpp has some performance optimizations that make it semi-practical to run a model of this size on a server with lots of memory bandwidth and a GPU or two for offload. You can get 6-10 tok/s with m
142.
by Gracana 1y ago
My employer was running Growthpower (ERP software) on an HP 3000 system up until 2018 or so. We replaced it with a "modern" .NET/MSSQL ERP solution that does a lot more, but it's slow and terrible to navigate compared to
143.
by Gracana 1y ago
I don’t own a laptop. I run DeepSeek-V3 IQ4_XS on a Xeon workstation with lots of RAM and a few RTX A4000s. It’s not very fast, and I built it up slowly without knowing quite where I was headed. If I could do it over again, I’d go with a re
144.
by Gracana 1y ago
Yep, you need to install a ground conditioner for that.
145.
by Gracana 1y ago
https://proot-me.github.io/ Wow, that is so cool. This looks a lot more approachable than other sandboxing tools.
146.
by Gracana 1y ago
I would perhaps put pin receptacles / sockets there instead.
147.
by Gracana 1y ago
Odd perspective, I guess? Here's a different angle: https://techland.time.com/2012/04/16/photos-the-apple-ii-tur...
148.
by Gracana 1y ago
I used Omron's K3GN panel meters in a project at work and I had to draw the alphabet in the configuration drawing because it is so unintuitive. It's not a whole lot worse than the one shown in the article, but still... it's p
149.
by Gracana 1y ago
$2.5k is about $1k more than you'd spend on a pair of 3090s, and people I know who've bought blower 4090s say they sound like hair driers.
150.
by Gracana 1y ago
They probably don’t do it at a scale large enough to do RLHF with it, but it’s still useful feedback the people working on the projects / products.

More ›