Gracana

22 ms·

241.
by Gracana 2y ago
I chose the 3090/4090 because it seems to me that this machine could be a replacement for a workstation or a homelab rig at a similar price point, but not a $100-250k server in a datacenter. It's not really surprising or interesti
242.
by Gracana 2y ago
It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of
243.
by Gracana 2y ago
I would love to hear that recording, haha.
244.
by Gracana 2y ago
In an industrial setting, these robots would be placed behind interlocked barriers and you wouldn't be able to approach them unless they were de-energized. Collaborative robotics (where humans work in close proximity to unguarded robot
245.
by Gracana 2y ago
This isn't DTMF. It's a form of MFSK like DTMF, but it operates on different frequencies and uses six tones at once vs DTMF's two.
246.
by Gracana 2y ago
> scrolling, panning, videos, gifs. It all feels as it should Is it really that good? Okay, I found a video... Wow! https://www.youtube.com/watch?v=iHeIw9rXzUQ Very expensive right now, too much to impulse buy, and the O
247.
by Gracana 2y ago
The article follows the typical pop-sci presentation of "scientists are gobsmacked by thing that's obvious to laymen," but of course it's really not like that. This is the abstract of the paper: Quantitative estimates o
248.
by Gracana 2y ago
Maybe you can meet yourself halfway and write assembly macros to make your life easier.
249.
by Gracana 2y ago
And if you want to use it as a build tool, you're going to have to crack open some books.
250.
by Gracana 2y ago
You can run a quantized version of the model to reduce the memory requirements, and you can do partial offload, where some of the model is on GPU and some is on CPU. If you are running a 70B Q4, that’s 40-ish GB including some context cache
251.
by Gracana 2y ago
Oh! That is an excellent solution. I wish it was that easy in every UI.
252.
by Gracana 2y ago
That's a great idea. In your frontend, do you write in the same text entry field as the bot? I use oobabooga/text-generation-webui and I findit's a little awkward to edit the bot responses.
253.
by Gracana 2y ago
llama.cpp readme still lists multimodal models.. Qwen2-VL and others. Is that inaccurate, or something different? [edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8
254.
by Gracana 2y ago
Thanks. That's probably what I should do next time around: Follow the video exactly and confirm it works before I start tinkering. I wanted to use my choice of model and I think I diverged from the beaten path enough that it caused pro
255.
by Gracana 2y ago
Are you talking about this? https://github.com/Acly/krita-ai-diffusion I tried using that but didn't have much luck. That is, I could get it to generate whole images, but none of the other tools seemed to work. I&
256.
by Gracana 2y ago
Maybe that's what they meant, and it'd be cool if nvidia still offered that on consumer cards, but thankfully you don't need it for LLM inference. The traffic between cards is very small.
257.
by Gracana 2y ago
It's "easy" if you have a place to build an open frame rig with riser cables and whatnot. I can't do that, so I'm going the single slot waterblock route, which unfortunately rules out 3090s due to the memory on the
258.
by Gracana 2y ago
Maybe EPYC can make better use of the available bandwidth, but for comparison I have a water cooled Xeon W5-3435X running at 4.7GHz all-core with 8 channels of DDR5-6400, and CPU inference is still dog slow. With a 70B Q8 model I get 1 tok&
259.
by Gracana 2y ago
Yeah that's fair, I suppose I see that sort of thing on reddit fairly regularly, especially in the "here's a story about my messed-up life" types of subreddits.
260.
by Gracana 2y ago
Yup, I'm the same, and I love my LLMs. They're fun and interesting to talk to and use, but it's obvious to everyone that they're not very reliable. If I think an article is LLM-generated, then the signal I'm getting
261.
by Gracana 2y ago
I was looking to see how you're supposed to configure v3, then realized you're probably using the API, and came across this: > The current version of the deepseek-chat model's Function Calling capabilitity is unstable, whi
262.
by Gracana 2y ago
Replacing the entire soldering station would be quite an overreaction to an issue with the tip.
263.
by Gracana 2y ago
I really want one of those machines, but I will be shocked if they are even "well supported," let alone "best supported."
264.
by Gracana 2y ago
I get what you mean, but look at it. It's horrible.
265.
by Gracana 2y ago
Please don't stop. I want to hear more about "said a meme to my wife."
266.
by Gracana 2y ago
I just found this explanation on the linked site: Many Germans incorrectly believe that copyrights cannot be abandoned. The actual situation in German law is as follows: "Nutzungsrechte" (literally "usage rights") includ
267.
by Gracana 2y ago
Inference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible fo
268.
by Gracana 2y ago
Yup. My first thought was that they look like "BCGs". https://en.wikipedia.org/wiki/GI_glasses
269.
by Gracana 2y ago
The clock redriver uses a phase locked loop to generate a few identical clock signals for use by the DRAM chips on the module. This same tech is also used for clock multipliers and dividers throughout digital systems. Rambus has details on
270.
by Gracana 2y ago
I recall Zoe Quinn wrote about having that done ~10 years ago. Article's still up on her site, it's a good read: http://www.beesgo.biz/magnet.html

More ›