menaerus

30 ms·

661.
by menaerus 2y ago
> This does sound like you are saying that memory access time does NOT dominate during the decode phase. But it does. Let's take llama3-8B for an example. GFLOPS needed for self-attention per-layer per-token is roughly 0.15 GFLOPS.
662.
by menaerus 2y ago
> But your claim on that decoding is compute-bound is plainly wrong. I did not say anything like that? What I said is that FlashAttention and arguably MLA will not make any significant gains in the inference time. And this is true. Also,
663.
by menaerus 2y ago
I don't think I got it backwards, I believe what I said is correct - FA does not improve inference time. From the authors of FlashAttention: > This [decoding] operation has been optimized with FlashAttention (v1 and v2 recently) in
664.
by menaerus 2y ago
Pretty significant improvements. However, my back on the napkin math suggests that MLA, FlashAttention and similar optimizations will provide the benefits only when memory access time dominates the compute in attention implementation? Those
665.
by menaerus 2y ago
I agree that DeepSeek continues to prove themselves as a great example of engineering but the number of job positions requiring this type of knowledge IME is typically very very low so I am not sure if this would be the right advice to foll
666.
by menaerus 2y ago
> Are we doing this? Yes, you're blatantly misrepresenting information and moving goalposts. Right now it has become clear that you're doing this because you're obviously affiliated with ktransformers project. $6k for 8 to
667.
by menaerus 2y ago
Wow, income tax seems to be pretty low in Bulgaria compared to other European countries? For 50k net per year, or ~4k per month, in most other countries you need to earn ~80k EUR per year (gross). In Bulgaria, you need ~57k EUR. For 75k net
668.
by menaerus 2y ago
But you can run and experiment with any model of your liking. And your data does not leave your desktop environment. You can build services. I don't think anybody doing this is doing it to save $20 a month.
669.
by menaerus 2y ago
It might burst but it may take years for that and until then it's going to be a struggle for most people in non-ML engineering jobs. But it might not burst at all? The trends in AI developments for the past few years are a bit concerni
670.
by menaerus 2y ago
I don't disagree with you but a different view that could support the OP hypothesis could be that prior to the AI boom we could see that 90% of the capital went into "normal" software engineering jobs while maybe only 10% of
671.
by menaerus 2y ago
I don't argue their idea, which is sound, but I argue that the cost needed to achieve the claimed performance is not "for a few thousand more" as you stubbornly continue to claim. The math is clear: single-socket ktransformer
672.
by menaerus 2y ago
It is. They will do nothing but distribute the EU taxpayers money into their pockets. Unfortunately.
673.
by menaerus 2y ago
Yes, dual socket Skylake. What's strange about that? Please price it out for us because I still don't see what's cost effective in a system that costs well over $10k and runs at 8 tok/s vs the dual zen4 system for $6k ru
674.
by menaerus 2y ago
> 8 channels of DDR5 4800 will still get you something like 300 GB per second bandwidth. That's the theory. In practice, Sapphire Rapids needs 24-28 cores to hit the 200 GB/s mark and it doesn't go much further than that.
675.
by menaerus 2y ago
To get the rough idea how things work, you can watch the Karpathy's series on YouTube. To get the actual understanding how things work, you will have to read through the papers. You probably want both. Finally, to really understand how
676.
by menaerus 2y ago
I acknowledged in my last comment that the cost doesn't have to be $100k but that it would still be very high if you opted for the workstation design. You're gonna need to add one more CPU to your design, add another 8 memory chan
677.
by menaerus 2y ago
Well, the response really sounded unnatural and llm-ey. If it's not then please take my apology. I write SIMD kernels and the conclusion drawn in the article makes no sense regardless of the fact who wrote it. I don't doubt the ob
678.
by menaerus 2y ago
Why would you slap an LLM response to my question?
679.
by menaerus 2y ago
Point being? Workstations that are fresh on the market and which have comparable performance of the server counterparts still easily cost anywhere between $20k and $40k. At least this is according to Dell workstations last time I looked.
680.
by menaerus 2y ago
You perhaps forgot to mention that for their AMX optimizations to be even feasible you'd need to spend ~$10k for a single CPU, let alone the whole system which is probably ~$100k.
681.
by menaerus 2y ago
I am not sure I understand this and I believe it's wrong and misleading unless I am missing something obvious. Why would it be the case that hand-written SIMD would perform worse than scalar and non-autovectorized code in debug builds?
682.
by menaerus 2y ago
I have no such aspirations. Commenting here when I have time and to learn something and not to prove something to somebody else.
683.
by menaerus 2y ago
6k is not that bad considering that top of the line Apple laptop costs as much. However, I don't have X so unfortunately I can't read the details.
684.
by menaerus 2y ago
You'd still need a fairly large amount of compute power to be able to run DeepSeek R1 locally, no?
685.
by menaerus 2y ago
> I see so many in the software industry cargo-culting "best practices" dogmatically, creating constant churn via useless trendchasing, that they keep reinventing - badly - new ways to do the exact same things as before. That&#
686.
by menaerus 2y ago
No, that's not how it works. The burden of proof is on the one who is claiming the regression or design flaw, and in this case I am not sure what exactly the author can even claim? He did not run a single experiment where he would show
687.
by menaerus 2y ago
No, I actually read the article and I don't think I was spectacularly rude. I was rather just straight to the point. And rather than bitching "omg, why, oh why did a compiler turn a ctr into a function-call here?!" what I thi
688.
by menaerus 2y ago
I usually only have LSP for a single language so coc is not of much use for me. Syntax highlighting rendering engine in vim OTOH could use some speed improvements. I see it drowning sometimes on larger files with sematic syntax highlighting
689.
by menaerus 2y ago
One more entitled developer with a claim that something is slow but can't really prove it because he can't measure it nor does he have any other working code to compare how blazingly fast can it be. Gotcha. Tip to the author: you
690.
by menaerus 2y ago
I am a long-time vim user and haven't had time to watch neovim developments more closely. What would be the incentive to move to Neovim, e.g. what are its best-selling features that vim doesn't have? Genuine question.

More ›