menaerus

48 ms·

841.
by menaerus 2y ago
> There is an argument, particularly from C++ people, that the CPU architectures designed after C++ 11 shipped with the Acquire/Release model have all chosen to provide features that can do Acquire/Release smoothly. I never hea
842.
by menaerus 2y ago
With all due respect, you have to figure this out since not many people are going to pay $700 for half of what a $100 device can do. Honest advice not a critique. It must be hard not to read the negativity from some comments but this is alr
843.
by menaerus 2y ago
> What will replace Internet data for training? Curated synthetic datasets? Perhaps a different take at this could be: if I wanted to train "state law" LLM that is exceedingly good in interpreting state law, what are the obstac
844.
by menaerus 2y ago
Scalability issue in OP exceptions is because their code is throwing a lot of exceptions from multiple threads at the same time and therefore causing resource contention in libgcc unwind implementation. Since you have only shown a simple ex
845.
by menaerus 2y ago
Hardware vendors adhering to C++ memory model to design their chips? I think you got it backwards or I misunderstood your point. Memory barriers exist because of how cache memory hierarchy and coherency protocols are implemented in multi-co
846.
by menaerus 2y ago
CPU design makes it inherently hard. C or C++ is just a thin layer above it making no tradeoffs. If you can live with the tradeoff then Rust land or VM-language land is more appropriate.
847.
by menaerus 2y ago
As always, and generally speaking, if anything they show that drawing the line is difficult since they show in their paper [1] that both std::expected and boost::LEAF are worse than std::exception. Excerpts from paper: > Single threaded
848.
by menaerus 2y ago
But that's low-hanging fruit "optimization", no? Once you get around it, and this has been forever, the bottleneck is in the backend generation. So if Rust has already solved this, the area where they can improve most is the
849.
by menaerus 2y ago
It's not really novel nor super-optimized. Many inference engines are out there. But interesting it is. Probably a good learning material too.
850.
by menaerus 2y ago
> more like ~$16k for 16 3090s I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000. > The parts are expensive Now that we invested ~$30k in GPUs, we only need to find a motherboard that can a
851.
by menaerus 2y ago
Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
852.
by menaerus 2y ago
It depends on your CPU microarchitectural details, on the complexity and size of your binary executable and the workload of your binary. So there's no universal answer to your question but it could very well be "much".
853.
by menaerus 2y ago
Charitable interpretation may be: Back then when the contract of this function was standardized, presumably in C89 which is ~35 years ago, CPUs but also C compilers were not as powerful so wasting an extra couple of CPU cycles to check this
854.
by menaerus 2y ago
Fascinating and scary at the same time. Is this the beginning when intelligence, domain expertise and ability to research becomes commodity?
855.
by menaerus 2y ago
Thanks. For example, if I feed it with the 10 MLoC repository, how long does it take before it can start working through the problem? Would it really work well with the mixture of at least C and assembly which you implied it would with Lin
856.
by menaerus 2y ago
How large the repositories are that it can "reason" about?
857.
by menaerus 2y ago
My question wasn't about how to run multiple queries against the LLM but rather how is it even possible from transformer architecture PoV to have a single LLM hosting multiple and different end clients. I'm probably missing someth
858.
by menaerus 2y ago
> The EPYC 9654 costs half as much as the A192-32x Where do you see EPYC 9654 being sold for 3k$? Only price I can find online is about the same as A192-32x, around 5k.
859.
by menaerus 2y ago
How do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
860.
by menaerus 2y ago
Granite Rapids memory bandwidth is between 614 and 844 GB/s.
861.
by menaerus 2y ago
So you're saying that if I have a sentence of 10 words, and I want the LLM to predict the 11th word, FFWD compute is going to be independent of the context size? I don't understand how since that very context is what makes the lik
862.
by menaerus 2y ago
> The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. FFWD input is self-attention output. And sinc
863.
by menaerus 2y ago
> One of the main issues pointed out by TFA is that there's too many small tasks allocated for parallel execution Valid concern but I don't think this was the OP case though? From my understanding, author gained the most benefi
864.
by menaerus 2y ago
Why do you think gaming community survey would be more relevant than Intel/AMD earning reports in which they unambiguously, for the most part, lay out the earnings per CPU type?
865.
by menaerus 2y ago
Exactly what I was thinking too.
866.
by menaerus 2y ago
I don't believe you can get any useable results. Perhaps only for R&D.
867.
by menaerus 2y ago
> I have vague memories of something like this happening with RAM in the late 90 We don't even need to go that far in the history. Crypto hype just few years ago skyrocketed the GPU prices.
868.
by menaerus 2y ago
Yes, I am.
869.
by menaerus 2y ago
We already have something similar in terms of HW accelerators for AI workloads in recent CPU designs but that's not enough. LLM inference workloads are bound by the compute power, sure, but that's not insurmountable IMO. Much bigg
870.
by menaerus 2y ago
From https://www.tomshardware.com/pc-components/cpus/amds-desktop... "When it comes to servers, AMD's share totaled 24.2%" and "Intel, of course, retained its volume lead with a 75.8% unit mark

More ›