menaerus

20 ms·

481.
by menaerus 1y ago
Nonsense but I digress.
482.
by menaerus 1y ago
Which is why I said that it can be roughly estimated. And it can be roughly estimated even without those numbers assuming a fleet of some size X and assuming the number of hours this fleet is utilized per day, for the whole year. Either way
483.
by menaerus 1y ago
You would probably understand if you knew how LLMs are run in the first place but, as ignorant as you are (sorry), I have no interest in debating this with you anymore. I tried to give a tractable clue which you unfortunately chose to count
484.
by menaerus 1y ago
There's one thing you're missing - inference is not cheap. HW is not cheap. Electricity is not cheap. This is w/o R&D. They show that, in average, they recorded ~2.627B requests/day. This is ~79B requests/month
485.
by menaerus 1y ago
I can't wrap my head around on how is it that triggering the L1 HW prefetcher so that it loads two pairs of cache-lines, from L2 into L1, can cause false-sharing. Perhaps what fb experiment measured was the artifact of L2 HW prefetcher
486.
by menaerus 1y ago
No, that's not what I was wondering. Cache-line size being a HW property it is not exactly "configurable", although I guess technically it could be, and I was confused how is it that the Apple sysctl returns 128B, which is a
487.
by menaerus 1y ago
Not sure which part of my comment you were referring to but if it was about the 1TB/s of mem bw it seems it is rather 409GB/s per-socket. From https://www.ibm.com/support/pages/ibm-aix-power10-performanc.
488.
by menaerus 1y ago
> The cited article is pretty clear: the M1 Max maxes out at (approximately) 100 Gb/sec per a single CPU core, 243 Gb/sec per a CPU cluster, and 409 Gb/sec per the entire SOC. So, which part of "We are talking about t
489.
by menaerus 1y ago
Sure, there's a different cost-benefit calculation. My argument was that there is an incentive to optimize for ARM64 because that translates to $$$. It's not only Amazon but Oracle and Microsoft too.
490.
by menaerus 1y ago
We are talking about the memory bandwidth available to the CPU cores and not all the co-processors/accelerators present in the SoC so you're pulling in the argument that is not valid. https://web.archive.org/web&#x
491.
by menaerus 1y ago
Are you serious? But maybe you're not aware how such businesses are run - Linux is not officially supported by Apple and someone has to take the liability when something goes wrong, either you loose your data or your CPU melts down or
492.
by menaerus 1y ago
You're deceiving yourself and falling for Apple marketing. Regardless of a stick or SoC memory, which has been the case with pretty much SoC in 2010's (nowadays I have no idea), it is not possible to drive the memory with such hig
493.
by menaerus 1y ago
> They already are open enough to boot and run Linux, the things that Asahi struggles with are end-user peripherals. That's for entertainment and for geeks such as ourselves but not realistically for hosting a service in a data cent
494.
by menaerus 1y ago
> Apple Silicon beats it by a factor of 5x Really, 1TB/s of memory bandwidth to and from system memory? I don't believe it since that's impossible from HW limits PoV - there's no such DRAM that would allow such perfor
495.
by menaerus 1y ago
But x86_64 does outperform ARM64 in high-performance workloads. High-performance workloads are not single-threaded programs. Maybe if Apple decides one day to manufacture the server CPU, which I believe they will not since they would have t
496.
by menaerus 1y ago
Plenty of interesting details. > Regarding cache-line size, sysctl on macOS reports a value of 128 B, while getconf and the CTR_EL0 register on Asahi Linux returns 64 B, which is also supported by our measurements. How would this be even
497.
by menaerus 1y ago
> On the 48 core system, building linux peaks at about 48GB/s; LLVM peaks at something like 25GB/s LLVM peak is suspiciously low since building LLVM is heavier than the kernel? Anyway, on my machine, which is dual-socket 2x22-c
498.
by menaerus 1y ago
I doubt you can saturate the bandwidth with dual-socket configuration with each having 10 cores. Perhaps if you have very recent cores, which I believe you don't, but Intel design hasn't been that good. What you're also measu
499.
by menaerus 1y ago
It is not wildly wrong, be more respectful please since I am speaking from my own experience. Nowhere in my comment have I used Linux kernel as an example. It's not a great example neither since it's mostly trivial to compile in c
500.
by menaerus 1y ago
One of the biggest reasons why people see so much compilation improvement speed on Apple M chips - massive bandwidth improvement in contrast to other machines, even some older servers. 100G/s single core main memory. It starts to drop,
501.
by menaerus 1y ago
Output as a result is 100mb. Process of compilation accumulates magnitudes more data. Evidence is the constant memory pressure you have in 32G or 64G or even 128G systems. Now given that the process of compilation on even such high end syst
502.
by menaerus 1y ago
You're nitpicking, that's what I meant. Many processes in parallel or many threads in parallel, former will achieve better utilization of memory. Regardless, it doesn't invalidate what I said
503.
by menaerus 1y ago
On 40-core or 64-core machine there's more compute than you will ever need for a compilation process. Compilation is a heavy I/O workload not a heavy compute workload, in most cases, where it actually matters.
504.
by menaerus 1y ago
Chill out. Compiler is a heavily multithreaded program that is utilizing all of the cores in C and C++ compilation model. Since each thread is doing the work, it will obviously also consume memory, no? Computing 101. Total amount of data be
505.
by menaerus 1y ago
> The data I have obtained from practice ranges from 25% to 45%, excluding the build time of third-party libraries, including the standard library. > Online, this number varies widely. The most exaggerated figure I recall is a 26x imp
506.
by menaerus 1y ago
> But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first. We w
507.
by menaerus 1y ago
The fallacy may be in the fact that 98% of the software we write does not necessitate _very specialized_ programmers.
508.
by menaerus 1y ago
Yes, it is core and uncore, which we call a CPU, and you said "But Apple cannot beat Intel/AMD in single-thread performance." which is incorrect for the reasons above.
509.
by menaerus 1y ago
> But Apple cannot beat Intel/AMD in single-thread performance It's literally one of the main Apple M chips advantage over Intel/AMD. At the time when M chip came out, it was the only chip that managed to consume ~100GB&#x
510.
by menaerus 1y ago
The competitive advantage of DeepSeek IMO were the engineers. Some pretty hard-core optimizations went out of their lab, and this is what I think is a major differentiator between success and failure. You can have all the HW you can wish fo

More ›