menaerus

36 ms·

871.
by menaerus 2y ago
Last report I read it was ~80% (Intel) vs ~20% (AMD) for PC market. And ~75% (Intel) vs ~25% (AMD) for data center servers.
872.
by menaerus 2y ago
Exactly.
873.
by menaerus 2y ago
Mathematically speaking LLMs have very precise formulation and can be seen as F(context, X0, X1, ..., XP) = next_token. What science behind the LLMs is still lacking is how all these parameters are correlated one to each other and why one s
874.
by menaerus 2y ago
Yeah, I also think that the ~5k price is quite hefty. It's difficult for me to imagine that running sizeable LLMs on commodity/consumer hardware will be possible without another breakthrough in the field. The prices of GPUs I woul
875.
by menaerus 2y ago
Look, I understand that some people are short-sighted and can hardly think out of the box and that is totally fine by me. I don't judge you for being that so I kindly ask you not to judge my question. Learn to give some benefit of the
876.
by menaerus 2y ago
I think that the European investment capital is scarce and optimized for low-risk ideas. The last time I had a look, pre-seed funding from EU funds, but also some private funds IIRC, was laughable 50-60k EUR for a single year. How do you at
877.
by menaerus 2y ago
Did you really not understand rethoric nature of my question and assumed that I can't do 1st grade primary school math?
878.
by menaerus 2y ago
Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day in
879.
by menaerus 2y ago
> My experience actually diverges here. I've had cases where I've done a bunch of optimizations in the 0.5% range, and then when you go and benchmark the system against the version that was three months ago, you actually see a
880.
by menaerus 2y ago
EDIT: sorry for the wall of text but I really find this topic to be quite interesting to discuss. > where I run benchmarks before-and-after but only look at the single relevant function in the profile. That removes a whole lot of noise Y
881.
by menaerus 2y ago
I agree and I am totally happy to say "I tried to measure but the result I found is inconclusive" or "I believe that this is at worst neutral commit - e.g. it won't add any regression". Having spent probably thousan
882.
by menaerus 2y ago
Don't beat yourself about it. My advice would be to get not too crazy about programming languages. Become proficient at one. Pick either C++ or Rust. Don't use HN as a source for making your next moves in career - it is very skewe
883.
by menaerus 2y ago
> In your 5 sample example, you can't determine if there are any outliers. You need more samples I think the same issue is present no matter how many samples we collect. Statistical apparatus of choice may indeed tell us that given
884.
by menaerus 2y ago
> These workarounds might be good enough to detect ~1-5% changes from a baseline of a native, pre-compiled application. and > However this won't be sufficient for many dynamic, JIT-compiled languages which will usually still have
885.
by menaerus 2y ago
Dell PowerEdge XE9680 hosting 8x H100 SXM's with 100+ cores is ~0.5M USD. 40 days * 24 hrs * 0.99 USD/hr * 8 = 7600 USD.
886.
by menaerus 2y ago
... and soon with this type of setting you will arrive at siloing the information. Having all communication public in the channels comes with its cost but everything is as transparent as it can be. Important distinction is that you get to c
887.
by menaerus 2y ago
Right, it's almost certainly a magnitude larger than 32B.
888.
by menaerus 2y ago
Vendor microarchitectural and optimization manuals.
889.
by menaerus 2y ago
It is not. It is more powerful than M1 and M2 and somewhat at the margin with the M3. > M4 delivers up to 1.5x faster CPU performance over the powerful M2 This is from Apple's first public announcement of M4.
890.
by menaerus 2y ago
I don't think it is entirely pointless but what I think it is pointless is to presumably leave so much performance on the table for a device that does not even run on a battery.
891.
by menaerus 2y ago
M4 is Apple's top-of-the-line CPU design atm and this very same CPU is used across all their products and not only Mac Mini's. It's used in MacBook Pro. Is M4 TDP different in MacBook Pro than what it is in Mac Mini?
892.
by menaerus 2y ago
What's the market for Mac minis if not desktop nor laptop?
893.
by menaerus 2y ago
So ... performance-wise results seem to be quite apart from Intel/AMD?
894.
by menaerus 2y ago
> One FMA counts as two floating-point operations: one multiplication and one addition. Right. This is where the discrepancy comes from. I counted FMA as a single FLOP.
895.
by menaerus 2y ago
I believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.
896.
by menaerus 2y ago
> Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. Isn't it that zen4 doesn't have "native" support for AVX-512 but "mi
897.
by menaerus 2y ago
Top of the line AMD zen5 core can sustain ~80GFLOPS@FP64 and ~160GFLOPS@FP32 using AVX-512, 2x FMA units and ~5Ghz of clock frequency. This is way way lower than what you claim M2 Pro is capable of and since I'm comparing it against th
898.
by menaerus 2y ago
I also recently went through the specs and noticed "with sparsity" but I didn't quite understand what it specifically refers to - the premise is that a lot of weights in matmul operations will be zero or insignificant - also
899.
by menaerus 2y ago
> Today these oils, particularly soybean and canola, are the second highest source of calories in the average American diet, and the single highest source of fats Isn't canola oil one of the oils with the fewest saturated fatty acid
900.
by menaerus 2y ago
It's maybe because the assumption about low latency because everything fits in SRAM is not valid? CS-1 had 18G of SRAM, CS-2 extended it to 40G and CS-3 has 44G of SRAM. None of these is sufficient to run the inference of Llama 70B and

More ›