menaerus

34 ms·

751.
by menaerus 2y ago
The numbers were really too good to be true but there's always a chance that there may be something I am not seeing correctly so I wanted to give the benefit of the doubt. Otherwise, the idea is not too crazy regardless of much higher
752.
by menaerus 2y ago
I didn't get the impression that the math around it is that simplistic. The first obvious reason I can think of now is the attention mechanism being used. Both GQA and MQA demand less compute and therefore less bandwidth than MHA.
753.
by menaerus 2y ago
Now go find us the benchmark that shows M1 Max is able to saturate the bus with more GB/s than what I said above.
754.
by menaerus 2y ago
M1 Max for example has ~240 GB/s of memory bandwidth. Do you think this is because they're using 256-bit LPDDR5@6400MT/s part in combination with not sure how many (?) memory channels or because they soldered it to the chip?
755.
by menaerus 2y ago
Apple M1 Max LPDDR6-6400 measured latency is 111 ns so it's not true that soldered RAM is faster than socketed RAM. Source: first figure from https://www.anandtech.com/show/17024/apple-m1-max-performanc...
756.
by menaerus 2y ago
How much bandwidth do we actually need per-token generation? Let's take one open-source model as a starting point since not all models are created the same.
757.
by menaerus 2y ago
I am not sure if that would work well since 8 cores is really small, it can't really scale well wrt to attention head algorithm, and more importantly this particular EPYC part can only achieve ~50% of the theoretical memory bandwidth,
758.
by menaerus 2y ago
48 servers at 60K also does not math to me at all. Even if I consider the second-hand ones, I could hardly find zen4 under ~10K. And this is without GPUs.
759.
by menaerus 2y ago
Hypothetically you could be right and I don't know if "this time will be different" nor am I trying to predict what will happen on the global economic scale. That's out of my reach. My question is rather of much narrower
760.
by menaerus 2y ago
In my 10-person team example, what in your opinion would the company with the rest of the 9 people do once the AI proves its value in that team? Your hypothesis is AFAIU is that the company will just continue to scale because there's a
761.
by menaerus 2y ago
You're taking your opinion to extreme because I don't think anyone is talking about replacing all engineers with a single AI computer doing the work for a one-person mega-corporation. The actual question, which is much more realis
762.
by menaerus 2y ago
Personal experience and also from many people I know around. Previously I would receive a request for an interview every two days or so. Lately, perhaps once a month, if at all. Foundational skills that I have were always scarce on the mark
763.
by menaerus 2y ago
Massive overhiring or not, it's the fact that many (skilled) engineers can't find a job. Many companies were shut off during the past few years and market became oversaturated over the night. Whether AI will help to correct the ma
764.
by menaerus 2y ago
Exactly. My point was rather that (formal) financial degree background isn't quite the necessity. Quite the contrary in practice I think.
765.
by menaerus 2y ago
People developing statistical models that are excercising the financial market at scale are the quants. These people don't come from financial degree background.
766.
by menaerus 2y ago
So not an actual DeepSeek-R1 model but a distilled Qwen or Llama model. From DeepSeek-R1 paper: > As shown in Table 5, simply distilling DeepSeek-R1’s outputs enables the efficient DeepSeekR1-7B (i.e., DeepSeek-R1-Distill-Qwen-7B, abbrev
767.
by menaerus 2y ago
> for ICEs it usually revolves around changing the oil and filters, it doesn't include comprehensive checks of the car's functionality. It actually does. The list of the checks is different and expanded by each new year and one
768.
by menaerus 2y ago
Much apologist to a man who non ambiguously did what he did and saying other people to refrain from talking about it? No, we should absolutely talk about this precedent in recent history regardless of a fact that we are on a technical forum
769.
by menaerus 2y ago
Brakes are not used as much because of regenerative braking and that somehow seems to impair the braking system. I was also interested in what is meant by "steering" - here it seems that the main culprit is the weight of the batte
770.
by menaerus 2y ago
That could be said if same defects and similar results hadn't been found by yet another test from TÜV (Germany). https://www.adac.de/news/tuev-report-2025/
771.
by menaerus 2y ago
> Mold gets about a 10% speedup by using a faster allocator (mimalloc). I've tried using mimalloc with Wild and didn't get any measurable speedup Perhaps it is worth repeating the experiment with heavy MLoC codebases. jmalloc o
772.
by menaerus 2y ago
Exactly. But also even in build-from-scratch use-case when there's a multitude of binaries to be built - think 10s or 100s of (unit, integration, performance) test binaries or utilities that come along with the main release binary etc.
773.
by menaerus 2y ago
That's not how it works because if what you're saying had been true then the self-attention memory complexity would be O(1), e.g. regardless of the batch size. This obviously isn't the case since each batch computation necess
774.
by menaerus 2y ago
Since qwen 2.5 turbo with 1M context size is advertised to be able to crunch ~30k LoC, I guess we can say then that the 32k qwen 2.5 model is capable of ~960 LoC and therefore 32k model with an upper bound of context set to 8k is capable of
775.
by menaerus 2y ago
Ok, so --ctx-size with a value != 0 means that we can override the default model context size. Since for obvious computation cost reasons we cannot use the 32k fresh context per each request, the trick you make is to use the 1k context (bat
776.
by menaerus 2y ago
Interesting approach. Am I correct to understand that you're basically minimizing the latencies and required compute/mem-bw by avoiding the KV cache? And encoding the (local) context in the input tokens instead? I ask this because
777.
by menaerus 2y ago
This obviously depends on the hardware and the shape of the LLM model itself but, generally speaking, it's quite the opposite. The idea of batching is to grow the compute bandwidth per single request, bigger batch sizes with much more
778.
by menaerus 2y ago
Finetuning, as I understand it, is mostly laborious and mostly very boring and exhausting work that is not appealing to many engineers. It can be done by people who have some skills in Python or similar language and who have some background
779.
by menaerus 2y ago
And I said something else?
780.
by menaerus 2y ago
Incorrect. Self-attention is a highly parallel algorithm that makes it a great candidate for being a memory-bound workload once you have enough compute. Both datacenter grade CPUs and GPUs have enough compute to carry out the self-attention

More ›