menaerus

33 ms·

631.
by menaerus 2y ago
It's not impossible to imagine. Not many LoC and it's a lightweight python layer on top of the stack that actually does the heavy lifting - DuckDB and 3FS.
632.
by menaerus 2y ago
Why do you think medical science wrt complexity is any different than applied math, which computer science essentially is? People already can use LLMs to assist them in diagnosing health issues so why would it be hard to believe that the do
633.
by menaerus 2y ago
Thanks for the clarification on the NVidia, I didn't know that. What I also found is that NVidia groups 32 SIMD lanes into what they call a warp. Then 4 warps are grouped into what they're calling a streaming multiprocessor (SM).
634.
by menaerus 2y ago
I appreciate the response with a lot of interesting details, however, I don't believe it answers the question I had? My doubt was why is it so that the CPU design suffers from clock frequency issues in AVX-512 workloads whereas GPUs wh
635.
by menaerus 2y ago
Yes, I agree that it was premature to say that GPUs aren't suffering from the same symptoms. There's just not enough evidence but the differences in the compute power are still large.
636.
by menaerus 2y ago
1x 512-bit FMA or 2x 256-bit FMAs or 4x 128-bit FMAs is irrelevant here - it's still a single physical unit in a CPU that consumes 512 bits of data bandwidth. The question is why the CPU budget allows for 2x 512-bit or 4x 256-bit while
637.
by menaerus 2y ago
I don't understand why 2x FMAs in CPU design poses such a challenge when GPUs literally have hundreds of such ALUs? Both operate at similar TDP so where's the catch? Much lower GPU clock frequency?
638.
by menaerus 2y ago
This morning I saw RecOps - yes, recruiter ops
639.
by menaerus 2y ago
I think so yes. There's very few engineers that can pull out this type of work IME. In a pool of ~30M SEs around the globe I'd say there's no more than ~30K of such engineers. This is 0.001% and it's a very optimistic nu
640.
by menaerus 2y ago
3FS: 180 nodes, 2x200Gbps InfiniBand and 16x 14TiB NVMe SSDs per node, ~500 clients, 6.6 TiB/s of read throughput with training jobs workload Ceph: 68 nodes, 2x100Gbps Mellanox and 10x 14TiB NVMe SSDs per node, 504 clients, 1TiB/s
641.
by menaerus 2y ago
That would be terrifying in itself if true because for this type of work you really need the best of the best. But I doubt this is the case here. LLMs as we know them today are not quite yet there for this type of work.
642.
by menaerus 2y ago
I must say that I have never thought about that before but that's the reality of the tech market. Many people I talk to, who are outside the tech, can't comprehend this at all - they all assume the same - the more experience and k
643.
by menaerus 2y ago
> The danger is on the other side, you give the AI all your symptoms, it tells you you're fine, nothing is wrong, don't do anything. It goes both ways. Replace "you give the AI" with "you give the doctor" an
644.
by menaerus 2y ago
Not sure if many folks remember this video or have seen it at all but this is what Mark Zuckerberg had a say on young vs experienced engineers. This is I believe from 2011 and I think it perfectly portrays the majority of SV companies to th
645.
by menaerus 2y ago
Of course, business incentive is to have people complete as much stuff as they can per time unit. I don't think I ever had a luck to work in an environment where anyone in the team wouldn't be working on multiple things at the sam
646.
by menaerus 2y ago
> but why is it the case that at 50+ it is harder to find work? I think this begins to be visible even sooner, 38 if you graduated at 23. The majority of the job market requires very very few 15+ years experienced engineers. 5 to 10 year
647.
by menaerus 2y ago
> This isn't really professional - it felt wrong I don't see anything wrong with that tbh. If it works that's great since this means that you're bound by the compute power you have and not the amount of devs available
648.
by menaerus 2y ago
I saw that, I was more interested in other examples you might have since you said that you have started to run two sessions at once to be more productive. And this bug doesn't look like such workload.
649.
by menaerus 2y ago
Can you say more about the type of work? How big are those repositories? Thanks
650.
by menaerus 2y ago
True. I wrote some software that does these calculations for me besides the ones I already have on the paper. I confused two different graphs. So, the final number would be ~0.6 GFLOPS (self-attention across heads) + ~0.15 GFLOPS (attention
651.
by menaerus 2y ago
Sure but that's the total manufacturing CO2 footprint and that's not what I was trying to deduce here. I was trying to understand how much of the problem are we trying to solve with the transition to the EVs. Manufacturing CO2 emi
652.
by menaerus 2y ago
Can you please elaborate what do you mean more precisely? I am not sure I understand. If 5 bln tonnes is total CO2 emissions, how else would you calculate the contribution of petrol/diesel car industry?
653.
by menaerus 2y ago
The source you mention is > according to a report from the European Environment Agency and I don't know how they got to their numbers. Don't have time to read it right now but do you see a mistake in my calculations? I think I
654.
by menaerus 2y ago
Yeah, c/p mistake, thanks for noticing. ~1% it is. > Not that EV are a bad idea From an engineering point of view, if we were to solve the pollution problem, and if the numbers above are not terribly wrong, then the EVs are not an a
655.
by menaerus 2y ago
> How much does the fossile fuel car industry contribute to these numbers? I couldn't find that information. Let's try to deduce this number. According to https://www.acea.auto/files/ACEA-report-vehicles-in
656.
by menaerus 2y ago
> Sadly this isn't true. CO2 emissions from fossil fuels is 38 billion tones in 2023 ( https://ourworldindata.org/co2-emissions ) Oceania: ~0.5 billion tones Aviation: ~1 billion tones Africa: ~1.5 billion tones Europ
657.
by menaerus 2y ago
Let's just summarize the FlashAttention into the following: Att(i) computation without FA runs in O(seq_len*dk + seq_len^2) whereas Att(i) computation with FA runs in O(seq_len^2*dk^2/SRAM_size) Q, K, V computation
658.
by menaerus 2y ago
I am not gonna downvote you but you will need to find your manners. People around and most certainly your colleagues will be grateful for that. Perhaps also learn to deal with the arguments and different opinions without coloring them &quot
659.
by menaerus 2y ago
> Decode is memory bound. > FlashAttention ... such that you can remain compute bound at lower batch sizes during decode. So, which one is it then?
660.
by menaerus 2y ago
Re-computing everything every time is the worst-case scenario and which is why I included it in the example (1k tokens). In that case, KV-cache is obviously set to 0 but it is also obvious that it is a much worse alternative than using the

More ›