3 ms·
Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was
by zozbot234 2mo ago
Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
- desmondl 2mo agoThe animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
- whimsicalism 2mo agoi assume they meant serially....
- jcims 2mo ago>Answering 2,500 independent questions is an embarrassingly parallel workload So is calculating the total time required to answer all of the questions.
- baq 2mo agoAmadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
- moralestapia 2mo agoSuch a deep and smart insight. But why is that important if they're measuring time?
- moralestapia 2mo agoAnd just like that, GP disappears forever ...
- xyzsparetimexyz 2mo agoYes. but for embarrassingly parallel tasks it barely matters.
- modeless 2mo agoThe task the majority of people here care about (and which makes up a huge part of AI lab revenue these days) is collaborative coding with a single human which is not embarrassingly parallel.
- xyzsparetimexyz 2mo agoI am aware. I was talking about HLE. Talking about how fast you can run through HLE in a serial manner is like talking about how how fast your non-simd single threaded cpu ray tracer is. Its simply not how any sensible person would tackle this problem.
- senordevnyc 2mo ago[dead]
- eru 2mo agoMaybe not embarrassingly parallel, but can be surprisingly parallel.
- deleted 2mo ago[deleted]
- hgoel 2mo agoPresumably when they're talking about benchmarking, they're using a fixed amount of resources, like everyone else does when benchmarking?
- walrus01 2mo ago> Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. I believe the assumption of the benchmark creators is that it's meant to measure the sequential speed that a single instance of the LLM and piece of hardware can get through all the tasks from start to finish, like running a race. As a rudimentary comparison, sort of like doing prime number calculations as a benchmark of the CPUs in one bare metal server. Of course you'd get a speedup in total number of primes searched if you ran the same software of GIMPS on 8 servers with the same hardware in parallel rather than 1 server. Of course if you took all the individual questions in humanity's last exam and fed them in parallel into separate queries to Claude that land on separate hardware instances of the claude model you'd get a speed up. Because each question is independent and not related to knowledge/calculations that are performed in any other question it is indeed very open to speed up by breaking it into separately dispatched parallel tasks.
- rpdillon 2mo ago750t/s tells the story. This isn't purely (or even mostly) driven by parallelization.