5 ms·
I hate this kind of ephemeral research that will be outdated in a few months. Instead of analyzing "why" at a fundamental level LLMs can or cannot do something,
by behnamoh 2y ago
I hate this kind of ephemeral research that will be outdated in a few months. Instead of analyzing "why" at a fundamental level LLMs can or cannot do something, researchers just show what current models are or are not capable of.
And often times the results are not replicable either.
- tossandthrow 2y agoWell, science moves. It is because of research like this is will be outdated. Great you enjoy more foundational work!
- rgmerk 2y agoThe potential utility in the research is not so much the results but the benchmark itself. I haven’t read the whole thing so I can’t really judge whether this specific benchmark is useful, but if it is, every time a new model comes out they can run the benchmark and breathlessly report its improved performance.
- lolinder 2y ago> However, existing code generation benchmarks primarily focus on general-purpose scenarios, leaving the code generation performance of LLMs for specific application domains largely unknown. In this paper, we introduce a new benchmark, MultiCodeBench, to fill this gap. This is about the benchmark they're introducing, which would have real uses for all subsequent models.