3 ms·
FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows: Questions are variants of: Refactor the _se
by usaar333 2y ago
FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows:
Questions are variants of:
Refactor the _set_csrf_cookie method in the CsrfViewMiddleware class to be a stand alone, top level function. Name the new function _set_csrf_cookie, exactly the same name as the existing method. Update any existing self._set_csrf_cookie calls to work with the new _set_csrf_cookie function.
- jstummbillig 2y agoAssuming that that is indeed what most of the benchmark does: If the LLMs are as bad as it as the numbers suggest, then it seems like a perfectly good benchmark. I would definitely want them to be able to do stuff like that when I let them write my code.