3 ms·
> If f(paper) -> code is the weakest part of the chain, it makes sense to target that. my point is that LLMs are already potentially seeing solution on github,
by riku_iki 2y ago
> If f(paper) -> code is the weakest part of the chain, it makes sense to target that.
my point is that LLMs are already potentially seeing solution on github, so you can't use that benchmark as metric unless there is some explanation.
- kelseyfrog 2y agoHow does that work with knowledge cutoff?
- riku_iki 2y agoIt could work with knowledge cut off if they can reliably guarantee it, and also make sure LLMs are not searching github under the surface.
- kelseyfrog 2y agoWhat's the likelihood that the researchers have done this? It seems fairly easy.
- riku_iki 2y agoI honestly have no idea how OAI researchers can guarantee cut off date for Antropic models for example.