4 ms·
The results I get are so-so. The rubric I use to evaluate coding LLM's is to ask it to create a Python script that determines if the contents of a given directo
by seidleroni 3y ago
The results I get are so-so. The rubric I use to evaluate coding LLM's is to ask it to create a Python script that determines if the contents of a given directory have been changed since the last time the script was run. This should be done recursively and handle files being added, removed, or modified and be based off the contents of the files and not the timestamps.
When I ask it as one statement it performed ok, but if I made more specifications with follow-up statements, it kept trying to go down one path even though I told it to do it a different way. A solid start but it definitely needs some improvements, IMO.
- rushingcreek 3y agoThanks for the feedback. We're working on improving consistency and precise instruction following in followups.
- webappguy 3y agoIf you can make this best in class for code outside just human eval, wow, that's the differentiator. Add cursor, replit and vscode support after. But best in class for code, it would be my daily driver
- jiggawatts 3y agoThis is a problem that human programmers screw up… regularly. E.g.: the efficient and robust file change monitoring on Windows is to read the NTFS change journal. For a single process lifetime there are other change notification APIs as well. Most software does neither and is either very slow or misses changes…