5 ms·
My point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed i
by bongodongobob 2y ago
My point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.
- bottom999mottob 2y agoExactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.
- TechDebtDevin 2y agoSure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
- TechDebtDevin 2y agoI see your point but hand holding isn't really a good way to benchmark a models coding capabilities.
- Closi 2y agoDepends if benchmarking is the aim, rather than decreasing the time it takes to build things.
- TechDebtDevin 2y agoWell sure, but that wasn't what we were discussing. The original comment says they use that as their benchmark. While their coding task is a bit complex compared to other benchmarking prompts, it's not that crazy. Here is an example of prompts used for benchmarking with Python for reference: https://huggingface.co/datasets/mbpp?row=98 https://huggingface.co/datasets/mbpp?row=98 At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.