4 ms·
Damn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial
by TechDebtDevin 2y ago
Damn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial task.
- bongodongobob 2y agoMy point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.
- bottom999mottob 2y agoExactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.
- TechDebtDevin 2y agoSure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
- TechDebtDevin 2y agoI see your point but hand holding isn't really a good way to benchmark a models coding capabilities.
- Closi 2y agoDepends if benchmarking is the aim, rather than decreasing the time it takes to build things.
- TechDebtDevin 2y agoWell sure, but that wasn't what we were discussing. The original comment says they use that as their benchmark. While their coding task is a bit complex compared to other benchmarking prompts, it's not that crazy. Here is an example of prompts used for benchmarking with Python for reference: https://huggingface.co/datasets/mbpp?row=98 https://huggingface.co/datasets/mbpp?row=98 At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.
- ben_w 2y agoPrompts like yours (I ask them for a fluid dynamics simulator which also doesn't succeed) inform us of the level they have reached. A useful benchmark, given how many of the formal ones they breeze through. I'm glad they can't quite manage this yet. Means I still have a job.
- Closi 2y agoBreak your prompt up into smaller pieces and it can.
- qeternity 2y agoTaken to the extreme, a sufficiently broken down prompt is simply the code itself. The whole point is to prompt less?
- meiraleal 2y ago> Taken to the extreme, a sufficiently broken down prompt is simply the code itself it is not. But the artifacts generated through the steps will be code. The last prompt will have most of the code supplied to it as the context.
- achierius 2y agoA prompt is just a specification for an output. Code is just what we call a sufficiently detailed specification.
- buddhistdude 2y agoNo he is right, he is saying taken to the extreme. The point is the more and more specific you have to prompt, the more you are actually contributing to the result yourself and the less the model is