5 ms·
Cool, you've identified that your prompt is inadequate for the task. 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come
by bongodongobob 2y ago
Cool, you've identified that your prompt is inadequate for the task.
'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'
- TechDebtDevin 2y agoDamn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial task.
- bongodongobob 2y agoMy point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.
- bottom999mottob 2y agoExactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.
- TechDebtDevin 2y agoSure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
- TechDebtDevin 2y agoI see your point but hand holding isn't really a good way to benchmark a models coding capabilities.
- Closi 2y agoDepends if benchmarking is the aim, rather than decreasing the time it takes to build things.
- TechDebtDevin 2y agoWell sure, but that wasn't what we were discussing. The original comment says they use that as their benchmark. While their coding task is a bit complex compared to other benchmarking prompts, it's not that crazy. Here is an example of prompts used for benchmarking with Python for reference: https://huggingface.co/datasets/mbpp?row=98 https://huggingface.co/datasets/mbpp?row=98 At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.
- ben_w 2y agoPrompts like yours (I ask them for a fluid dynamics simulator which also doesn't succeed) inform us of the level they have reached. A useful benchmark, given how many of the formal ones they breeze through. I'm glad they can't quite manage this yet. Means I still have a job.
- Closi 2y agoBreak your prompt up into smaller pieces and it can.
- qeternity 2y agoTaken to the extreme, a sufficiently broken down prompt is simply the code itself. The whole point is to prompt less?
- meiraleal 2y ago> Taken to the extreme, a sufficiently broken down prompt is simply the code itself it is not. But the artifacts generated through the steps will be code. The last prompt will have most of the code supplied to it as the context.
- achierius 2y agoA prompt is just a specification for an output. Code is just what we call a sufficiently detailed specification.
- buddhistdude 2y agoNo he is right, he is saying taken to the extreme. The point is the more and more specific you have to prompt, the more you are actually contributing to the result yourself and the less the model is
- ddavis 2y agoIt's something I know how to do after figuring it out myself and discovering the potential sharp edges, so I've made it into a fun game to test the models. I'd argue that it's a great prompt (to keep using consistently over time) to see the evolution of this wildly accelerating field.
- kergonath 2y agoDo you notice any progress over time?
- AnimalMuppet 2y agoHow is that "putting in wrong figures"? It's a perfectly valid prompt, written in clear, proper English.