9 ms·
I've found this to be one of the most useful ways to use (at least) GPT-4 for programming. Instead of telling it how an API works, I make it guess, maybe start
by kragen 1y ago
I've found this to be one of the most useful ways to use (at least) GPT-4 for programming. Instead of telling it how an API works, I make it guess, maybe starting with some example code to which a feature needs to be added. Sometimes it comes up with a better approach than I had thought of. Then I change the API so that its code works.
Conversely, I sometimes present it with some existing code and ask it what it does. If it gets it wrong, that's a good sign my API is confusing, and how.
These are ways to harness what neural networks are best at: not providing accurate information but making shit up that is highly plausible, "hallucination". Creativity, not logic.
(The best thing about this is that I don't have to spend my time carefully tracking down the bugs GPT-4 has cunningly concealed in its code, which often takes longer than just writing the code the usual way.)
There are multiple ways that an interface can be bad, and being unintuitive is the only one that this will fix. It could also be inherently inefficient or unreliable, for example, or lack composability. The AI won't help with those. But it can make sure your API is guessable and understandable, and that's very valuable.
Unfortunately, this only works with APIs that aren't already super popular.
- deleted 1y ago[deleted]
- golergka 1y agoGreat point. Also, it may not be the best possible API designer in the world, but it sure sounds like a good way to forecast what an _average_ developer would expect this API to look like.
- deleted 1y ago[deleted]
- beefnugs 1y agoComplete insanity, it might change constantly even before a whole new version-retrain Insanity driven development: altering your api to accept 7 levels of "broken and different" structures so as to bend to the will of the llms
- kragen 1y agoYes, that's a bonus. In fact, I've found it worthwhile to prompt it a few times to get several different guesses at how things are supposed to work. The super lazy way is to just say, "No, that's wrong," if necessary adding, "Frotzl2000 doesn't have an enqueueCallback function or even a queue." Of course when it suggests a bad interface you shouldn't implement it.
- fourside 1y agoI think you’re missing the OP’s point. They weren’t saying that the goal is to modify their APIs just to appease an LLM. It’s that they ask LLMs to guess what the API is and use that as part of their design process. If you automatically assume that what the LLM spits out is what the API ought to be then I agree that that’s bad engineering. But if you’re using it to brainstorm what an intuitive interface would look like, that seems pretty reasonable.
- suzzer99 1y ago> Sometimes it comes up with a better approach than I had thought of. IMO this has always been the killer use case for AI—from Google Maps to Grammarly. I discovered Grammarly at the very last phase of writing my book. I accepted maybe 1/3 of its suggestions, which is pretty damn good considering my book had already been edited by me dozens of times AND professionally copy-edited. But if I'd have accepted all of Grammarly's changes, the book would have been much worse. Grammarly is great for sniffing out extra words and passive voice. But it doesn't get writing for humorous effect, context, deliberate repetition, etc. The problem is executives want to completely remove humans from the loop, which almost universally leads to disastrous results.
- normie3000 1y agoWhat's wrong with passive?
- kragen 1y agoSometimes it's used without thinking, and often the writing is made shorter and clearer when the passive voice is removed. But not always; rewriting my previous sentence to name the agents in each case, as the active voice requires in English, would not improve it. (You could remove "made", though.)
- Rayhem 1y ago> Sometimes authors use it without thinking, but removing the passive voice often makes writing shorter and clearer. I don't think mentioning "authors" is absolutely necessary, but I think this is both a faithful attempt to convert this to natural active voice and easier to read/understand.
- kragen 1y agoI agree that it's a perfectly fine way to say it in the active voice, but I think it makes the sentence read slightly worse and require slightly more effort to understand. Certainly it's slightly longer. Possibly it depends on the reader; the passive voice seems to be more of an obstacle for Spanish-as-a-first-language speakers, for example, than for native English speakers.
- afavour 1y agoFrom my perspective that’s fascinatingly upside down thinking that leads to you asking to lose your job. AI is going to get the hang of coding to fill in the spaces (i.e. the part you’re doing) long before it’s able to intelligently design an API. Correct API design requires a lot of contextual information and forward planning for things that don’t exist today. Right now it’s throwing spaghetti at the wall and you’re drawing around it.
- kragen 1y agoMaybe. So far it seems to be a lot better at creative idea generation than at writing correct code, though apparently these "agentic" modes can often get close enough after enough iteration. (I haven't tried things like Cursor yet.) I agree that it's also not currently capable of judging those creative ideas, so I have to do that.
- b112 1y agoThis sort of discourse really grinds my gears. The framing of it, the conceptualization. It's not creative at all, any more than taking the sum of text on a topic, and throwing a dart at it. It's a mild, short step beyond a weighted random, and certainly not capable of any real creativity. Myriads of HN enthusiasts often chime in here "Are humans any more creative" and other blather. Well, that's a whataboutism, and doesn't detract from the fact that creative does not exist in the AI sphere. I agree that you have to judge its output. Also, sorry for hanging my comment here. Might seem over the top, but anytime I see 'creative' and 'AI', I have all sorts of dark thoughts. Dark, brooding thoughts with a sense of deep foreboding.
- kragen 1y agoI understand. I share the foreboding, but I try to subscribe to the converse of Hume's guillotine.
- Dylan16807 1y agoPoint taken but if slushing up half of human knowledge and picking something to fit into the current context isn't creative then humans are rarely creative either.
- bryanlarsen 1y agoI used this to great success just this morning. I told the AI to write me some unit tests. It flailed and failed badly at that task. But how it failed was instructive, and uncovered a bug in the code I wanted to test.
- kragen 1y agoHaha, that's awesome! Are you going to change the interface? What was the bug?
- bryanlarsen 1y agoIt used nonsensical parameters to the API in way that I didn't realize was possible (though obvious in hindsight). The AI got confused; it didn't think the parameters were nonsensical. It also didn't quite use them in the way that triggered the error. However it was close enough for me to realize that "hey, I never though of that possibility". I needed to fix the function to return a proper error response for the nonsense. It also taught me to be more careful about checkpointing my work in git before letting an agent go wild on my codebase. It left a mess trying to fix its problems.
- kragen 1y agoYeah, that's a perfect example of what I'm talking about!
- ChaoPrayaWave 1y agoIn a way, AI’s failure can be its own kind of debugger. By watching where it stumbles, you sometimes spot flaws you’d have missed otherwise.
- momojo 1y agoA light-weight anecdote: Many many python image-processing libraries have an `imread()` function. I didn't know about this when designing our own bespoke image-lib at work, and went with an esoteric `image_get()` that I never bothered to refactor. When I ask ChatGPT for help writing one-off scripts using the internal library I often forget to give it more context than just `import mylib` at the top, and it almost always defaults to `mylib.imread()`.
- kragen 1y agoThat's a perfect example! I wonder if changing it would be an improvement? If you can just replace image_get with imread in all the callers, maybe it would save your team mental effort and/or onboarding time in the future.
- data-ottawa 1y agoI strongly prefer `image_get/image_read` for clarity, but I would just stump in a method called `imread` which is functionally the same and hide it from the documentation.
- dimatura 1y agoI don't know if there's an earlier source, but I'm guessing Matlab originally popularized the `imread` name, and that OpenCV (along with its python wrapper) took it from there, same for scipy. Scikit-image then followed along, presumably.
- bandofthehawk 1y agoAs someone not familiar with these libraries, image_get or image_read seems much clearer to me than imread. I'm wondering if the convention is worse than your instinct in this case. Maybe these AI tools will push us towards conventions that aren't always the best design.
- kragen 1y agoimage_get is clearer—unless you've used Matlab, Octave, matplotlib, SciPy, OpenCV, scikit-learn, or other things that have copied Matlab's interface. In that case, using the established name is clearer. (Unless, on the gripping hand, your image_get function is subtly different from Matlab's imread, for example by not returning an array, in which case a different name might be better.)
- layer8 1y agoHDD — hallucination-driven development
- codingwagie 1y agoThis works for UX. I give it vague requirements, and it implements something i didnt ask for, but is better than i would have thought of
- skygazer 1y agoYou’re fuzzing the API, unusually, before it’s written.
- groestl 1y ago> and being unintuitive is the only one that this will fix That's also how I'm approaching it. If all the condensed common wisdom poured into the model's parameters says that this is how my API is supposed to work to be intuitive, how on earth do I think it should work differently? There needs to be a good reason (like composability, for example). I break expectations otherwise.
- escapecharacter 1y agoThis is similar to an old HCI design technique called Wizard of Oz by the way, where a human operator pretends to be the app that doesn’t exist yet. It’s great for discovering new features. https://en.m.wikipedia.org/wiki/Wizard_of_Oz_experiment https://en.m.wikipedia.org/wiki/Wizard_of_Oz_experiment
- kragen 1y agoI'd never heard that term! Thank you! I feel like LLMs ought to be fantastic at doing that, as well. This is sort of like the inverse.
- data-ottawa 1y agoThis was a big problem starting out writing MCP servers for me. Having an LLM demo your tool, then taking what it does wrong or uses incorrectly and adjusting the API works very very well. Updating the docs to instruct the LLM on how to use your tool does not work well.
- a_e_k 1y agoI've played with a similar idea for writing technical papers. I'll give an LLM my draft and ask it to explain back to me what a section means, or otherwise quiz it about things in the draft. I've found that LLMs can be kind of dumb about understanding things, and are particularly bad at reading between the lines for anything subtle. In this aspect, I find they make good proxies for inattentive anonymous reviewers, and so will try to revise my text until even the LLM can grasp the key points that I'm trying to make.
- kragen 1y agoThat's fantastic! I agree that it's very similar. In both cases, you might get extra bonus usability if the reviewers or the API users actually give your output to the same LLM you used to improve the draft. Or maybe a more harshly quantized version of the same model, so it makes more mistakes.
- djsavvy 1y agohow do prompt it to make it guess about the API for a library? I'm confused how you would structure that in a useful way.
- kragen 1y agoOften I've started with some example code that invokes part of the API, but not all of it. Or in C I can give it the .h file, maybe without comments. Sometimes I can just say, "How do I use the <made-up name> API in Python to do <task>?" Unfortunately the safeguards against hallucinations in more recent models can make this more difficult, because it's more likely to tell me it's never heard of it. You can usually coax it into suspension of disbelief, but I think the results aren't as good.
- slowmovintarget 1y agoThat's not creativity. That's closer to simply observing the mean. For an analogy, it's like waiting to pave a path until people tread the grass in a specific pattern. (Some courtyard designers used to do just that. Wait to see where people were walking first.) Making things easy for Chat GPT means making things close to ordinary, average, or mainstream. Not creative, but can still be valuable.
- sigbottle 1y agoBest way to put it. It's very hard to discuss even slightly unique concepts with GPT. It just keeps strawmanning ideas back to a common consensus without actually understanding the deep idea. On the bright side, a lot of work is just finding the mean solution so.
- ldeian 1y ago> Sometimes it comes up with a better approach than I had thought of. Then I change the API so that its code works. “Sometimes” being a very important qualifier to that statement. Claude 4 naturally doesn’t write code with any kind of long term maintenance in-mind, especially if it’s trying to make things look like what the less experienced developers wrote in the same repo. Please don’t assume just because it looks smart that it is. That will bite you hard. Even with well-intentional rules, terrible things happen. It took me weeks to see some of it.
- rcthompson 1y agoIn a similar vein, some of my colleagues have been feeding their scientific paper methods sections to LLMs and asking them to implement the method in code, using the LLM's degree of success/failure as a vague indicator of the clarity of the method description.
- djtango 1y agoIn essence, a LLM is a crystallisation of a large corpus human opinion and you are using that to focus group your API as it is representative of a reasonable third party perspective?
- junon 1y agoYeah, basically. For example, it's really good at generating critical HN comments. Whenever I have a design or an idea I formulate it to GPT and ask it to generate a bunch of critical HN comments. It usually points out stuff I hadn't considered, or at least prepares me to think about and answer the tough questions.
- eru 1y ago> These are ways to harness what neural networks are best at: not providing accurate information but making shit up that is highly plausible, "hallucination". Creativity, not logic. This is also similar to which areas TD-Gammon excelled at in Backgammon. Which is all pretty amusing, if you compare it to how people usually tended to characterise computers and AI, especially in fiction.
- dotancohen 1y ago> I don't have to spend my time carefully tracking down the bugs GPT-4 has cunningly concealed in its code If anyone is stuck in this situation, give me a holler. My Gmail username is the same as my HN username. I've always been the one to hunt down my coworkers' bugs, and I think I'm the only person on the planet will finds it enjoyable to find ChatGPT'S oversights and sometimes seemingly malicious intent. I'll charge you, don't get me wrong, but I'll save you time, money, and frustration. And future bug reports and security issues.
- visarga 1y agoWhen I see comments like yours I can't help but decry how bad was the "stochastic parrots" framing. A parrot does not hallucinate a better API.
- kragen 1y agoFrom https://tonsky.me/blog/gaslight-driven-development/ https://tonsky.me/blog/gaslight-driven-development/ today: > Any person who has used a computer in the past ten years knows that doing meaningless tasks is just part of the experience. Millions of people create accounts, confirm emails, dismiss notifications, solve captchas, reject cookies, and accept terms and conditions—not because they particularly want to or even need to. They do it because that’s what the computer told them to do. Like it or not, we are already serving the machines. (...) > You might’ve heard a story of Soundslice [adding a feature because ChatGPT kept telling people it exists](https://www.holovaty.com/writing/chatgpt-fake-feature/ https://www.holovaty.com/writing/chatgpt-fake-feature/). We see the same at Instant: for example, we used `tx.update` for both inserting and updating entities, but LLMs kept writing `tx.create` instead. Guess what: we now have `tx.create`, too. > Is it good or is it bad? It definitely feels strange. In a sense, it’s helpful: LLMs here have seen millions of other APIs and are suggesting the most obvious thing, something every developer would think of first, too. > It’s also a unique testing device: if developers use your API wrong, they blame themselves, read the documentation, and fix their code. In the end, you might never learn that they even had the problem. But with ChatGPT, you yourself can experience “newbie’s POV” at any time.