4 ms·
Related: "If an AI agent can't figure out how your API works, neither can your users" (from my employer's blog) https://stytch.com/blog/if-an-ai-agent-cant-fig
by bobbiechen 1y ago
Related: "If an AI agent can't figure out how your API works, neither can your users" (from my employer's blog)
https://stytch.com/blog/if-an-ai-agent-cant-figure-out-how-your-api-works-neither-can-your-users/ https://stytch.com/blog/if-an-ai-agent-cant-figure-out-how-y...
- thom 1y agoYeah, I've started to think AI smoke tests for cognitive complexity should be a fundamental part of API/schema design now. Even if you think the LLMs are dumb, Stupidity as a Service is genuinely useful.
- truculent 1y agoIs this you have implemented in practice? Sounds like a great idea, but I have no idea how you would make it work it a structured way (or am I missing the point…?)
- thom 1y agoCan be easy depending on your setup - you can basically just write high level functional tests matching use cases of your API, but as prompts to a system with some sort of tool access, ideally MCP. You want to see those tests pass, but you want them to pass with the simplest possible prompt (a sort of regularization penalty, if you like). You can mutate the prompts using an LLM if you like to try different/shorter phrasings. The Pareto front of passing tests and prompt size/complexity is (arguably) how good a job you're doing structuring/documenting your API.
- truculent 1y agoLovely idea - thanks
- Cthulhu_ 1y agoIt's a good tool to use for code reviewing, especially if you don't have peers with Strong Opinions on it. Which is another issue, indifference. It's hard to find people that actually care about things like API design, let alone multiple that check each other's work. In my experience, a lot of the time people just get lazy and short-circuit the reviews to "oh he knows what he's doing, I'm sure he thought long and hard about this".