3 ms·
Nicely written. If I ever want to teach someone how to write a black box implementation for some kind of software interface, I'll point them to this to get sta
by doodlesdev 9mo ago
Nicely written.
If I ever want to teach someone how to write a black box implementation for some kind of software interface, I'll point them to this to get started. The French café analogy is pretty good. It's also great because Parisians aren't always the nicest people around, just like the servers and clients you'll be working with during your implementation.
I guess nowadays you could also automate some part of the protocol discovery with LLM agents? Has anyone tried this before with any promising results? My idea would be to have a traditional fuzzer poking at the server, but use an LLM agent whenever you get a non-error message or a different error message to attempt the well-crafted request without having to shotgun every possibility under the sun into the server.
- lionkor 9mo agoI feel like using an LLM for this is not a good fit, because it's super difficult to verify whether the knowledge it found is true or made up. LLMs are much better at coming to a conclusion when a human wouldn't be sure at all, and that seems really important here.
- doodlesdev 9mo agoI guess the LLM doesn't need to verify whether what it found is true or made up, but rather just save the request and answer for later, so it can be reviewed by a developer and documented.
- yencabulator 9mo agoIn this case, you verify whether the knowledge was made up by comparing the virtual waiter behaviour to the actual waiter. Having a strong test suite like that is actually the ideal scenario for agentic development. (It still incredibly hard to pull off for real, because of complex stateful protocols and edge cases around timing and transfer sizes. Samba did take 12 years to develop, so even with LLM help you'd probably still be looking at several years.)