4 ms·
Thank you for the kind words on the rules/documentation! It was definitely an iterative process to figure out how to get good results. We have an llms.txt and
by nezaj 1y ago
Thank you for the kind words on the rules/documentation! It was definitely an iterative process to figure out how to get good results.
We have an llms.txt and llms-full.txt (~9k lines) which contains all our documentation. Feeding these to the claude didn't get great results, it was just too much information.
We manually compressed our llms-full.txt into a rules file (~1.5k lines) which declared the API upfront and provided snippets of how to do different things with callouts to common examples. This condensed version did better but would cause Claude to make subtle mistakes.
Looking at the kind of mistakes Claude made, it seemed like a human could make those mistakes too (very useful feedback for us to improve our API ). We thought “what's one of the smallest fully contained examples we can make that packs a bunch of info on how to use Instant?” That would probably be useful for both a human and an agent. And indeed it seemed to be the case.
- arscan 1y ago> Looking at the kind of mistakes Claude made, it seemed like a human could make those mistakes too (very useful feedback for us to improve our API ). This is something we've found for our API -- just having LLMs attempt to use it helps us identify things that we haven't documented well or placed enough emphasis on (for things that are critical but are non-obvious or may be drowned out by other less important information). Improvements that help the LLM tend to be good for developers too.
- stopachka 1y agoYes. Fun fact, Instant got the `create` method because of how many times LLMs hallucinated it.
- DaiPlusPlus 1y agoLife imitates art: afar you’re describing there is basically _The Secret_ (I.e. if I wish hard enough for something then eventually it will come true), except it’s LLMs that get wish-fulfilment, not us.
- stopachka 1y agoThis made me chuckle, you're right. In a sense it comes back to us, as LLMs are trained on our intuitions.
- TeMPOraL 1y agoHuh, you've reframed The Secret for me - now I see it being, if you wish hard enough for something then eventually the Universe will make it so, just to shut you up.