I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a

I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.