What we do, admittedly not with chaining, is use OTel tracing end-to-end. Our RAG pipeline is (currently) 42 spans, and the end-to-end of RAG + LLM call + parse

What we do, admittedly not with chaining, is use OTel tracing end-to-end. Our RAG pipeline is (currently) 42 spans, and the end-to-end of RAG + LLM call + parse/validate is 48 spans. That's just a part of the whole trace, though. Our feature is a natural language querying system built onto a querying engine. And so when someone has their input typed out and clicks the button (or hits enter), we trace that user interaction, RAG, LLM call, parse/validate, and queuing up to run against the querying engine. Getting that "whole experience" visibility really helped us iterate in production. All of it was done just by manually instrumenting things with opentelemetry tracing and having our own conventions for naming key:value pairs that we're looking to upstream. It's more work to set up if you're not familiar with opentelemetry, but it's worth it because you can then start plugging in user IDs or really any other interesting data that helps correlate with LLM behavior so you get a full picture of what's going on.