4 ms·

LLM observability strikes me as an extremely, extremely crowded space. And YC has funded an enormous number of them. What do you think is the key differentiato

• nirga 3y ago
  Note that these products aren't the same, even though they all fall under the category of observability - similarly to how you'd use Datadog AND Sentry, although both can be called "observability platforms". I do think vendor locking is a key differentiator, which some of the reasons why OpenTelemetry succeeded in the first place. I know that my previous company switched to OpenTelemetry for exactly this reason. You get the flexibility of using any platform you'd want (since we're compatible with OpenTelemetry), so it's not vendor-locking you to a specific platform with specific capabilities. Why use any of the ones you mention - maybe Datadog is enough if your use case is simple? But there are more advantages - you get much more than just observability to the LLM itself - you can see calls to vector DBs, network calls, DB queries, etc. - this can be extremely useful IMO for RAG and autonomous agents for example
• phillipcarter 3y ago
  Just to add to Nir's answer here: Let's say your application takes several steps to build up a prompt dynamically, such as a RAG pipeline. You'll end up with a different prompt for potentially each user, depending on the application. The result is you've likely increased the accuracy of the LLM, but at the expense of understanding the whole system's behavior by introducing more steps upstream of the LLM call. Those steps could be super simple, or they could be (like in our case) dozens of steps that could all potentially fail or have a bug or whatever. And so how do you wrangle all of this in context? You need something like OpenLLMetry that treats a request to an LLM as one of several components that make up a request and/or user experience. Otherwise you're just throwing stuff at the wall, guessing at what could improve stuff (or guessing at what could make an eval score better).