6 ms·

I love pydantic but I really have to wonder why they chose this route. There's already a ton of companies that do this really well and I'm trying to figure out

I love pydantic but I really have to wonder why they chose this route. There's already a ton of companies that do this really well and I'm trying to figure out how their solution is any different. The llm integration seems promising but if you care about LLM observability you probably also care about evals, guardrails and a million other things that are very specific to LLM's. Is it possible to build all this under one platform? I do hope I'm wrong for the sake of pydantic-core.

thenipper 2y ago
Agreed. I use pydantic daily and I'm in the position to try this out... but why? It really seems to be a weird divergence from their core product.
- Yenrabbit 2y ago
  They also have a UI framework (fastui). They have funding and devs looking to be useful, I think it's fine that they get to try out new things that work well with their core product.
- deleted 2y ago
  [deleted]
wferrell 2y ago
Who do you like who does this well? I think Pydantic is great software and so I am inclined to see if this too will be great software.
- clintonb 2y ago
  My last company used Splunk and SignalFx. They were fine, but I wasn't necessarily in the weeds with configuration and usage. The Observability Team made it easy to use. My current company, a much small startup, primarily uses Datadog and we are starting to better integrage Honeycomb. We mostly abandoned Google Cloud Monitoring because the UI/UX are not that great. Honeycomb is a paradigm, so took some time for us/me to understand. It's growing on us. Despite the ability to completely blow up our bill (which is now better controlled), Datadog is a good product that lets us quickly answer questions when things go wrong. It's not perfect, but it's the best we have right now. The UI is intuitive, and facilitates more discovery (esp. for metrics and their attributes).
  - cipherself 2y ago
    I am kinda confused by your comment, OP is about tracing as far as I understood, but you're referencing Google Cloud Monitoring (whereas the comparable thing would be Google Cloud Trace) and then again (esp. for metrics and their attributes). but OP isn't about metrics at all rather traces.
    - gazpacho 2y ago
      Hi, I'm one of the developers of Logfire. We do support metrics! Our vision is a single pane of glass observability where you can easily go from metrics <-> traces both via correlation and by creating metrics/dashboards from traces. We also support logging as an integrated concept into tracing (you can emit events without a duration that are just like a log but carry context of where they were emitted from relative to a trace).
    - clintonb 2y ago
      Google Cloud Monitoring (https://console.cloud.google.com/monitoring https://console.cloud.google.com/monitoring), perhaps not what you call it, supports the big three signals—logs, traces, metrics.
- emmanueloga_ 2y ago
  Looks like "Pydantic Logfire" is another entry on the category of "APM"s? [1] Gotta echo the sentiment that Logfire doesn't seem to be too closely related to Pydantic... Also, afaict it looks like the frontend is not open source, unless I'm missing something [2]. So, not a tool that one could self-host? -- 1: https://github.com/topics/apm https://github.com/topics/apm 2: https://github.com/pydantic/logfire/ https://github.com/pydantic/logfire/
  - threecheese 2y ago
    I’ve observed that Pydantic - which we’ve used for years in our API stack - has become very popular in LLM applications, for its type-adjacent features. It serves as a foundational technology for prompting libraries like [DSPy](https://github.com/stanfordnlp/dspy https://github.com/stanfordnlp/dspy) which are abstracting “up the stack” of LLM apps. (some opinions there) Operating AI apps reveals a big challenge, in that debugging probabilistic code paths requires more than the usual introspective abilities, and in an environment where function calls can have very real monetary impact we have to be able to see what’s happening in the runtime. See LangChain’s hosted solution (can’t recall the name) that allows an operator to see prompts and responses “on the wire”. (It just occurred to me that Langchain and Pydantic have a lot in common here, in approach.) Having a coupling between Pydantic - which is *just about* the data layer itself - and an observability tool seems very interesting to me, and having this come from the folks who built it does not seem unreasonable. WRT open source and monetization, I would be lying if I said I wasn’t a little worried - given the recent few months - but I am choosing to see this in a positive light, given this team’s “believability weight” (to overuse Dalio) and history of delivering solid and really useful tooling.
- OutOfHere 2y ago
  This is more a service than it is software. It can't independently be used as a software without relying on their service.
scolvin 2y ago
Pydantic creator here. I understand why this might be your reaction, but let me just share my thoughts: Can we build all the LLM things people want? Yes I think so, early feedback is that Logfire is already much more comprehensive than LLM specific solutions. How is our solution any different? AFAIK: * no one else offers opinionated wrapper for OTel to make it nicer to use, but with all the advantages * no one else does auto-tracing (basically profiling) using import hooks the way we do * no one else has dedicated instrumentation for OpenAI or Pydantic * no one else provides metadata about python objects the way we do so we can render a semi-faithful repr of complex objects (Pydantic models, dataclasses, dataframes etc.) in the web UI * no one else (except Sentry, who are doing something different) makes it really easy to start using it * no one else lets you query observability data with SQL In my mind, the obvious alternative would be to do the thing most OSS companies do, and build "Pydantic Cloud", then start adding features to that instead of the open source package. I didn't want to do that, and I don't think our users would like that either. In the end I decided to build Logfire for two reasons: 1. I wanted it, and have wanted it for years 2. I think building a commercial product like this, then using OSS to spread the word and drive adoption is a really exciting way to incentivize us to make the open source as good as possible — good as-in permissive and good as-in well maintained. And it means that our commercial success is not in tension with the adoption of our open source, which has been a recurring issue for companies trying to capitalize on their open source brand.
- hakanderyal 2y ago
  Useful product, great marketing site. This is obviously a labor of love. Keep it up! Sadly I've moved on from Python world. Funny anecdote, using Pydantic everywhere to improve maintainability made me realize I'm fighting an uphill battle with Python and I should move to a statically typed language, so I switched to C#. Thanks for your work.
  - sirsinsalot 2y ago
    Honestly, after spending the last 10 years developing in Python, I'm considering doing something similar. I've lived the problems. They suck for engineering software with longevity.

thenipper 2y ago

Agreed. I use pydantic daily and I'm in the position to try this out... but why? It really seems to be a weird divergence from their core product.

Yenrabbit 2y ago
They also have a UI framework (fastui). They have funding and devs looking to be useful, I think it's fine that they get to try out new things that work well with their core product.
deleted 2y ago
[deleted]

wferrell 2y ago

Who do you like who does this well? I think Pydantic is great software and so I am inclined to see if this too will be great software.

clintonb 2y ago
My last company used Splunk and SignalFx. They were fine, but I wasn't necessarily in the weeds with configuration and usage. The Observability Team made it easy to use. My current company, a much small startup, primarily uses Datadog and we are starting to better integrage Honeycomb. We mostly abandoned Google Cloud Monitoring because the UI/UX are not that great. Honeycomb is a paradigm, so took some time for us/me to understand. It's growing on us. Despite the ability to completely blow up our bill (which is now better controlled), Datadog is a good product that lets us quickly answer questions when things go wrong. It's not perfect, but it's the best we have right now. The UI is intuitive, and facilitates more discovery (esp. for metrics and their attributes).
- cipherself 2y ago
  I am kinda confused by your comment, OP is about tracing as far as I understood, but you're referencing Google Cloud Monitoring (whereas the comparable thing would be Google Cloud Trace) and then again (esp. for metrics and their attributes). but OP isn't about metrics at all rather traces.
  - gazpacho 2y ago
    Hi, I'm one of the developers of Logfire. We do support metrics! Our vision is a single pane of glass observability where you can easily go from metrics <-> traces both via correlation and by creating metrics/dashboards from traces. We also support logging as an integrated concept into tracing (you can emit events without a duration that are just like a log but carry context of where they were emitted from relative to a trace).
  - clintonb 2y ago
    Google Cloud Monitoring (https://console.cloud.google.com/monitoring https://console.cloud.google.com/monitoring), perhaps not what you call it, supports the big three signals—logs, traces, metrics.
emmanueloga_ 2y ago
Looks like "Pydantic Logfire" is another entry on the category of "APM"s? [1] Gotta echo the sentiment that Logfire doesn't seem to be too closely related to Pydantic... Also, afaict it looks like the frontend is not open source, unless I'm missing something [2]. So, not a tool that one could self-host? -- 1: https://github.com/topics/apm https://github.com/topics/apm 2: https://github.com/pydantic/logfire/ https://github.com/pydantic/logfire/
- threecheese 2y ago
  I’ve observed that Pydantic - which we’ve used for years in our API stack - has become very popular in LLM applications, for its type-adjacent features. It serves as a foundational technology for prompting libraries like [DSPy](https://github.com/stanfordnlp/dspy https://github.com/stanfordnlp/dspy) which are abstracting “up the stack” of LLM apps. (some opinions there) Operating AI apps reveals a big challenge, in that debugging probabilistic code paths requires more than the usual introspective abilities, and in an environment where function calls can have very real monetary impact we have to be able to see what’s happening in the runtime. See LangChain’s hosted solution (can’t recall the name) that allows an operator to see prompts and responses “on the wire”. (It just occurred to me that Langchain and Pydantic have a lot in common here, in approach.) Having a coupling between Pydantic - which is *just about* the data layer itself - and an observability tool seems very interesting to me, and having this come from the folks who built it does not seem unreasonable. WRT open source and monetization, I would be lying if I said I wasn’t a little worried - given the recent few months - but I am choosing to see this in a positive light, given this team’s “believability weight” (to overuse Dalio) and history of delivering solid and really useful tooling.
OutOfHere 2y ago
This is more a service than it is software. It can't independently be used as a software without relying on their service.

scolvin 2y ago

Pydantic creator here. I understand why this might be your reaction, but let me just share my thoughts: Can we build all the LLM things people want? Yes I think so, early feedback is that Logfire is already much more comprehensive than LLM specific solutions. How is our solution any different? AFAIK: * no one else offers opinionated wrapper for OTel to make it nicer to use, but with all the advantages * no one else does auto-tracing (basically profiling) using import hooks the way we do * no one else has dedicated instrumentation for OpenAI or Pydantic * no one else provides metadata about python objects the way we do so we can render a semi-faithful repr of complex objects (Pydantic models, dataclasses, dataframes etc.) in the web UI * no one else (except Sentry, who are doing something different) makes it really easy to start using it * no one else lets you query observability data with SQL In my mind, the obvious alternative would be to do the thing most OSS companies do, and build "Pydantic Cloud", then start adding features to that instead of the open source package. I didn't want to do that, and I don't think our users would like that either. In the end I decided to build Logfire for two reasons: 1. I wanted it, and have wanted it for years 2. I think building a commercial product like this, then using OSS to spread the word and drive adoption is a really exciting way to incentivize us to make the open source as good as possible — good as-in permissive and good as-in well maintained. And it means that our commercial success is not in tension with the adoption of our open source, which has been a recurring issue for companies trying to capitalize on their open source brand.

hakanderyal 2y ago
Useful product, great marketing site. This is obviously a labor of love. Keep it up! Sadly I've moved on from Python world. Funny anecdote, using Pydantic everywhere to improve maintainability made me realize I'm fighting an uphill battle with Python and I should move to a statically typed language, so I switched to C#. Thanks for your work.
- sirsinsalot 2y ago
  Honestly, after spending the last 10 years developing in Python, I'm considering doing something similar. I've lived the problems. They suck for engineering software with longevity.