4 ms·

> Y’all, Datadog and Prometheus are the last, best metrics-backed tools that will ever be built. You can’t catch up to them or beat them at that; no one can. Do

> Y’all, Datadog and Prometheus are the last, best metrics-backed tools that will ever be built. You can’t catch up to them or beat them at that; no one can. Do something different. Build for the next generation of software problems, not the last generation. Heard a very similar thing from Plenty Of Fish creator in 2012, I unfortunately believed him; "the dating space was solved". Turns out it never was, and like every space, solutions will keep on changing.

abeppu 2y ago
... is that a good example? I think people who use the dating apps today mostly hate them, and find that they have misaligned incentives and/or encourage poor behavior. There's been generation of other services that shift in popularity (and network effects mean that lots of people shift) but I'm not convinced that this has ever involved delivering a better solution.
- michaelt 2y ago
  People mostly hate observability tooling too. The point isn't that people like or dislike - it's that the fact a system someone in the industry tells you isn't worth even trying to compete with might be replaced a handful of years later.
  - abeppu 2y ago
    The post author didn't claim that no one else would make money or attract customers in metrics-backed tools after Datadog and Prometheus -- but that they were the last and best. The "at that" in "You can’t catch up to them or beat them at that" seems pretty clearly about "best", i.e. quality of the solution. I claim that the in the intervening decade, dating apps have changed but not gotten better which suggests to me that the Plenty of Fish person may have been right, and this example is not convincingly making the point that flockonus wants to make.
- bloodyplonker22 2y ago
  Indeed. I hate to say this, but most people hate the dating apps because they're ugly. The top 10% are getting all the dates on these apps and the rest are left with endless swiping and only likes from scammers, bots, and pig butcherers. Trust me, I know because I'm ugly.
suyash 2y ago
I'll put InfluxDB right up there as well.
phillipcarter 2y ago
IMO dating apps didn't evolve because they got better at matchmaking based on stated preferences in a profile (something Plenty of Fish nailed quite well!), they shifted the paradigm towards swiping on pictures and an evolving matchmaking algorithm based on user interactions within the app. This is sort of what the article is getting at. For the purposes of gathering, aggregating, sending, and analyzing a bunch of metrics, you'll be hard-pressed to beat Datadog at this game. They're extremely good at this and, by virtue of having many teams with tons of smart people on them, have figured out many of the best ways to squeeze as much analysis value as you can with this kind of data. The post is arguing that better observability demands a paradigm shift away from metrics as the source of truth for things, and with that, many more possibilities open up.
FridgeSeal 2y ago
I hear good (but expensive) things about Datadog, and Prometheus is _useful_but I would never call it “the peak”. Configuring it is awful, driving it is awful, the query language is part good, and part broken glass, relabelling is “not actively broken” but it’s far from “sensible, well designed and thoughtful”. Grafana’s whole stack is massively overwrought if you’re self hosting, and rapidly expensive for managed services. The devs often ignore and react aggressively to issues. Improvements to UX or correctness are ignored, denigrated or just outright denied. There’s some really weird design choices around distributed stuff that makes them annoying in my opinion, and there seems to be no intention of ever making that better. Prometheus and worse, Mimir have been some of the most annoying and fragile things I’ve had the displeasure of operating. Prometheus might have been a lot better than what we had before, but I really thing we can do a lot, a lot better than Prometheus, and I see “improved in every way” solutions like Victoria Metrics as direct evidence of that.
- spimmy 2y ago
  i just think that metrics are the right tool for the job when the job is summarizing vast quantities of data. not when the job is understanding complex systems. in order to do that, you need a ton of context and cardinality, etc. i know so many observability engineering teams that spend an outright majority of their time trying to skate the line between "enough cardinality to understand what's happening" but not so much that it bankrupts them. it's the wrong tool for the job. we need something much more like BI for technical data.

abeppu 2y ago

... is that a good example? I think people who use the dating apps today mostly hate them, and find that they have misaligned incentives and/or encourage poor behavior. There's been generation of other services that shift in popularity (and network effects mean that lots of people shift) but I'm not convinced that this has ever involved delivering a better solution.

michaelt 2y ago
People mostly hate observability tooling too. The point isn't that people like or dislike - it's that the fact a system someone in the industry tells you isn't worth even trying to compete with might be replaced a handful of years later.
- abeppu 2y ago
  The post author didn't claim that no one else would make money or attract customers in metrics-backed tools after Datadog and Prometheus -- but that they were the last and best. The "at that" in "You can’t catch up to them or beat them at that" seems pretty clearly about "best", i.e. quality of the solution. I claim that the in the intervening decade, dating apps have changed but not gotten better which suggests to me that the Plenty of Fish person may have been right, and this example is not convincingly making the point that flockonus wants to make.
bloodyplonker22 2y ago
Indeed. I hate to say this, but most people hate the dating apps because they're ugly. The top 10% are getting all the dates on these apps and the rest are left with endless swiping and only likes from scammers, bots, and pig butcherers. Trust me, I know because I'm ugly.

suyash 2y ago

I'll put InfluxDB right up there as well.

phillipcarter 2y ago

IMO dating apps didn't evolve because they got better at matchmaking based on stated preferences in a profile (something Plenty of Fish nailed quite well!), they shifted the paradigm towards swiping on pictures and an evolving matchmaking algorithm based on user interactions within the app. This is sort of what the article is getting at. For the purposes of gathering, aggregating, sending, and analyzing a bunch of metrics, you'll be hard-pressed to beat Datadog at this game. They're extremely good at this and, by virtue of having many teams with tons of smart people on them, have figured out many of the best ways to squeeze as much analysis value as you can with this kind of data. The post is arguing that better observability demands a paradigm shift away from metrics as the source of truth for things, and with that, many more possibilities open up.

FridgeSeal 2y ago

I hear good (but expensive) things about Datadog, and Prometheus is _useful_but I would never call it “the peak”. Configuring it is awful, driving it is awful, the query language is part good, and part broken glass, relabelling is “not actively broken” but it’s far from “sensible, well designed and thoughtful”. Grafana’s whole stack is massively overwrought if you’re self hosting, and rapidly expensive for managed services. The devs often ignore and react aggressively to issues. Improvements to UX or correctness are ignored, denigrated or just outright denied. There’s some really weird design choices around distributed stuff that makes them annoying in my opinion, and there seems to be no intention of ever making that better. Prometheus and worse, Mimir have been some of the most annoying and fragile things I’ve had the displeasure of operating. Prometheus might have been a lot better than what we had before, but I really thing we can do a lot, a lot better than Prometheus, and I see “improved in every way” solutions like Victoria Metrics as direct evidence of that.

spimmy 2y ago
i just think that metrics are the right tool for the job when the job is summarizing vast quantities of data. not when the job is understanding complex systems. in order to do that, you need a ton of context and cardinality, etc. i know so many observability engineering teams that spend an outright majority of their time trying to skate the line between "enough cardinality to understand what's happening" but not so much that it bankrupts them. it's the wrong tool for the job. we need something much more like BI for technical data.