4 ms·

My company is currently paying through the roof for Datadog. While I'm not certain that it's cheaper to staff a full-time observability platform team, there are

My company is currently paying through the roof for Datadog. While I'm not certain that it's cheaper to staff a full-time observability platform team, there are a lot of open source off-the-shelf TSDB solutions like M3, Timescale, Influx, etc. that should make maintaining an in-house platform less arduous.

jitl 4y ago
How big is your company? I would try to stick with Datadog until 1000+ engineers at least
- subradios 4y ago
  The biggest issue here is that you have to hire real engineers to manage the bought product, and they feel dead ended or just waiting for a layoff opportunity. The institutional knowledge around your data never develops because the engineer whose job it is to manage datadog and observability never gets the opportunity to learn or demonstrate technical depth, and so jumps ship to another team or company literally as soon as possible. SRE recruiters make this worse, making point after point of gaslighting new hires.
  - jitl 4y ago
    I don’t buy your take at all. Isn’t this like arguing we should be running our own Postgres bare-metal instead of relying on RDS because engineers who work with RDS are “dead ended” and a “layoff opportunity” and will never develop institutional knowledge about our database schema? In my experience working on all kinds of infra, I have 1000 things I need to do, and buying a product that lets me avoid worrying implementing 300 of those things myself is a no-brainer.
    - xyzzy_plugh 4y ago
      If you're spending time jumping through hoops with RDS, then it's very possible bare-metal will be easier. I don't believe the same is true today but in the past we did exactly this because we were spending a bunch of time banging our collective heads against some RDS idiosyncrasies and it turned out running our own Postgres was a breeze. Plenty of things can and should just simply be bought. It's not always so cut and dry, though.
  - xyzzy_plugh 4y ago
    All too often the integration costs are overlooked. Do we spend two engineers to build a system or buy a system (and spend two engineers integrating and maintaining it)? I've seen this happen more times than I can count.
bpicolo 4y ago
Wouldn't be so sure. Observability tools usually have the highest traffic, highest data volume, and highest availability requirements of any system a company runs. The times you need them most tend to be the times all of your other systems are failing, so having a shared blast radius is a really bad thing. In any microservices architecture, they're multi-tenant systems with plenty of noisy neighbors. Those factors make running them well both harder and more expensive than running any ol system.
poulsbohemian 4y ago
I did triage / performance consulting for a long time with several products in that space, added features to shipping commercial projects, and worked on a product of my own in that space. While in theory you could build your own tool tailored to your specific organizational needs, I can't really say I'd recommend it. In fact, it wasn't completely uncommon for me to replace whatever homegrown tooling had been built with our kit because of scale, features, better UI / time to resolution tools, etc. If you are just doing some basic app logging, sure - roll your own if you want - but if you are trying to do real-time tiered analysis across something bigger than a few servers, there are commercial products that in this day and age are relatively cheap (or at least a lot more affordable than in the past)
icelancer 4y ago
I've looked into all sorts of APM platforms and have used most. All the free, open-source ones have absolutely nothing on Datadog and New Relic (which is what we use). I wish this wasn't the case, but all the time I spent on Honeycomb and various other OpenTelemetry packages simply can't get it done. Automated APM platforms are simply that much better.
slyall 4y ago
A monitoring system is more than just the TSDB. You've got all those pre-built integrations with everything else. All those log, traces and a nice GUI for your user to actually be able to use them. Currently in the later stages of a Datadog project. Which of course means we are working to get the cost down a little. Modern apps product a lot of logs, traces and metrics. SaaS solutions for monitoring are usually great except for the cost. You can build (I'd advise going with the Grafana stack) but I'd only go down that route if SaaS was way too expensive. Monitoring always takes a long time to setup whatever you do. Manually setting up every app, team, server just takes ages.
geoffbp 4y ago
The elastic stack is pretty good, but you’ll need to either run it on your own cluster/s or pay for the cloud product. It’s likely to be much cheaper than datadog but not quite as good

jitl 4y ago

How big is your company? I would try to stick with Datadog until 1000+ engineers at least

subradios 4y ago
The biggest issue here is that you have to hire real engineers to manage the bought product, and they feel dead ended or just waiting for a layoff opportunity. The institutional knowledge around your data never develops because the engineer whose job it is to manage datadog and observability never gets the opportunity to learn or demonstrate technical depth, and so jumps ship to another team or company literally as soon as possible. SRE recruiters make this worse, making point after point of gaslighting new hires.
- jitl 4y ago
  I don’t buy your take at all. Isn’t this like arguing we should be running our own Postgres bare-metal instead of relying on RDS because engineers who work with RDS are “dead ended” and a “layoff opportunity” and will never develop institutional knowledge about our database schema? In my experience working on all kinds of infra, I have 1000 things I need to do, and buying a product that lets me avoid worrying implementing 300 of those things myself is a no-brainer.
  - xyzzy_plugh 4y ago
    If you're spending time jumping through hoops with RDS, then it's very possible bare-metal will be easier. I don't believe the same is true today but in the past we did exactly this because we were spending a bunch of time banging our collective heads against some RDS idiosyncrasies and it turned out running our own Postgres was a breeze. Plenty of things can and should just simply be bought. It's not always so cut and dry, though.
- xyzzy_plugh 4y ago
  All too often the integration costs are overlooked. Do we spend two engineers to build a system or buy a system (and spend two engineers integrating and maintaining it)? I've seen this happen more times than I can count.

bpicolo 4y ago

Wouldn't be so sure. Observability tools usually have the highest traffic, highest data volume, and highest availability requirements of any system a company runs. The times you need them most tend to be the times all of your other systems are failing, so having a shared blast radius is a really bad thing. In any microservices architecture, they're multi-tenant systems with plenty of noisy neighbors. Those factors make running them well both harder and more expensive than running any ol system.

poulsbohemian 4y ago

I did triage / performance consulting for a long time with several products in that space, added features to shipping commercial projects, and worked on a product of my own in that space. While in theory you could build your own tool tailored to your specific organizational needs, I can't really say I'd recommend it. In fact, it wasn't completely uncommon for me to replace whatever homegrown tooling had been built with our kit because of scale, features, better UI / time to resolution tools, etc. If you are just doing some basic app logging, sure - roll your own if you want - but if you are trying to do real-time tiered analysis across something bigger than a few servers, there are commercial products that in this day and age are relatively cheap (or at least a lot more affordable than in the past)

icelancer 4y ago

I've looked into all sorts of APM platforms and have used most. All the free, open-source ones have absolutely nothing on Datadog and New Relic (which is what we use). I wish this wasn't the case, but all the time I spent on Honeycomb and various other OpenTelemetry packages simply can't get it done. Automated APM platforms are simply that much better.

slyall 4y ago

A monitoring system is more than just the TSDB. You've got all those pre-built integrations with everything else. All those log, traces and a nice GUI for your user to actually be able to use them. Currently in the later stages of a Datadog project. Which of course means we are working to get the cost down a little. Modern apps product a lot of logs, traces and metrics. SaaS solutions for monitoring are usually great except for the cost. You can build (I'd advise going with the Grafana stack) but I'd only go down that route if SaaS was way too expensive. Monitoring always takes a long time to setup whatever you do. Manually setting up every app, team, server just takes ages.

geoffbp 4y ago

The elastic stack is pretty good, but you’ll need to either run it on your own cluster/s or pay for the cloud product. It’s likely to be much cheaper than datadog but not quite as good