5 ms·

the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting r

by swyx 1y ago

the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs

9283409232 1y ago
People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease.
- closeparen 1y ago
  Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.
  - 9283409232 1y ago
    Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.
    - nemothekid 1y ago
      1. Leadership doesn’t want to burn engineer cycles on undifferentiated features. 2. Management doesn’t get recognized for working on undifferentiated features. 3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs. Saving money “makes” sense but getting people to actually prioritize it is hard.
      - uaas 1y ago
        Well, saving money is a differentiator, and one of the best things an engineer can put on their CVs.
    - deleted 1y ago
      [deleted]
pphysch 1y ago
Most of the complexity in observability is clientside. It is not hard to spin up Grafana and VictoriaMetrics (and now VictoriaLogs) and keep them running. It is not hard to build a Grafana dashboard that correlates data across both metrics and logs sources, and alerting functionality is pretty good now. The "heavy lift" is instrumenting your applications and infrastructure to provide valuable metrics and logs without exceeding a performance budget. I'm skeptical that Datadog actually does much of that heavy-lifting and that they are actually worth the money. You can probably save 10x with same/better outcomes by paying for managed Grafana + managed DBs and a couple FTEs as observability experts.
- lerchmo 1y ago
  You could hire 100 people to manage your timeseries data and save 70%
- ljm 1y ago
  I used to be quite fond of Datadog but after one or two completely surprising bills (thanks to their granular but unintuitive pricing model), I wouldn't recommend it to anybody any more. If I were more cynical I would say the pricing model is designed to be confusing so customers spend more than they need, and this is only made worse by the extreme breadth of the platform now. These days I'd suggest to just suck it up, spin up a Grafana box, and wire up OpenTelemetry.
QuinnyPig 1y ago
This is very well stated.
- swyx 1y ago
  haha thanks Corey, i echo the best (you)
wavemode 1y ago
I wouldn't really say "very often". Occasionally, perhaps. Even from a pure zero-sum mathematical perspective, it can make sense to invest even as much as 2 or 3 months of engineer time on cloud cost savings measures. If the engineer is making $200K, that's a $30000 - $50000 investment. When you see the eye-watering cloud bills many startups have, you would realize that, that investment is peanuts in comparison to the potential savings over the next several years. And then you also have to keep in mind that, these things are usually not actually zero-sum. The engineer could be new, and working on the efficiency project helps them onboard to your stack. It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is, so you actually improve the product by improving the infrastructure. Or it could just be the very common case that there isn't actually a higher-value thing for that engineer to be working on at that time.
- happymellon 1y ago
  > It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is If Jira has taught me anything, it's that ignoring customers when they complain its too slow makes financial sense.

by swyx 1y ago

9283409232 1y ago

People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease.

closeparen 1y ago
Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.
- 9283409232 1y ago
  Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.
  - nemothekid 1y ago
    1. Leadership doesn’t want to burn engineer cycles on undifferentiated features. 2. Management doesn’t get recognized for working on undifferentiated features. 3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs. Saving money “makes” sense but getting people to actually prioritize it is hard.
    - uaas 1y ago
      Well, saving money is a differentiator, and one of the best things an engineer can put on their CVs.
  - deleted 1y ago
    [deleted]

pphysch 1y ago

Most of the complexity in observability is clientside. It is not hard to spin up Grafana and VictoriaMetrics (and now VictoriaLogs) and keep them running. It is not hard to build a Grafana dashboard that correlates data across both metrics and logs sources, and alerting functionality is pretty good now. The "heavy lift" is instrumenting your applications and infrastructure to provide valuable metrics and logs without exceeding a performance budget. I'm skeptical that Datadog actually does much of that heavy-lifting and that they are actually worth the money. You can probably save 10x with same/better outcomes by paying for managed Grafana + managed DBs and a couple FTEs as observability experts.

lerchmo 1y ago
You could hire 100 people to manage your timeseries data and save 70%
ljm 1y ago
I used to be quite fond of Datadog but after one or two completely surprising bills (thanks to their granular but unintuitive pricing model), I wouldn't recommend it to anybody any more. If I were more cynical I would say the pricing model is designed to be confusing so customers spend more than they need, and this is only made worse by the extreme breadth of the platform now. These days I'd suggest to just suck it up, spin up a Grafana box, and wire up OpenTelemetry.

QuinnyPig 1y ago

This is very well stated.

swyx 1y ago
haha thanks Corey, i echo the best (you)

wavemode 1y ago

I wouldn't really say "very often". Occasionally, perhaps. Even from a pure zero-sum mathematical perspective, it can make sense to invest even as much as 2 or 3 months of engineer time on cloud cost savings measures. If the engineer is making $200K, that's a $30000 - $50000 investment. When you see the eye-watering cloud bills many startups have, you would realize that, that investment is peanuts in comparison to the potential savings over the next several years. And then you also have to keep in mind that, these things are usually not actually zero-sum. The engineer could be new, and working on the efficiency project helps them onboard to your stack. It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is, so you actually improve the product by improving the infrastructure. Or it could just be the very common case that there isn't actually a higher-value thing for that engineer to be working on at that time.

happymellon 1y ago
> It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is If Jira has taught me anything, it's that ignoring customers when they complain its too slow makes financial sense.