11 ms·

Datadog releases Incident Management, Profiler, Error Tracking, and more

167 points by dbenamy 6y ago

deleted 6y ago
[deleted]
mdaniel 6y ago
I rage closed that window for downloading a 720p(!) 2.5MB video showing some swooshing squares that takes up the entirety of the "welcome to this content" section. It's one level of disrespect for my time to eat a huge swath of space with a gif, it's a whole other level to put a movie trailer above your blog content just to make squares dance around In an attempt to rescue this rant with some actual content: one might have interest in the recent series of threads over in r/devops about people's differing experiences between datadog's _product_ and datadog's _organization_. I would guess if there are problems knowing ahead of time how much mere monitoring costs, that "attack surface" grows the more products one attempts to consume from datadog
- alexpapworth 6y ago
  Doesn't bode well for DataDog's Performance Monitoring tool, especially if they don't use it on their own site!
- shoulderfake 6y ago
  you are angry about 2.5mb video, its 2020 pal get a broadband connection
  - jschwartzi 6y ago
    Look buddy sometimes we just want to read some text while we're sitting at a bus stop.
  - vikramkr 6y ago
    depending on where they are that might not be that easy
  - calabin 6y ago
    I just moved to Phoenix from Chicago - even in a major US city I'm forced to use DSL by the apartment complex I live in. I didn't think I'd have to check to see if I can get broadband because it seemed insane to think that it wouldn't be available. Nonetheless, pages with needless weight have become incredibly annoying, inconvenient, and cumbersome. Also, the much of the rest of the US (outside of metro areas) has some really terrible access to decent internet speeds. That's not even to mention anywhere in the rest of the world. So kindly cool it with your needless condescension. It's unnecessary and contributes nothing.
    - mech422 6y ago
      Hey Neighbor! If you can get it... Cox has gigablast available here for about $120/month. I get about 800ish down and 35ish up.
      - calabin 6y ago
        Thanks for the advice! Unfortunately, the apartment complex has an exclusivity agreement with CenturyLink. Cox and others have expressed willingness to pull cable into my unit, but the management company just won't allow it. It's driving me insane. It's unfortunate that this type of practice is even allowed/legal.
        mech422 6y ago
        oh wow...That really does suck. I haven't tried any of the WISPs[0]yet, but maybe they would work? 0 like https://www.airfiber.cc/ https://www.airfiber.cc/ ??
- vosper 6y ago
  > one might have interest in the recent series of threads over in r/devops about people's differing experiences between datadog's _product_ and datadog's _organization_. I briefly tried to find some of these, but I'm not clear on which threads you're referring to. If you have time to post some that'd be interesting to me. The one I did find was someone complaining about being "scammed" but it seemed pretty clear that they'd enabled an extra paid feature, without understanding the pricing, and it didn't really seem like Datadog did anything wrong... So if that's all people are complaining about it doesn't seem very interesting.
  - mdaniel 6y ago
    We may be talking about the same thread: https://old.reddit.com/r/devops/comments/i63xxv/datadog_shady_billing_13000_for_1_month_what/ https://old.reddit.com/r/devops/comments/i63xxv/datadog_shad... but my take away wasn't from the original poster's story, it was from the numerous replies with their own story about the product being fine but the interactions with the business being suboptimal. I also came into that thread having just heard a similar story last week from an non-reddit source. "..., twice is coincidence, three times is enemy action," as the tale goes. Even surfing around the new comments _here_ seems to reinforce that same theme, so whatever the next phase is beyond enemy action, I'm pretty sure we've crossed that threshold
deleted 6y ago
[deleted]
rememberlenny 6y ago
These tools are major. The last company I was at extensively planned error budgets and SLAs, but the tooling was very squishy. Seeing the incident board and tooling around monitoring feels like they hit the nail on the head. I don't know much about Datadog, but it seems like they are building exactly what their customers need.
abledon 6y ago
so is the "error tracking" feature equivalent to Sentry?
- kevsim 6y ago
  Yep, seems very much the same type of thing.
  - scaryclam 6y ago
    Except more expensive...
    - cloudpundit 6y ago
      Didn't they say it was included for free?
      - scaryclam 6y ago
        Not that I can see. The article doesn't say it's free or at no additional cost that I can find, so I'll not assume it's either.
- zeeg 6y ago
  I'd say it's more "how can we ship the simplest version of Sentry possible". Important to know we do much, much more than what they're offering, for far more than just JavaScript. It's more of a checkbox than a real alternative. Probably useful for companies who just want to buy one tool and not worry about if it's actually what their developers want or need.
jeffbee 6y ago
Profiler looks pretty dope. Does anyone know why neither DataDog nor Google Cloud Profiler support C++? Technically Google supports anything if you implement the agent yourself, but none of these tools ship a C++ profiler out of the box.
- mmclean 6y ago
  Regarding Google Cloud Profiler (I'm the PM), this is for a few reasons: - We have good support for Go, Java, JS, and Python, but are still adding a few features for these languages (MUSL support for Alpine just shipped, still need to add heap profiling for Python) - C++ isn't as heavily used by our ops tools customers as Go, Java, JS, and Python - We have several new analysis features in the pipe, and the cost supporting an additional language would slow down the delivery of these There's no lack of desire to add new languages, but we chose to prioritize completing existing language support and new analytics functionality this year. I'm guessing that the other teams making products in this space face similar constraints and made similar tradeoffs.
  - jeffbee 6y ago
    Thanks for your perspective. When I was at dropbox I wrote a quick tool that would convert perf data files to cloud profiler protobufs and upload them to google. I really appreciate that the API exists allowing people to use whatever language they want. I'm not a huge fan of the datadog model where you need their agent and they don't have a documented API.
- dbenamy 6y ago
  I work on Datadog Continuous Profiler. Thanks :-) We picked the initial feature set based on a mix of factors including where we expected a lot of demand, what we use internally so could dogfood, and what we could build a great version of most quickly. We’ve got more languages and more features coming! And feedback on what we’ve got so far or what (else) you want welcome!
stewart87 6y ago
Anyone using their APM? why/ why not?
- mman0114 6y ago
  Using it, and would love to roll it out to more of our services, but cost is the big concern for us. Just to add, I love how easy it is to set up and roll out.
- palijer 6y ago
  Using it on all of our services, we all love it. Troubleshooting and optimizations are ridiculously simply, there isn't much investigation, APM just tells you what is wrong, saves a lot of Dev time and we'll worth the money IMO
  - aeyes 6y ago
    What language? I tried Python Django a year ago and didn't get half the detail NewRelic shows out of the box. Given the much higher price I didn't even bother looking into custom instrumentalization. But a year has passed, maybe I should give it another try since NewRelic is trying to make everything as hard as possible with their new UI.
- SteveNuts 6y ago
  We use it and love it, but self-limit its use because $$$$. We'd be willing to give Datadog more money to use their full suite, but there's no way we could do it at their full price.
- throckmortra 6y ago
  I'm using their nodejs apm product. We have had numerous problems with it due to its dependencies (async_hooks)
  - throckmortra 6y ago
    though tbf, i don't think there is any other way to do apm in nodejs
bdcravens 6y ago
I feel like Datadog has a great product. I only wish I hadn't had such negative experiences with their sales team.
- sam_lowry_ 6y ago
  This is part of the product. They are trying to do the impossible, embrace all customer configurations, and failing badly when it comes to the support.
  - bdcravens 6y ago
    In my experience, it was more than that - it was strong arming by sales.
    - sam_lowry_ 6y ago
      Depends on the sales guys, I guess. We set a hard budget limit and then harassed support and sales to keep us within this limit. Anyway. We used Datadog for almost half a year already. Looks like they are overcomplicating their offer to the point of no-return. Time to look for the next market disruptor.
  - tengbretson 6y ago
    My experience was getting on a call with them where they explained how their new pricing strategy was going to be so much cheaper and great for us and having them present a nice break down showing exactly how. Eventually we asked "say what's this bit at the end here about a marginal cost per metric+tag permutation?" And it was only after this prompting did we find out that given our current and projected usage profile, we were going to end up spending like 20x the quoted price.
- overlordalex 6y ago
  I've never tried Datadog, and I've anti-promoted them at my company because of this. They somehow scraped my work email from somewhere (only company that managed this so far) and started contacting me once a month with a typical sales pitch. First one I ignored. Second one I politely emailed back to not contact me. Third one I went through their website and found a data-privacy reporting link, and seemed to get a person that promised that I would be unsubscribed. Fourth one I reported to their DPO that I was invoking my EU rights and requesting that they delete my email address and any other data about me, they apologised and I didnt get an email the next month!... ...The fifth one I reported to my countries GDPR authorities for unauthorised marketing without consent along with copies of all my attempts to get them to stop. That seems to have done it, but I didn't follow-up if that was the nail in the coffin or if they've just given up
kevsim 6y ago
We used DataDog extensively for metrics at my last job. It was awesome, but terribly expensive. Then people wanted to get logs in the same service, so we started rolling that out and it got even more expensive (multiple times competitors like SumoLogic). Would assume these new features will also come with a hefty pricetag (though for APM it's not like competitors like NewRelic are inexpensive).
- coredog64 6y ago
  Wow. Sumo costs are usually 75-80% of Splunk. Does this allow DataDog to take the crown for “sales rep most likely to arrive in a Bentley?”
- dalyons 6y ago
  interesting at a prev job we switched _to_ ddlogs from sumologic because ddlogs was so much cheaper. Enterprise pricing isnt exactly transparent though, so who knows what goes into it.
  - kevsim 6y ago
    Yeah, we (by me I mean not me but someone else at my company) ran a pretty extensive process getting offers from the various vendors. SumoLogic may have cut us a nice deal since we were already giving them lots of money and they didn't want to lose us.
- gregwebs 6y ago
  I think they have the least expensive logging product available. But you have to 1) Not run the DataDog agent (you can use fluent-bit) 2) Filter logs so they are not indexed (and in some cases not ingested)
  - manigandham 6y ago
    Try LogDNA - it's been the cheapest, fastest and more reliable logging service I've used. https://logdna.com/ https://logdna.com/
allyant 6y ago
Recently implemented datadog at a large financial organisation - it does well at focusing on its core features. We had to create a lot of glue to make it enterprise ready - Active directory sync due to it not supporting groups, permission model is poor. If you wanted anything done raise a support ticket rather than with your sales contact too.
- psionn 6y ago
  I work on authentication and access control at Datadog - we've recently improved the group mapping through SAML and we have a number of features in the permission model that we're excited about. Eager to show it off if you're interested.
staysaasy 6y ago
I continue to be really impressed by Datadog. They're in the very exciting post-IPO enterprise SaaS phase where they still have startup mojo + the resources of a large company + the need to continually grow and expand. I feel for the teams that will need to compete against them as they enter new markets, although I expect that their forays will fail at a fairly high rate (as most product line expansions do).
- BozeWolf 6y ago
  Was reading the whole thread, but got distracted by “post-IPO enterprise SaaS phase”. You won buzzword/corporate bingo ;-)
  - staysaasy 6y ago
    Heh, yeah. If you have a less buzzwordy way to phrase that I'd be curious to hear it ;-), I think that the description is accurate although upon reflection I'm with you that it feels pulled out of corporate hell.
rob-olmos 6y ago
If I'm understanding correctly, currently "RUM Errors" is not "Error Tracking", and "RUM Errors" only records an error if that RUM session is actually profiled/recorded? Ie., you have a RUM sample rate less than 100%, so some errors aren't recorded in RUM Errors? Will that be the same situation for "Error Tracking"? Curious to see what this pricing is.. eg., any client-side errors get auto-included as a RUM session? There's a good amount of bogus browser errors due to incomplete or oddly sandboxed javascript environments.
pain_perdu 6y ago
How does their Incident Management differ to that of Blameless.com which I believe is the current market leader.
mazatta 6y ago
The only Datadog feature announcement I look forward to is "we've solved our stability issues!". I am sick of playing the "is our infrastructure or Datadog's on fire?" game at 3 AM whenever I'm on call. I can only imagine the firefighting their engineering teams are dealing with, but for what you pay, I expect better.
sergiotapia 6y ago
Datadog pricing is ridiculously complex. Prepare to not really know how much you're paying and how much you owe and why. It's the AWS of metrics. If you need an APM I recommend appsignal. Predictable pricing, and easy to use/find the right info you need.
socialist_coder 6y ago
Be prepared to pay an arm and leg once you start putting custom metrics into your app.

167 points by dbenamy 6y ago

deleted 6y ago

[deleted]

I rage closed that window for downloading a 720p(!) 2.5MB video showing some swooshing squares that takes up the entirety of the "welcome to this content" section. It's one level of disrespect for my time to eat a huge swath of space with a gif, it's a whole other level to put a movie trailer above your blog content just to make squares dance around In an attempt to rescue this rant with some actual content: one might have interest in the recent series of threads over in r/devops about people's differing experiences between datadog's _product_ and datadog's _organization_. I would guess if there are problems knowing ahead of time how much mere monitoring costs, that "attack surface" grows the more products one attempts to consume from datadog

alexpapworth 6y ago
Doesn't bode well for DataDog's Performance Monitoring tool, especially if they don't use it on their own site!
shoulderfake 6y ago
you are angry about 2.5mb video, its 2020 pal get a broadband connection
- jschwartzi 6y ago
  Look buddy sometimes we just want to read some text while we're sitting at a bus stop.
- vikramkr 6y ago
  depending on where they are that might not be that easy
- calabin 6y ago
  I just moved to Phoenix from Chicago - even in a major US city I'm forced to use DSL by the apartment complex I live in. I didn't think I'd have to check to see if I can get broadband because it seemed insane to think that it wouldn't be available. Nonetheless, pages with needless weight have become incredibly annoying, inconvenient, and cumbersome. Also, the much of the rest of the US (outside of metro areas) has some really terrible access to decent internet speeds. That's not even to mention anywhere in the rest of the world. So kindly cool it with your needless condescension. It's unnecessary and contributes nothing.
  - mech422 6y ago
    Hey Neighbor! If you can get it... Cox has gigablast available here for about $120/month. I get about 800ish down and 35ish up.
    - calabin 6y ago
      Thanks for the advice! Unfortunately, the apartment complex has an exclusivity agreement with CenturyLink. Cox and others have expressed willingness to pull cable into my unit, but the management company just won't allow it. It's driving me insane. It's unfortunate that this type of practice is even allowed/legal.
      - mech422 6y ago
        oh wow...That really does suck. I haven't tried any of the WISPs[0]yet, but maybe they would work? 0 like https://www.airfiber.cc/ https://www.airfiber.cc/ ??
vosper 6y ago
> one might have interest in the recent series of threads over in r/devops about people's differing experiences between datadog's _product_ and datadog's _organization_. I briefly tried to find some of these, but I'm not clear on which threads you're referring to. If you have time to post some that'd be interesting to me. The one I did find was someone complaining about being "scammed" but it seemed pretty clear that they'd enabled an extra paid feature, without understanding the pricing, and it didn't really seem like Datadog did anything wrong... So if that's all people are complaining about it doesn't seem very interesting.
- mdaniel 6y ago
  We may be talking about the same thread: https://old.reddit.com/r/devops/comments/i63xxv/datadog_shady_billing_13000_for_1_month_what/ https://old.reddit.com/r/devops/comments/i63xxv/datadog_shad... but my take away wasn't from the original poster's story, it was from the numerous replies with their own story about the product being fine but the interactions with the business being suboptimal. I also came into that thread having just heard a similar story last week from an non-reddit source. "..., twice is coincidence, three times is enemy action," as the tale goes. Even surfing around the new comments _here_ seems to reinforce that same theme, so whatever the next phase is beyond enemy action, I'm pretty sure we've crossed that threshold

deleted 6y ago

[deleted]

rememberlenny 6y ago

These tools are major. The last company I was at extensively planned error budgets and SLAs, but the tooling was very squishy. Seeing the incident board and tooling around monitoring feels like they hit the nail on the head. I don't know much about Datadog, but it seems like they are building exactly what their customers need.

abledon 6y ago

so is the "error tracking" feature equivalent to Sentry?

kevsim 6y ago
Yep, seems very much the same type of thing.
- scaryclam 6y ago
  Except more expensive...
  - cloudpundit 6y ago
    Didn't they say it was included for free?
    - scaryclam 6y ago
      Not that I can see. The article doesn't say it's free or at no additional cost that I can find, so I'll not assume it's either.
zeeg 6y ago
I'd say it's more "how can we ship the simplest version of Sentry possible". Important to know we do much, much more than what they're offering, for far more than just JavaScript. It's more of a checkbox than a real alternative. Probably useful for companies who just want to buy one tool and not worry about if it's actually what their developers want or need.

jeffbee 6y ago

Profiler looks pretty dope. Does anyone know why neither DataDog nor Google Cloud Profiler support C++? Technically Google supports anything if you implement the agent yourself, but none of these tools ship a C++ profiler out of the box.

mmclean 6y ago
Regarding Google Cloud Profiler (I'm the PM), this is for a few reasons: - We have good support for Go, Java, JS, and Python, but are still adding a few features for these languages (MUSL support for Alpine just shipped, still need to add heap profiling for Python) - C++ isn't as heavily used by our ops tools customers as Go, Java, JS, and Python - We have several new analysis features in the pipe, and the cost supporting an additional language would slow down the delivery of these There's no lack of desire to add new languages, but we chose to prioritize completing existing language support and new analytics functionality this year. I'm guessing that the other teams making products in this space face similar constraints and made similar tradeoffs.
- jeffbee 6y ago
  Thanks for your perspective. When I was at dropbox I wrote a quick tool that would convert perf data files to cloud profiler protobufs and upload them to google. I really appreciate that the API exists allowing people to use whatever language they want. I'm not a huge fan of the datadog model where you need their agent and they don't have a documented API.
dbenamy 6y ago
I work on Datadog Continuous Profiler. Thanks :-) We picked the initial feature set based on a mix of factors including where we expected a lot of demand, what we use internally so could dogfood, and what we could build a great version of most quickly. We’ve got more languages and more features coming! And feedback on what we’ve got so far or what (else) you want welcome!

stewart87 6y ago

Anyone using their APM? why/ why not?

mman0114 6y ago
Using it, and would love to roll it out to more of our services, but cost is the big concern for us. Just to add, I love how easy it is to set up and roll out.
palijer 6y ago
Using it on all of our services, we all love it. Troubleshooting and optimizations are ridiculously simply, there isn't much investigation, APM just tells you what is wrong, saves a lot of Dev time and we'll worth the money IMO
- aeyes 6y ago
  What language? I tried Python Django a year ago and didn't get half the detail NewRelic shows out of the box. Given the much higher price I didn't even bother looking into custom instrumentalization. But a year has passed, maybe I should give it another try since NewRelic is trying to make everything as hard as possible with their new UI.
SteveNuts 6y ago
We use it and love it, but self-limit its use because $$$$. We'd be willing to give Datadog more money to use their full suite, but there's no way we could do it at their full price.
throckmortra 6y ago
I'm using their nodejs apm product. We have had numerous problems with it due to its dependencies (async_hooks)
- throckmortra 6y ago
  though tbf, i don't think there is any other way to do apm in nodejs

bdcravens 6y ago

I feel like Datadog has a great product. I only wish I hadn't had such negative experiences with their sales team.

sam_lowry_ 6y ago
This is part of the product. They are trying to do the impossible, embrace all customer configurations, and failing badly when it comes to the support.
- bdcravens 6y ago
  In my experience, it was more than that - it was strong arming by sales.
  - sam_lowry_ 6y ago
    Depends on the sales guys, I guess. We set a hard budget limit and then harassed support and sales to keep us within this limit. Anyway. We used Datadog for almost half a year already. Looks like they are overcomplicating their offer to the point of no-return. Time to look for the next market disruptor.
- tengbretson 6y ago
  My experience was getting on a call with them where they explained how their new pricing strategy was going to be so much cheaper and great for us and having them present a nice break down showing exactly how. Eventually we asked "say what's this bit at the end here about a marginal cost per metric+tag permutation?" And it was only after this prompting did we find out that given our current and projected usage profile, we were going to end up spending like 20x the quoted price.
overlordalex 6y ago
I've never tried Datadog, and I've anti-promoted them at my company because of this. They somehow scraped my work email from somewhere (only company that managed this so far) and started contacting me once a month with a typical sales pitch. First one I ignored. Second one I politely emailed back to not contact me. Third one I went through their website and found a data-privacy reporting link, and seemed to get a person that promised that I would be unsubscribed. Fourth one I reported to their DPO that I was invoking my EU rights and requesting that they delete my email address and any other data about me, they apologised and I didnt get an email the next month!... ...The fifth one I reported to my countries GDPR authorities for unauthorised marketing without consent along with copies of all my attempts to get them to stop. That seems to have done it, but I didn't follow-up if that was the nail in the coffin or if they've just given up

kevsim 6y ago

We used DataDog extensively for metrics at my last job. It was awesome, but terribly expensive. Then people wanted to get logs in the same service, so we started rolling that out and it got even more expensive (multiple times competitors like SumoLogic). Would assume these new features will also come with a hefty pricetag (though for APM it's not like competitors like NewRelic are inexpensive).

coredog64 6y ago
Wow. Sumo costs are usually 75-80% of Splunk. Does this allow DataDog to take the crown for “sales rep most likely to arrive in a Bentley?”
dalyons 6y ago
interesting at a prev job we switched _to_ ddlogs from sumologic because ddlogs was so much cheaper. Enterprise pricing isnt exactly transparent though, so who knows what goes into it.
- kevsim 6y ago
  Yeah, we (by me I mean not me but someone else at my company) ran a pretty extensive process getting offers from the various vendors. SumoLogic may have cut us a nice deal since we were already giving them lots of money and they didn't want to lose us.
gregwebs 6y ago
I think they have the least expensive logging product available. But you have to 1) Not run the DataDog agent (you can use fluent-bit) 2) Filter logs so they are not indexed (and in some cases not ingested)
- manigandham 6y ago
  Try LogDNA - it's been the cheapest, fastest and more reliable logging service I've used. https://logdna.com/ https://logdna.com/

allyant 6y ago

Recently implemented datadog at a large financial organisation - it does well at focusing on its core features. We had to create a lot of glue to make it enterprise ready - Active directory sync due to it not supporting groups, permission model is poor. If you wanted anything done raise a support ticket rather than with your sales contact too.

psionn 6y ago
I work on authentication and access control at Datadog - we've recently improved the group mapping through SAML and we have a number of features in the permission model that we're excited about. Eager to show it off if you're interested.

staysaasy 6y ago

I continue to be really impressed by Datadog. They're in the very exciting post-IPO enterprise SaaS phase where they still have startup mojo + the resources of a large company + the need to continually grow and expand. I feel for the teams that will need to compete against them as they enter new markets, although I expect that their forays will fail at a fairly high rate (as most product line expansions do).

BozeWolf 6y ago
Was reading the whole thread, but got distracted by “post-IPO enterprise SaaS phase”. You won buzzword/corporate bingo ;-)
- staysaasy 6y ago
  Heh, yeah. If you have a less buzzwordy way to phrase that I'd be curious to hear it ;-), I think that the description is accurate although upon reflection I'm with you that it feels pulled out of corporate hell.

rob-olmos 6y ago

If I'm understanding correctly, currently "RUM Errors" is not "Error Tracking", and "RUM Errors" only records an error if that RUM session is actually profiled/recorded? Ie., you have a RUM sample rate less than 100%, so some errors aren't recorded in RUM Errors? Will that be the same situation for "Error Tracking"? Curious to see what this pricing is.. eg., any client-side errors get auto-included as a RUM session? There's a good amount of bogus browser errors due to incomplete or oddly sandboxed javascript environments.

pain_perdu 6y ago

How does their Incident Management differ to that of Blameless.com which I believe is the current market leader.

mazatta 6y ago

The only Datadog feature announcement I look forward to is "we've solved our stability issues!". I am sick of playing the "is our infrastructure or Datadog's on fire?" game at 3 AM whenever I'm on call. I can only imagine the firefighting their engineering teams are dealing with, but for what you pay, I expect better.

sergiotapia 6y ago

Datadog pricing is ridiculously complex. Prepare to not really know how much you're paying and how much you owe and why. It's the AWS of metrics. If you need an APM I recommend appsignal. Predictable pricing, and easy to use/find the right info you need.

socialist_coder 6y ago

Be prepared to pay an arm and leg once you start putting custom metrics into your app.