4 ms·

Typical post deal RIF, but bigger question - what do people think of the "observability" space from here? My sneaking suspicion is that outside vertical integr

Typical post deal RIF, but bigger question - what do people think of the "observability" space from here? My sneaking suspicion is that outside vertical integrations like this.. they are going to see some revenue issues in this downturn. Outside of huge FAANG infra, paid observability seems like a "nice to have" rather than a "must have".. the type of thing you cut before you cut engineering staff, and extend your runway. You can cobble together something worse with FOSS.. And biggest question is - who cares about p99 latencies anyway when your 2 year runway has shrunk to 6 months, ZIRP is over and you are doubtful of your next funding round. I can't imagine another whale like Coinbase writing a $65M/year check to a Datadog at this point... "Things you only see at the top"

toomuchtodo 3y ago
I get to peak into a few different financial services org's IT strategies (and where infosec overlaps) and in every org I have seen ruthless cutting (hundreds of millions of dollars in one case) if there is any hint of overlap or the ability to go without. Have also seen aggressively playing VARs/reseller off of each other to drive down cost. I would not want to be someone attempting to GTM in this space right now unless you are very confident in your value prop and your ability to communicate that your customer absolutely cannot live without your product. Top passed, winter is coming.
- eigenvalue 3y ago
  Yes, and Splunk is likely the most overpriced offering across all vendors, both in terms of absolute cost and in in terms of how overpriced it is relative to how good it is. So easy to see how it would be on the chopping block.
  - postexitus 3y ago
    doubts in datadog
    - eigenvalue 3y ago
      Yes, datadog and newrelic are both crazy expensive. Seems like the space could really use a disruptive competitor with a totally different business model and approach. I think a lot of customers would be OK with higher latency if it could cost 90% less. Would also be cool if you could somehow use your spare on-prem compute to shoulder some of the load to bring the cost to serve down.
- kjs3 3y ago
  in every org I have seen ruthless cutting (hundreds of millions of dollars in one case) if there is any hint of overlap or the ability to go without Can confirm; I'm on a team doing this analysis right now. And, frankly, it's a really good idea. When I started asking questions like "why do we have 4 different proxy technologies solving the same use case", "why are we using an on-prem solution over here and a SaaS over there to do the same thing" and "why are you buying this tool when we have an MSA with one of their competitors that will save us 60% on the licensing", it was enlightening. Occasionally, there was a really good reason ("it has feature X which makes us Y percent more efficient"...ok, that makes sense) but almost always we got something like "I didn't know anyone was using Product D when we purchased Product E". I dunno if it's going to be 100s of millions in savings, but we're definitely going to see 10s of millions. Have also seen aggressively playing VARs/reseller off of each other to drive down cost. Hey now...that's the fun part.
orochimaaru 3y ago
It’s hard to get application performance monitoring (APM) tools using FOSS because of the complexity of writing libraries. You could ideally work most things with FOSS. Use opentelemtry extensions to a time dries DB of your choice. Use elk or opensearch for logs, osquery for Linux, native cloud monitoring tools that come out of the box, etc. Problem is it takes a lot of setup and maintenance. It can bite you back in the worst of times - when you’re trying to work an outage and find out alerting is dead because your fluentbit daemon is hung and no logs came through.
- marcosdumay 3y ago
  Or you can spending a day or two creating enough things to track the "end to end" numbers and debug problems the hard way when they appear. Modern APM is completely out of proportion, and most people that use it do absolutely not get enough benefit for paying for those systems. My impression is this is yet another of those Gartner-created hypes that survive as a meme on the high-management minds and don't survive touching the ground.
  - steveBK123 3y ago
    Worse yet I saw a lot of APM-or-nothing mentality from CTOs & senior management such that we skipped over very simple to implement, basic monitoring that catches the typical issues.. because "APM is coming" and "we hired an SRE" and and and.. People want to chase the bright new thing that Gartner is pushing (its always these guys).
  - geodel 3y ago
    > My impression is this is yet another of those Gartner-created hypes... Bang on! Its part of same buzzword bullshittery of Kubernetes/ Cloud native / Micro services / observability and so on. Create an out of proportion tech stack that no one would understand and then one needs these APMs to monitor "p99 latencies" because who the fuck knows now on how incoming http request will get processed.
    - hughesjj 3y ago
specialp 3y ago
There might be pricing pressure, and a lot of people entering the space, but it is more than a nice to have. If you do not have some sort of SIEM solution, you will pay more for or not be insurable for cybersecurity/ransomware insurance. Implementing observability/SIEM yourself has also become a lot more complicated because the domain of things that have to be observed is increasing. There are many organizations now that have things that need to be monitored in several clouds, infrastructures like k8s, and stuff on prem. So you need a vendor that has the tools to collect this data easily and aggregate it. The startup world and startup whales might be dying now. But there is a giant iceberg of companies that are becoming more hybrid and their systems that used to do things like funnel syslogs and firewall flow logs to a collector are no longer adequate.
- marcosdumay 3y ago
  SIEM like in SolarWinds? Yeah, I imagine insure companies still require this.
- steveBK123 3y ago
  Grumpy old developer devils advocate voice: Why does a lot of the modern tech stack seem designed to make you have to pay for a bunch of other SaaS? Like containers/serverless was supposed to mean "cattle not pets" and it autoscales and autoheals and blah blah blah. And then we break everything into micro services and we need to pay for some SaaS to orchestrate it, and another SaaS to observe/monitor it, and another SaaS to...
  - geodel 3y ago
    These questions are heresies. Just had meeting where discussion was whether system can handle additional 8tps (not 8k tps!) load or they need to stagger it.
alephnerd 3y ago
> what do people think of the "observability" space from here? Splunk and other observability companies (Datadog has been hiring like crazy on the security side) have been pivoting into becoming a SIEM 2.0 for a couple years now. Infra budgets now include Security spend, so an entire generation of Infra companies in segments such as Observability/Application Management (Datadog), Data Platform (Databricks, Snowflake), API Management (Imperva, Wallarm, Postman), and DevTooling (Hashicorp) have begun pivoting into the Security segment. The same thing happened with networking in the 2010s.
- steveBK123 3y ago
  Right I figured Splunks defensive play was to market themselves as infosec which every CTO needs to buy in order to tick a box.
  - alephnerd 3y ago
    Yep. It's a 2 birds-1 stone strategy because Splunk can be used by your Platform team as an APM and by your Infosec team as a SIEM. This way you can file 1 PO and use a smaller portion of your budget. This is was a major reason why Cisco acquired Splunk as their SIEM offering is shit, and there is a massive overlap in customer base.
jeffbee 3y ago
I work at an observability service and have a personal stake in the outcome so you can disregard my opinion I guess. But it seems to me that observability is there to enable you to get along with fewer people. The more stats and logs and profiles you have the fewer human operators you need.
- steveBK123 3y ago
  The savings only really works at scale though right. I've worked in plenty of engineering departments with 200 engineers and essentially 5 operators. You need bare minimum 3 of them for holidays/timezones and key man risk reasons. So you can maybe help a 200 person org save 0.5-1% of their lower paid staff? Is your bill + engineering integration costs going to be less than $200-400k/year?
  - jacobr1 3y ago
    Depends on how much the non-operators are using the data. In SaaS companies it is common for the engineering teams to own the performance of their sub-systems, which entails the need for instrumentation, monitoring, dashboards, alerting and investigative tooling. The trend has been to combine things like logs, distributed tracing, APM, error collection and general metrics collection into the same toolset so that you can more easily cross-coordinate between them when tracking down what is happening in a large system. microservices architecture and "devops" make this a much bigger thing requiring every team to take on some aspect of operational capability.
  - jeffbee 3y ago
    I've worked in 100-person groups where everyone spent all of their time trying to figure out what the hell the system was doing because it did not have tracing or logging.
    - steveBK123 3y ago
      microservices I take it ;-) ?
dehrmann 3y ago
> what do people think of the "observability" space from here? I don't think it's cut entirely, but when you're looking at your cloud spend, it's got a target painted on it. It doesn't produce revenue on its own, so people will ask a lot of "do we really need X" or "can we lower retention."
- pm90 3y ago
  100%. Obs tools are still needed for debugging, incident recovery and just being able to make better products, but there is a lot of optimization going on. The market still does seem big. Datadog is needlessly very expensive and the experience is just ok.
  - robertlagrant 3y ago
    The experience (other than the sales process) is great.
codemac 3y ago
It's a classic "dev tooling" play that struggles in downturns. - Company wants to save money, tells team they're cancelling service that costs them $20k/month to observe their production env. - Dev team starts building small parts they need anyways, chews up significant engineering resources. - Company now pays more for highly custom solution no one can compete with. If the company ends up being successful, they'll probably point to all their custom stuff as why. If they end up failing they'll talk about all the boondoggles they went on. This replays so much.
- steveBK123 3y ago
  Argument to be made that it may be a better outcome anyway. Custom solution no one can compete with. Engineers usually have some wiggle room to build things they find interesting & useful in their 10% time. You build the parts that are most urgent as they come up..
  - hughesjj 3y ago
    Where is this mythical place with 10% time to muck around? Or is that the 10% of after 9-5 free time you're talking about?
Lammy 3y ago
> what do people think of the "observability" space from here? They're all effectively surveillance systems with extra steps. The Network itself is always listening.

toomuchtodo 3y ago

I get to peak into a few different financial services org's IT strategies (and where infosec overlaps) and in every org I have seen ruthless cutting (hundreds of millions of dollars in one case) if there is any hint of overlap or the ability to go without. Have also seen aggressively playing VARs/reseller off of each other to drive down cost. I would not want to be someone attempting to GTM in this space right now unless you are very confident in your value prop and your ability to communicate that your customer absolutely cannot live without your product. Top passed, winter is coming.

eigenvalue 3y ago
Yes, and Splunk is likely the most overpriced offering across all vendors, both in terms of absolute cost and in in terms of how overpriced it is relative to how good it is. So easy to see how it would be on the chopping block.
- postexitus 3y ago
  doubts in datadog
  - eigenvalue 3y ago
    Yes, datadog and newrelic are both crazy expensive. Seems like the space could really use a disruptive competitor with a totally different business model and approach. I think a lot of customers would be OK with higher latency if it could cost 90% less. Would also be cool if you could somehow use your spare on-prem compute to shoulder some of the load to bring the cost to serve down.
kjs3 3y ago
in every org I have seen ruthless cutting (hundreds of millions of dollars in one case) if there is any hint of overlap or the ability to go without Can confirm; I'm on a team doing this analysis right now. And, frankly, it's a really good idea. When I started asking questions like "why do we have 4 different proxy technologies solving the same use case", "why are we using an on-prem solution over here and a SaaS over there to do the same thing" and "why are you buying this tool when we have an MSA with one of their competitors that will save us 60% on the licensing", it was enlightening. Occasionally, there was a really good reason ("it has feature X which makes us Y percent more efficient"...ok, that makes sense) but almost always we got something like "I didn't know anyone was using Product D when we purchased Product E". I dunno if it's going to be 100s of millions in savings, but we're definitely going to see 10s of millions. Have also seen aggressively playing VARs/reseller off of each other to drive down cost. Hey now...that's the fun part.

orochimaaru 3y ago

It’s hard to get application performance monitoring (APM) tools using FOSS because of the complexity of writing libraries. You could ideally work most things with FOSS. Use opentelemtry extensions to a time dries DB of your choice. Use elk or opensearch for logs, osquery for Linux, native cloud monitoring tools that come out of the box, etc. Problem is it takes a lot of setup and maintenance. It can bite you back in the worst of times - when you’re trying to work an outage and find out alerting is dead because your fluentbit daemon is hung and no logs came through.

marcosdumay 3y ago
Or you can spending a day or two creating enough things to track the "end to end" numbers and debug problems the hard way when they appear. Modern APM is completely out of proportion, and most people that use it do absolutely not get enough benefit for paying for those systems. My impression is this is yet another of those Gartner-created hypes that survive as a meme on the high-management minds and don't survive touching the ground.
- steveBK123 3y ago
  Worse yet I saw a lot of APM-or-nothing mentality from CTOs & senior management such that we skipped over very simple to implement, basic monitoring that catches the typical issues.. because "APM is coming" and "we hired an SRE" and and and.. People want to chase the bright new thing that Gartner is pushing (its always these guys).
- geodel 3y ago
  > My impression is this is yet another of those Gartner-created hypes... Bang on! Its part of same buzzword bullshittery of Kubernetes/ Cloud native / Micro services / observability and so on. Create an out of proportion tech stack that no one would understand and then one needs these APMs to monitor "p99 latencies" because who the fuck knows now on how incoming http request will get processed.
  - hughesjj 3y ago

specialp 3y ago

There might be pricing pressure, and a lot of people entering the space, but it is more than a nice to have. If you do not have some sort of SIEM solution, you will pay more for or not be insurable for cybersecurity/ransomware insurance. Implementing observability/SIEM yourself has also become a lot more complicated because the domain of things that have to be observed is increasing. There are many organizations now that have things that need to be monitored in several clouds, infrastructures like k8s, and stuff on prem. So you need a vendor that has the tools to collect this data easily and aggregate it. The startup world and startup whales might be dying now. But there is a giant iceberg of companies that are becoming more hybrid and their systems that used to do things like funnel syslogs and firewall flow logs to a collector are no longer adequate.

marcosdumay 3y ago
SIEM like in SolarWinds? Yeah, I imagine insure companies still require this.
steveBK123 3y ago
Grumpy old developer devils advocate voice: Why does a lot of the modern tech stack seem designed to make you have to pay for a bunch of other SaaS? Like containers/serverless was supposed to mean "cattle not pets" and it autoscales and autoheals and blah blah blah. And then we break everything into micro services and we need to pay for some SaaS to orchestrate it, and another SaaS to observe/monitor it, and another SaaS to...
- geodel 3y ago
  These questions are heresies. Just had meeting where discussion was whether system can handle additional 8tps (not 8k tps!) load or they need to stagger it.

alephnerd 3y ago

> what do people think of the "observability" space from here? Splunk and other observability companies (Datadog has been hiring like crazy on the security side) have been pivoting into becoming a SIEM 2.0 for a couple years now. Infra budgets now include Security spend, so an entire generation of Infra companies in segments such as Observability/Application Management (Datadog), Data Platform (Databricks, Snowflake), API Management (Imperva, Wallarm, Postman), and DevTooling (Hashicorp) have begun pivoting into the Security segment. The same thing happened with networking in the 2010s.

steveBK123 3y ago
Right I figured Splunks defensive play was to market themselves as infosec which every CTO needs to buy in order to tick a box.
- alephnerd 3y ago
  Yep. It's a 2 birds-1 stone strategy because Splunk can be used by your Platform team as an APM and by your Infosec team as a SIEM. This way you can file 1 PO and use a smaller portion of your budget. This is was a major reason why Cisco acquired Splunk as their SIEM offering is shit, and there is a massive overlap in customer base.

jeffbee 3y ago

I work at an observability service and have a personal stake in the outcome so you can disregard my opinion I guess. But it seems to me that observability is there to enable you to get along with fewer people. The more stats and logs and profiles you have the fewer human operators you need.

steveBK123 3y ago
The savings only really works at scale though right. I've worked in plenty of engineering departments with 200 engineers and essentially 5 operators. You need bare minimum 3 of them for holidays/timezones and key man risk reasons. So you can maybe help a 200 person org save 0.5-1% of their lower paid staff? Is your bill + engineering integration costs going to be less than $200-400k/year?
- jacobr1 3y ago
  Depends on how much the non-operators are using the data. In SaaS companies it is common for the engineering teams to own the performance of their sub-systems, which entails the need for instrumentation, monitoring, dashboards, alerting and investigative tooling. The trend has been to combine things like logs, distributed tracing, APM, error collection and general metrics collection into the same toolset so that you can more easily cross-coordinate between them when tracking down what is happening in a large system. microservices architecture and "devops" make this a much bigger thing requiring every team to take on some aspect of operational capability.
- jeffbee 3y ago
  I've worked in 100-person groups where everyone spent all of their time trying to figure out what the hell the system was doing because it did not have tracing or logging.
  - steveBK123 3y ago
    microservices I take it ;-) ?

dehrmann 3y ago

> what do people think of the "observability" space from here? I don't think it's cut entirely, but when you're looking at your cloud spend, it's got a target painted on it. It doesn't produce revenue on its own, so people will ask a lot of "do we really need X" or "can we lower retention."

pm90 3y ago
100%. Obs tools are still needed for debugging, incident recovery and just being able to make better products, but there is a lot of optimization going on. The market still does seem big. Datadog is needlessly very expensive and the experience is just ok.
- robertlagrant 3y ago
  The experience (other than the sales process) is great.

codemac 3y ago

It's a classic "dev tooling" play that struggles in downturns. - Company wants to save money, tells team they're cancelling service that costs them $20k/month to observe their production env. - Dev team starts building small parts they need anyways, chews up significant engineering resources. - Company now pays more for highly custom solution no one can compete with. If the company ends up being successful, they'll probably point to all their custom stuff as why. If they end up failing they'll talk about all the boondoggles they went on. This replays so much.

steveBK123 3y ago
Argument to be made that it may be a better outcome anyway. Custom solution no one can compete with. Engineers usually have some wiggle room to build things they find interesting & useful in their 10% time. You build the parts that are most urgent as they come up..
- hughesjj 3y ago
  Where is this mythical place with 10% time to muck around? Or is that the 10% of after 9-5 free time you're talking about?

Lammy 3y ago

> what do people think of the "observability" space from here? They're all effectively surveillance systems with extra steps. The Network itself is always listening.