3 ms·

I belive OpenTelemetry will make distributed tracing mainstream. Traces are the best tool to debug problems, yet as their are expensive (both in dev work and li

I belive OpenTelemetry will make distributed tracing mainstream. Traces are the best tool to debug problems, yet as their are expensive (both in dev work and license) to implement they are way less popular than logs. Historicaly APM (application performance monitoring) also owned collection, vendor locked-in and didn’t evolve price per value. Open-source collection standard allows users to choose or switch to the best vendor. Personally, I recommend Sumo Logic (disclaimer: I work there). We have rich full-text query capabilities on spans with parsing and aggreagations on query time…

heyzk 5y ago
I'm curious, who's your target customer? Should small teams be looking early on or is this more of a mid-sized company thing?
- smarx007 5y ago
  At almost $100 per GB of logs ingested, I think it’s targeted at SV startups.
  - jakozaur 5y ago
    It is ~$93 / month for 1 "GB per day" not per GB.
    - smarx007 5y ago
      Thanks! It was not clear from https://www.sumologic.com/pricing/ https://www.sumologic.com/pricing/
- jakozaur 5y ago
  Though Sumo Logic started by targeting mid-size to enterprise, recently it invests more in self-serve: 1. Free plan includes Tracing. 2. There is monthly self-serve subscription available.
- sjtindell 5y ago
  In my experience Sumo is a mid to large solution. A startup can get by with their own stack to start. Then they might use something managed as they get big, and once they’re big enough to have dedicated teams, they can run something themselves again. Minus storage and data transfer (the bulk of expense) the hardware to run your own stack probably runs $20 a day or less on a cloud provider, and with a small team where communication about standards is easier it takes less engineering time. My professional opinion is that they should then switch to a managed solution like Sumo when the number of services they are supporting grows. Everyone, myself included, underestimates how many full time people you’re going to need to dedicate to running a serious OpenTel/Elk/Prometheus/Jaeger stack yourself. The pain of migration and upgrading, managing indexes and proper whitelists, even just working with teams to get onboarded and get their dashboards/alerts setup right, using spot instances to desperately claw back every dollar…at my current workplace our custom stack was supposed to be cheaper and ended up costing magnitudes more (under control now) during this learning curve. When I start at a new company I’m probably going to suggest we use Sumo, and this is coming from someone who used to say “we don’t buy solutions we build solutions”. Lately I am leaning into managed more and more - I want to spend my time building great dashboards, or analyzing my data, or writing some code for something else entirely. If you run your own stack you are always keeping up not doing the really useful stuff. Getting a working baseline is hard enough, let alone shiny extras. You’ll watch the talks at whatever conference, you’ll read the single line “quick start docs”, it will sound amazing. Then you’ll find yourself deep in months later with a giant data transfer or storage bill and hours of painstaking tuning ahead of you. If you’re really large, then you can go back to running it yourself.
curryst 5y ago
I don't think it will. The pricing is still too high for it to become mainstream. I've worked at places that have those systems and it really does make debugging an incredible amount easier, but Sumo Logic is $93/GB for data. That says with a 30 day retention period, so I'm assuming that's the charge to store a 1 GB of data for a month. It's just too hard of a sell for management. I don't even want to calculate how many times the cost of S3 storage that is. And then add on top of it that you can't gauge how much data you'll generate until you've actually started implementing it. I don't think distributed tracing will become mainstream until there's something like ELK for it that lets you start out in-house. Then people start to see the value, and you have a solid case when maintaining it becomes a full-time job. I haven't seen anything that seems production-ready enough to use at work, but also doesn't involve running an entire Cassandra cluster or something. Is there an implementation that can run using Postgres? I have no doubt that it wouldn't scale well, but I need a foot in the door to prove the value before I can justify paying $93/GB to my director.

heyzk 5y ago

I'm curious, who's your target customer? Should small teams be looking early on or is this more of a mid-sized company thing?

smarx007 5y ago
At almost $100 per GB of logs ingested, I think it’s targeted at SV startups.
- jakozaur 5y ago
  It is ~$93 / month for 1 "GB per day" not per GB.
  - smarx007 5y ago
    Thanks! It was not clear from https://www.sumologic.com/pricing/ https://www.sumologic.com/pricing/
jakozaur 5y ago
Though Sumo Logic started by targeting mid-size to enterprise, recently it invests more in self-serve: 1. Free plan includes Tracing. 2. There is monthly self-serve subscription available.
sjtindell 5y ago
In my experience Sumo is a mid to large solution. A startup can get by with their own stack to start. Then they might use something managed as they get big, and once they’re big enough to have dedicated teams, they can run something themselves again. Minus storage and data transfer (the bulk of expense) the hardware to run your own stack probably runs $20 a day or less on a cloud provider, and with a small team where communication about standards is easier it takes less engineering time. My professional opinion is that they should then switch to a managed solution like Sumo when the number of services they are supporting grows. Everyone, myself included, underestimates how many full time people you’re going to need to dedicate to running a serious OpenTel/Elk/Prometheus/Jaeger stack yourself. The pain of migration and upgrading, managing indexes and proper whitelists, even just working with teams to get onboarded and get their dashboards/alerts setup right, using spot instances to desperately claw back every dollar…at my current workplace our custom stack was supposed to be cheaper and ended up costing magnitudes more (under control now) during this learning curve. When I start at a new company I’m probably going to suggest we use Sumo, and this is coming from someone who used to say “we don’t buy solutions we build solutions”. Lately I am leaning into managed more and more - I want to spend my time building great dashboards, or analyzing my data, or writing some code for something else entirely. If you run your own stack you are always keeping up not doing the really useful stuff. Getting a working baseline is hard enough, let alone shiny extras. You’ll watch the talks at whatever conference, you’ll read the single line “quick start docs”, it will sound amazing. Then you’ll find yourself deep in months later with a giant data transfer or storage bill and hours of painstaking tuning ahead of you. If you’re really large, then you can go back to running it yourself.

curryst 5y ago

I don't think it will. The pricing is still too high for it to become mainstream. I've worked at places that have those systems and it really does make debugging an incredible amount easier, but Sumo Logic is $93/GB for data. That says with a 30 day retention period, so I'm assuming that's the charge to store a 1 GB of data for a month. It's just too hard of a sell for management. I don't even want to calculate how many times the cost of S3 storage that is. And then add on top of it that you can't gauge how much data you'll generate until you've actually started implementing it. I don't think distributed tracing will become mainstream until there's something like ELK for it that lets you start out in-house. Then people start to see the value, and you have a solid case when maintaining it becomes a full-time job. I haven't seen anything that seems production-ready enough to use at work, but also doesn't involve running an entire Cassandra cluster or something. Is there an implementation that can run using Postgres? I have no doubt that it wouldn't scale well, but I need a foot in the door to prove the value before I can justify paying $93/GB to my director.