6 ms·

> The problem with me and tracing is nobody uses it. When I monitor the teams usage of traces, it is always a small fraction of the development team that ever l

> The problem with me and tracing is nobody uses it. When I monitor the teams usage of traces, it is always a small fraction of the development team that ever logs in to use them. I don't know why the tool hasn't gotten more popularity among developers. I think there's two primary reasons: 1. Historically it was too hard to get easy value early. That's probably changed with OpenTelemetry since it has so much automatic instrumentation you can install. Even 3 years ago it just wasn't feasible to install an agent or write 5 lines of code and then get most of the libraries you use instrumented for you. Now it is, and it's fully portable across tools. 2. Cost, which still isn't solved yet. Tail-based sampling is still too hard, but you need it if you want to do stuff like always capture 100% of error traces but get only a statistically representative sample of 200 OK traces. There's some solutions for that today (e.g., my employer Honeycomb has such a tool), but it's still locked away with vendors and too hard to use. I do expect the narrative on tracing to change in the coming years, though. OpenTelemetry is growing in popularity and it has enough features and stability for a wide range of people to adopt it. I think the cost issue will also be dealt with over time.

wspeirs 3y ago
>Cost, which still isn't solved yet. I'd argue it _is_ solved... store your logs in S3. At ~$0.02/GB you can store a _lot_ of logs for like $20. The problem is that _most_ solutions (Honeycomb included) are SaaS-based solutions, and so they have to charge a margin on top of whatever provider is charging them. You just need a tool (like log-store.com) that can search through logs when they're stored in S3!
- phillipcarter 3y ago
  At the small volume of data where you could "just put stuff in S3 and search it later", the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else.
  - OldManRyan 3y ago
    Depends on how much you value not being vendor locked in. I'll spend a week of dev time in order to save a month or two in migration pain later down the line when my vendor jacks up cost. See Datadog as an example.
    - phillipcarter 3y ago
      You can have your cake and eat it too with OpenTelemetry (see my original post). I don't see how that has to do with the cost of tracing being a factor in inhibiting its adoption though? I agree that vendor neutral instrumentation is worth it.
      - OldManRyan 3y ago
        > ...the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else. I'm mainly responding to this point regarding developer time outweighing the cost of just going with SaaS vendor that provides that capability. I don't think we disagree with each other though
  - wspeirs 3y ago
    >At the small volume of data where you could "just put stuff in S3 and search it later" I was arguing exactly the opposite... leverage S3's near-infinite storage so you can store whatever you'd like. Searching through it _can_ be fast with the right tool.
- chimeracoder 3y ago
  > You just need a tool (like log-store.com) that can search through logs when they're stored in S3! As someone who has run logging systems professionally, "just" is doing a lot of work in that sentence.
- roflyear 3y ago
  Depends on what you're logging. If you're logging everything it can add up.
- poulsbohemian 3y ago
  I spent a lot of years traveling around the world fixing problem for major companies... logs aren't enough. Not only do you have the issue of auditors who are scared you might put things in logs that shouldn't be there, but there are security / access problems and the very basic problem of "did we log the thing we now need?" Not to say that logs can't be helpful, but they generally aren't enough on their own.
doctorpangloss 3y ago
If I'm going to pay for a vendor APM, who's going to outpace NewRelic? They already do all this stuff - automatic instrumenting, "tail based sampling," etc.
- phillipcarter 3y ago
  In some cased OpenTelemetry already has just as much or more support as the vendor-specific agents. But really the play here is future-proofing. A lot of observability vendors are in the business of offering steep discounts in year one, locking you in with even more proprietary tech, and increasing your bill. When your instrumentation is done with OpenTelemetry, it takes 15 minutes to start sending data to a competitor. As for sampling, it's still not great with New Relic. Their proprietary system lets you sample based on span attributes and combine that with a random sampling. But they don't dynamically upsample or downsample based on arbitrary tags in the data, density of traffic, and a target sampling rate or throughput. They don't let you define rules and combine them with a dynamic system or something else. All of this is necessary once you have a large volume of trace data and don't want to break the bank to observe it. They don't track the sampling rate and re-weight counts on the backend. At lower or medium-volume traffic what they have suffices (as does the OSS solutions), though. This isn't a knock on New Relic -- it's a similar story pretty much anymore. Cost management at any level just isn't there yet for tracing. The building blocks are there and I expect it to be fixed, but it does prevent widespread adoption.
  - doctorpangloss 3y ago
    > All of this is necessary once you have a large volume of trace data and don't want to break the bank to observe it In my opinion, there are two approaches: - I try to do this monitoring as cheaply as possible, which means 100% free software and on premises hardware / minimally used cloud resources. No vendors. - I pay a vendor. And nothing else. From my experience as a SaaS vendor, evaluating paid monitoring options (including NewRelic and Honeycomb) and having used them in the past: Anything more than $0 that doesn't increase the bottom line by a greater asymptotic value is "expensive." When I worked in an organization that is trying to find (1) a marginally cheaper way to do monitoring between two vendors, and therefore (2) predict the future, like how much their usage will be or how much they will grow... the most expensive part of the whole process is all the people working at the organization. That number never seems to go down in response to the choice you make between vendors, it only seems to go down in response to interest rates or catastrophic business externalities. So in my opinion, it's sort of moot which vendor you choose, if you have already crossed the Rubicon to paying vendors. So in an intellectually honest way, no matter how you slice it, NewRelic is "just as expensive" as everything else. > A lot of observability vendors are in the business of offering steep discounts in year one, locking you in with even more proprietary tech, and increasing your bill. The hard, intellectually honest, no-bullshit truth of the matter is that the SaaS vendor's job with pricing is turning whatever the customer likes to hear (upfront invoices and discounts, royalties, per use pricing, whatever) into whatever the investors / owners like to hear (90% margins, hockey stick revenue growth, hockey stick user growth, you can take a long vacation, etc.) via deal term nonsense. For me, for example, I decide I want to make a certain amount of money, and then I turn it into one set of nonsense for customers (a premium fee or up front payment, set up to be economically equivalent) and another set of nonsense for investors (a 90% margin, a hockey stick growth). You might be thinking: this is totally fringe and unorthodox. Listen, go read the Sony emails hack, and see how Michael Lynton deals with this obvious pricing reality. They are a huge, expertly run organization, and all their deals with people were constantly about making up some deal term bullshit the counterparty likes, and then Sony's expert staff of financial modelers gaming out the deal terms to determine the real economic value important to Sony. You don't have to operate in this vacuum as an IC anymore, you can peer directly into how executives actually work at giant corporations, if you dare, and see the realities for yourself. Call me jaded, but having done this long enough, there is no story simpler than "the price is whatever the market will bear, and we'll tell you whatever you want to hear that maximizes it." So the most interesting price point is zero, which is not at all an unorthodox opinion, and indeed, many of the very most disruptive entrepreneurs got to where they were going by valuing their time at zero and rolling out metrics themselves.
prpl 3y ago
Another thing - Traces (and often metrics for that matter) are often treated as something that is investigated one at a time. There's a lot of tools out there to help visualize a trace. They are not, usually, treated as something that should be analyzed at scale, and the tooling there is not great. I think PromScale looked promising for some of those problems but it's also a deprecated product. I think the state of the art is/will be ingesting traces into Iceberg for some form of analysis with whatever execution engine you want (Trino/Spark/etc...). A lot of places are doing this. OpenTelemetry is clearly the future of monitoring, probably with a lot of lifting from Prometheus and other things.
- zug_zug 3y ago
  So what I want is something like Splunk search for traces. I want to be able to write 12-shareable lines of custom-query and immediately get a pretty graph of multidimensional data "What % of users who just registered this month got a 500 more than half the time on this specific path?" and I want it as a pretty graph over time
  - phillipcarter 3y ago
    That's precisely what tools like Honeycomb and Lightstep offer. But most vendors don't have a traces-first approach to querying data.
    - zug_zug 3y ago
      Really? I haven't used honeycomb specifically, but you're saying it supports an arbitrarily complex query depth, with any number of joins essentially?
- 8note 3y ago
  I kinda hacked x-ray onto things, but it does give you an overview aggregate for each node I'm not sure if I've really found it useful though. Im sure if I didn't know the service, itd be great, but I already know where to look for whatever problem
fbdab103 3y ago
>When I monitor the teams usage of traces, it is always a small fraction of the development team that ever logs in to use them... Like fire extinguishers, they are nice to have when you need it.

wspeirs 3y ago

>Cost, which still isn't solved yet. I'd argue it _is_ solved... store your logs in S3. At ~$0.02/GB you can store a _lot_ of logs for like $20. The problem is that _most_ solutions (Honeycomb included) are SaaS-based solutions, and so they have to charge a margin on top of whatever provider is charging them. You just need a tool (like log-store.com) that can search through logs when they're stored in S3!

phillipcarter 3y ago
At the small volume of data where you could "just put stuff in S3 and search it later", the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else.
- OldManRyan 3y ago
  Depends on how much you value not being vendor locked in. I'll spend a week of dev time in order to save a month or two in migration pain later down the line when my vendor jacks up cost. See Datadog as an example.
  - phillipcarter 3y ago
    You can have your cake and eat it too with OpenTelemetry (see my original post). I don't see how that has to do with the cost of tracing being a factor in inhibiting its adoption though? I agree that vendor neutral instrumentation is worth it.
    - OldManRyan 3y ago
      > ...the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else. I'm mainly responding to this point regarding developer time outweighing the cost of just going with SaaS vendor that provides that capability. I don't think we disagree with each other though
- wspeirs 3y ago
  >At the small volume of data where you could "just put stuff in S3 and search it later" I was arguing exactly the opposite... leverage S3's near-infinite storage so you can store whatever you'd like. Searching through it _can_ be fast with the right tool.
chimeracoder 3y ago
> You just need a tool (like log-store.com) that can search through logs when they're stored in S3! As someone who has run logging systems professionally, "just" is doing a lot of work in that sentence.
roflyear 3y ago
Depends on what you're logging. If you're logging everything it can add up.
poulsbohemian 3y ago
I spent a lot of years traveling around the world fixing problem for major companies... logs aren't enough. Not only do you have the issue of auditors who are scared you might put things in logs that shouldn't be there, but there are security / access problems and the very basic problem of "did we log the thing we now need?" Not to say that logs can't be helpful, but they generally aren't enough on their own.

doctorpangloss 3y ago

If I'm going to pay for a vendor APM, who's going to outpace NewRelic? They already do all this stuff - automatic instrumenting, "tail based sampling," etc.

phillipcarter 3y ago
In some cased OpenTelemetry already has just as much or more support as the vendor-specific agents. But really the play here is future-proofing. A lot of observability vendors are in the business of offering steep discounts in year one, locking you in with even more proprietary tech, and increasing your bill. When your instrumentation is done with OpenTelemetry, it takes 15 minutes to start sending data to a competitor. As for sampling, it's still not great with New Relic. Their proprietary system lets you sample based on span attributes and combine that with a random sampling. But they don't dynamically upsample or downsample based on arbitrary tags in the data, density of traffic, and a target sampling rate or throughput. They don't let you define rules and combine them with a dynamic system or something else. All of this is necessary once you have a large volume of trace data and don't want to break the bank to observe it. They don't track the sampling rate and re-weight counts on the backend. At lower or medium-volume traffic what they have suffices (as does the OSS solutions), though. This isn't a knock on New Relic -- it's a similar story pretty much anymore. Cost management at any level just isn't there yet for tracing. The building blocks are there and I expect it to be fixed, but it does prevent widespread adoption.
- doctorpangloss 3y ago
  > All of this is necessary once you have a large volume of trace data and don't want to break the bank to observe it In my opinion, there are two approaches: - I try to do this monitoring as cheaply as possible, which means 100% free software and on premises hardware / minimally used cloud resources. No vendors. - I pay a vendor. And nothing else. From my experience as a SaaS vendor, evaluating paid monitoring options (including NewRelic and Honeycomb) and having used them in the past: Anything more than $0 that doesn't increase the bottom line by a greater asymptotic value is "expensive." When I worked in an organization that is trying to find (1) a marginally cheaper way to do monitoring between two vendors, and therefore (2) predict the future, like how much their usage will be or how much they will grow... the most expensive part of the whole process is all the people working at the organization. That number never seems to go down in response to the choice you make between vendors, it only seems to go down in response to interest rates or catastrophic business externalities. So in my opinion, it's sort of moot which vendor you choose, if you have already crossed the Rubicon to paying vendors. So in an intellectually honest way, no matter how you slice it, NewRelic is "just as expensive" as everything else. > A lot of observability vendors are in the business of offering steep discounts in year one, locking you in with even more proprietary tech, and increasing your bill. The hard, intellectually honest, no-bullshit truth of the matter is that the SaaS vendor's job with pricing is turning whatever the customer likes to hear (upfront invoices and discounts, royalties, per use pricing, whatever) into whatever the investors / owners like to hear (90% margins, hockey stick revenue growth, hockey stick user growth, you can take a long vacation, etc.) via deal term nonsense. For me, for example, I decide I want to make a certain amount of money, and then I turn it into one set of nonsense for customers (a premium fee or up front payment, set up to be economically equivalent) and another set of nonsense for investors (a 90% margin, a hockey stick growth). You might be thinking: this is totally fringe and unorthodox. Listen, go read the Sony emails hack, and see how Michael Lynton deals with this obvious pricing reality. They are a huge, expertly run organization, and all their deals with people were constantly about making up some deal term bullshit the counterparty likes, and then Sony's expert staff of financial modelers gaming out the deal terms to determine the real economic value important to Sony. You don't have to operate in this vacuum as an IC anymore, you can peer directly into how executives actually work at giant corporations, if you dare, and see the realities for yourself. Call me jaded, but having done this long enough, there is no story simpler than "the price is whatever the market will bear, and we'll tell you whatever you want to hear that maximizes it." So the most interesting price point is zero, which is not at all an unorthodox opinion, and indeed, many of the very most disruptive entrepreneurs got to where they were going by valuing their time at zero and rolling out metrics themselves.

prpl 3y ago

Another thing - Traces (and often metrics for that matter) are often treated as something that is investigated one at a time. There's a lot of tools out there to help visualize a trace. They are not, usually, treated as something that should be analyzed at scale, and the tooling there is not great. I think PromScale looked promising for some of those problems but it's also a deprecated product. I think the state of the art is/will be ingesting traces into Iceberg for some form of analysis with whatever execution engine you want (Trino/Spark/etc...). A lot of places are doing this. OpenTelemetry is clearly the future of monitoring, probably with a lot of lifting from Prometheus and other things.

zug_zug 3y ago
So what I want is something like Splunk search for traces. I want to be able to write 12-shareable lines of custom-query and immediately get a pretty graph of multidimensional data "What % of users who just registered this month got a 500 more than half the time on this specific path?" and I want it as a pretty graph over time
- phillipcarter 3y ago
  That's precisely what tools like Honeycomb and Lightstep offer. But most vendors don't have a traces-first approach to querying data.
  - zug_zug 3y ago
    Really? I haven't used honeycomb specifically, but you're saying it supports an arbitrarily complex query depth, with any number of joins essentially?
8note 3y ago
I kinda hacked x-ray onto things, but it does give you an overview aggregate for each node I'm not sure if I've really found it useful though. Im sure if I didn't know the service, itd be great, but I already know where to look for whatever problem

fbdab103 3y ago

>When I monitor the teams usage of traces, it is always a small fraction of the development team that ever logs in to use them... Like fire extinguishers, they are nice to have when you need it.