3 ms·

This person is simply misinformed. I worked at meta and used scuba, and it's like 6/10 (which makes it one of meta's best tools). A tool like splunk can do eve

This person is simply misinformed. I worked at meta and used scuba, and it's like 6/10 (which makes it one of meta's best tools). A tool like splunk can do everything scuba can do and a million things it can't. Sumologic can too. The reason that splunk/sumologic are so much better than scuba is that they have open-ended query languages rather than this on-rails "only ever do one group-by". Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo. I could write a whole essay on the topic really, but the gist of it is you need a full-scaled open-ended language for advanced querying because 1% of the time you need to do weird stuff like count-by -> a second count-by. What I will agree with is that traces/metrics do not inherently give you this ability, but absolutely traces could if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)

isburmistrov 3y ago
I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely. Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.
- adamgordonbell 3y ago
  Honeycomb was in fact built to originally be a replacement for scuba, by people who found it so valuable and missed it.
- zug_zug 3y ago
  It looks like a "wide-event" is just a structured log, you can send any log containing json to sumo/splunk at it'll parse it out as fields for you. So as I'm understanding it you're advocating for structured logs (which is fine, those are great). If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.
  - jimbobimbo 3y ago
    It exactly is a structured log or log in open telemetry. To make it easier for myself, I think of spans also as structured logs with a schema that everyone had agreed on, which make it possible to trace requests across multiple services/clients. It's probably more than that, but I don't need academic precision to see how this is more useful during livesite investigations than simply querying logs with unaligned schemas.
    - isburmistrov 3y ago
      Yes, structured log exactly. Why I prefer "wide event" as a term because it has this "wide" component that serves for 2 purposes: - it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it But just a personal preference in the end.
neilkk 3y ago
mipsytipsy also worked with scuba, and she specifically designed honeycomb, an observability tool based on wide events, around the motivating idea that a very imperfect tool like scuba could do lots of things better than much more polished and better engineered tools which didn't use the wide events approach.
deleted 3y ago
[deleted]
downWidOutaFite 3y ago
I've seen some really nice UI query systems that have plenty of expressive power for 99.9% of queries but also include an escape hatch to allow you to use a query language. Mixpanel has one that I really like and I've seen non-techy business types go to town on it no problem. The other option I've seen is a guided query language with helpful UI affordances as you type. I've seen it on Datadog as well as Jira's advanced search.
ot 3y ago
> Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo. You can trivially do that in Scuba too using a derived column, which is supported in the UI. If you need more complex stuff you can write your query in SQL and still use all the supported UI visualizations. And for even more complex stuff, for example if you need joins, the data is usually mirrored to Presto, so you can write arbitrarily complex queries and still visualize the results in the Scuba UI. I'm not sure if your assessment of Scuba is based on full knowledge of what you can do with it.
jensensbutton 3y ago
> if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1) Datadog has a new feature for querying the whole trace call graph that they seem to be testing out. Sounds like precisely the thing you're describing.

isburmistrov 3y ago

I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely. Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.

adamgordonbell 3y ago
Honeycomb was in fact built to originally be a replacement for scuba, by people who found it so valuable and missed it.
zug_zug 3y ago
It looks like a "wide-event" is just a structured log, you can send any log containing json to sumo/splunk at it'll parse it out as fields for you. So as I'm understanding it you're advocating for structured logs (which is fine, those are great). If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.
- jimbobimbo 3y ago
  It exactly is a structured log or log in open telemetry. To make it easier for myself, I think of spans also as structured logs with a schema that everyone had agreed on, which make it possible to trace requests across multiple services/clients. It's probably more than that, but I don't need academic precision to see how this is more useful during livesite investigations than simply querying logs with unaligned schemas.
  - isburmistrov 3y ago
    Yes, structured log exactly. Why I prefer "wide event" as a term because it has this "wide" component that serves for 2 purposes: - it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it But just a personal preference in the end.

neilkk 3y ago

mipsytipsy also worked with scuba, and she specifically designed honeycomb, an observability tool based on wide events, around the motivating idea that a very imperfect tool like scuba could do lots of things better than much more polished and better engineered tools which didn't use the wide events approach.

deleted 3y ago

[deleted]

downWidOutaFite 3y ago

I've seen some really nice UI query systems that have plenty of expressive power for 99.9% of queries but also include an escape hatch to allow you to use a query language. Mixpanel has one that I really like and I've seen non-techy business types go to town on it no problem. The other option I've seen is a guided query language with helpful UI affordances as you type. I've seen it on Datadog as well as Jira's advanced search.

ot 3y ago

> Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo. You can trivially do that in Scuba too using a derived column, which is supported in the UI. If you need more complex stuff you can write your query in SQL and still use all the supported UI visualizations. And for even more complex stuff, for example if you need joins, the data is usually mirrored to Presto, so you can write arbitrarily complex queries and still visualize the results in the Scuba UI. I'm not sure if your assessment of Scuba is based on full knowledge of what you can do with it.

jensensbutton 3y ago

> if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1) Datadog has a new feature for querying the whole trace call graph that they seem to be testing out. Sounds like precisely the thing you're describing.