14 ms·

All you need is Wide Events, not "Metrics, Logs and Traces"

267 points by talboren 3y ago

hackburg 3y ago
[dead]
swader999 3y ago
This seems like event sourcing with a nice tool to inspect, filter and visualize the event stream. The sampling rate idea is a decent tactic I hadn't heard of.
veeralpatel979 3y ago
Great article, here is a Python notebook I created earlier to show you how you can capture such wide events: https://colab.research.google.com/drive/1Y65qXXogoDgOnXFBDyFsW2EPsJRUf8_J?usp=sharing https://colab.research.google.com/drive/1Y65qXXogoDgOnXFBDyF...
timthelion 3y ago
At the company I work for we send json to kafka and subsiquently to Elastic search with great effect. That's basically 'wide events'. The magical thing about hooking up a bunch of pipelines with kafka is that all of a sudden your observability/metrics system becomes an amazing API for extending systems with aditional automations. Want to do something when a router connects to a network? Just subscribe to this kafka topic here. It doesn't matter that the topic was origionally intended just to log some events. We even created an open source library for writing and running these,pipelines in jupyter. Here's a super simple example https://github.com/bitswan-space/BitSwan/blob/master/examples/Jupyter/Kafka2Kafka/main.ipynb https://github.com/bitswan-space/BitSwan/blob/master/example... People tend to think kafka is hard, but as you can see from the example, it can be extremely easy.
- jeffbee 3y ago
  I'm glad that works for you but to me it sounds really expensive. At small scale you can do this any way you want but if you build an observability system with linear cost and a high coefficient it will become an issue if you run into some success.
  - timthelion 3y ago
    The only expensive part is the hardwarevfor the elastic servers. Kafka is cheap to run. We have an on prem elastic db pulling in tens of thousands of events per second. On prem servers aren't that expensive. It's really just 6 servers with 20tb each and another 40tb for backups. And it's not like you have to store everything forever... Compare that data flow to everyonevwatching youtube all the time. It's really nothing...
    - ricardobeat 3y ago
      I can name a single company in my area that runs their own servers, and they've been in the middle of a migration to the cloud for the past five years.
      - markhahn 3y ago
        it's really down to company culture: how many MBAs and PMs it normalizes.
    - kiitos 3y ago
      > We have an on prem elastic db pulling in tens of thousands of events per second. Many kinds of systems can work great at this scale, which is non-trivial but ultimately not particularly large. > It's really just 6 servers with 20tb each and another 40tb for backups. Hopefully you can ingest and store and query O(10k) RPS with an order of magnitude less resources than this?
- hibikir 3y ago
  This works well for a while. But eventually you get big, and have little to no idea of what is in your downstream. Then every single format change in any event you write must be treated like open heart surgery, because tracing your data dependencies is unreliable. Sometimes it seems that it's fixable by 'just having a list of people listening', and then you look and all that some of them do is mildly transform your data and pass it along. It doesn't take long before people realize that. 'just logging some events' is making future promises to other teams you don't know about, and people start being terrified of emitting anything. This is a story I've seen in at least 4 places in my career. Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database, and you'd see yearlong projects to do some mild changes to a data model, which was originally designed in 5 minutes. As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.
  - meandmycode 3y ago
    I think this is the crux of it, if something works for awhile then actually that's fine, as an industry we over index and scare new developers towards complexity. The counter is true too, what works at scale doesn't at non scale - not because of tech, but because holistically your asking for a lot, a lot of knowledge, a lot of complex tech to be deployed by a small team.
  - lmm 3y ago
    > This works well for a while. But eventually you get big, and have little to no idea of what is in your downstream. Then every single format change in any event you write must be treated like open heart surgery, because tracing your data dependencies is unreliable. Yeah, I'd always use protobuf or similar rather than JSON for that reason, and if you need a truly breaking change I'd emit a new version of the events to a new topic rather than trying to migrate the existing one in place. It's not actually so costly to keep writing events to an old topic (and if you really want you can move that part into a separate adapter process that reads your new topic and writes to your old one). Or you can do the whole avro/schema-registry stuff if you prefer. > Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database It should be significantly less scary: it's impossible to mutate data in-place, foreign key issues are something you go back and fix and reprocess rather than something that takes down your OLTP system, schema changes are better-understood and less big-bang, event streams that are generated by transforming another event stream are completely indistinguishable from "original" event streams as opposed to views being sort-of-like-tables but having all sorts of caveats and gotchas. > As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy. There are hard parts but also parts that are easier than a traditional database. There's no query planner, no MVCC, no locks, no deadlocks, no isolation levels, indices are not magic, ...
- dgellow 3y ago
  Zookeeper in production can be really a pain to maintain…
  - rockwotj 3y ago
    Check out Redpanda if you don’t like Zookeeper. Raft based, no JVM, lower cost and simpler to run Kafka compatible system
  - KptMarchewa 3y ago
    You don't need zookeeper for Kafka for a while now.
    - dgellow 3y ago
      good to hear, I haven’t dealt with Kafka for years. Zookeeper was a pain to deal with, glad to see there are alternatives
jeffbee 3y ago
The isomorphism of traces and logs is clear. You can flatten a trace to a log and you can perfectly reconstruct the trace graph from such a log. I don't see the unifying theme that brings metrics into this framework, though. Metrics feels fundamentally different, as a way to inspect the internal state of your program, not necessarily driven by exogenous events. But I definitely agree with the theme of the article that leaving a big company can feel like you got your memory erased in a time machine mishap. Inside a FANG you might become normalized to logging hundreds of thousands of informational statements, per second, per core. You might have got used to every endpoint exposing thirty million metric time series. As soon as you walk out the door some guy will chew you out about "cardinality" if you have 100 metrics.
- HeyImAlex 3y ago
  I think all metrics can be reconstructed as “wide events” since they’re just a bunch of arbitrary data? Counts, gauges, and histograms at least seem pretty straight forward to me. It seems like the main motivation for metrics is that sending + storing + querying wide events for everything is cost prohibitive and/or performance intensive. If you can afford it and it works well, wide events is definitely more flexible. A metric is kinda just a pre-aggregation on the event stream.
- growse 3y ago
  If you think of a metric as an event representing the act of measuring (along with the result of that measurement), then it becomes the same as any other event.
  - jeffbee 3y ago
    True. I guess the thing that I normally want from metrics is I want to have a huge number of them that exist in a way that I can look at them when I want. But I don't want to have to pay for collecting and aggregating them all the time. So in the scenario where they are just events then I need some other control system that can trigger the collection of events that aren't normally emitted
    - 3y ago
Solvency 3y ago
[flagged]
- deleted 3y ago
  [deleted]
Osmose 3y ago
This isn't an unknown idea outside of Meta, it's just really expensive, especially if you're using a vendor and not building your own tooling. Prohibitively so, even with sampling.
- staticautomatic 3y ago
  Maybe this is a dumb question but why wouldn’t it be cost effective to pre-aggregate counts occasionally and sample on the fly?
- ricardobeat 3y ago
  Exactly. When they say > Unlike with prometheus, however, with Wide Events approach we don’t need to worry about cardinality This is hinting at the hidden reason why not everyone does it. You have to 'worry' about cardinality because Prometheus is pre-aggregating data so you can visualize it fast, and optimizing storage. If you want the same speed on a massive PB-scale data lake, with an infinite amount of unstructured data, and in the cloud instead of your own datacenters, it's gonna cost you a lot, and for most companies it is not a sensible expense. It does work at smaller scale though, we once had an in-house system like this that worked well. Eventually user events were moved to MixPanel, and everything else to Datadog, metrics/logs/traces + a migration to OpenTel. It took months and added 2-digit monthly bills, and in the end debugging or resolving incidents wasn't much improved over having instant access to events and business metrics. Whoever figures out a system that can do "wide events" in a cost-effective way from startup to unicorn scale will absolutely make a killing.
  - hosh 3y ago
    Not everything emits wide events. Maybe you can get the entire application layer like that, but there is also value in logs and metrics emitted from the rest of the infra stack. To be fair, you could probably store and represent everything as wide events and build visualization tools out of that that can combine everything together, even if they are sourced from something else.
    - lelandbatey 3y ago
Veserv 3y ago
[flagged]
- deleted 3y ago
  [deleted]
rekwah 3y ago
> just put it there, it might be useful later > Also note that we have never mentioned anything about cardinality. Because it doesn’t matter - any field can be of any cardinality. Scuba works with raw events and doesn’t pre-aggregate anything, and so cardinality is not an issue. This is how we end up with very large, very expensive data swamps.
- _visgean 3y ago
  that depends on the sampling rate no? I would much rather have a rich log record sampled at 1% than more records that dont contain enough info to debug..
  - growse 3y ago
    The people feeling the pain of (and paying for) the expensive data swamp are often not the same people who are yolo'ing the sample rate to 100% in their apps, because why wouldn't you want to store every event? Put another way, you're in charge of a large telemetry event sink. How do you incentivise the correct sampling behaviour by your users?
    - Spivak 3y ago
      Don't let the user pick the sampling rate. In Honeycomb land this is called the EMA Dynamic Sampler. https://docs.honeycomb.io/manage-data-volume/refinery/sampling-methods/#deterministic-sampler https://docs.honeycomb.io/manage-data-volume/refinery/sampli...
    - kiitos 3y ago
      You should never need to sample telemetry data.
  - gtirloni 3y ago
    Metrics sample rate yes but logging sample? When an end-to-end transaction for a very important task breaks, do I get *some* breadcrumbs to debug it?
ojkelly 3y ago
Observability as a shared concept has followed Agile and DevOps. Something with a real meaning that is enables a step-change is development practices. Adoption is organic initially because the pain it solves is very real. But as awareness of the idea grows it threatens established institutions and vendors, who must co-opt the concept and redefine it such that they are included. If they can’t be explicitly included (logs, metrics, traces)[0], then they at least make sure the definition is becomes so vague and confused that they are not explicitly excluded[1]. Wide events and a good means to query them covers everything, but not if you as a vendor cannot store and query wide events. [0] as the article notes, one of these is not like the other. [1] Is Scrum Agile? What do you mean a standup can’t go for an hour? See also DevOps as a role.
- deleted 3y ago
  [deleted]
hacker_newz 3y ago
[flagged]
- deleted 3y ago
  [deleted]
guhcampos 3y ago
Incredible what you can do with infinite money! For everyone else, more specific data structures, sampling and careful consideration of what to record are essential.
- munchbunny 3y ago
  The key is that you pay for the bandwidth, sampling, cardinality, and schema considerations somewhere in the system. Depending on the problem, you may be able to get away with dealing with those issues later vs. earlier, but at some point you start dealing with required fields, aggregation, etc. I own a system in one of the tech giants where sampling is counterproductive to our problem domain, and where in the pipeline we deal with each of those issues is our bread and butter problem that can sometimes swing costs by $millions.
- strken 3y ago
  Hang on, you don't need infinite money if you've got a sampling rate, do you? Drop the sampling rate (mentioned in the article: they do use sampling) from 0.01 to 0.001 and you've reduced the data ingress by a factor of ten.
- deleted 3y ago
  [deleted]
zemo 3y ago
begging people to recognize that a person who sells a solution is going to view these problems through the lens of being rewarded for applying their solution to your problem, even if it's not appropriate. > Yet, per my own experience it’s still extremely hard to explain what does Charity meant by “logs are thrash”, let alone the fact that logs and traces are essentially the same things. Why is everyone so confused? Charity is not confused, Charity is incentivized. What she means by "logs are trash" is "I do not sell a logging product". (and, to be clear, I'm only naming Charity individually here because that's who the author named in their article.) > When I was working at Meta, I wasn’t aware that I was privileged to be using the best observability system ever. The observability system that is appropriate for Meta is not necessarily appropriate for your project. Those tools are cool but also require a pretty serious investment to build and operate correctly. It's very easy to wade into a cardinality explosion problem when tagging and indexing everything you can imagine, it's very easy to wade into problems regarding mixed retention policies when some events are important and others are less-important, it's very easy to wade into a latency-sensitivity issue if you're building a log/event collection infra that you don't allow to ever lose data, etc. As it turns out, observability is a large topic. The idea that there's one "best" way to do observability is a little ridiculous. Like when I worked at Etsy some of the data was literally money, when I worked at Jackbox Games we made fart joke games (Quiplash, Drawful, Fibbage, You Don't Know Jack, etc) and the infrastructure was nothing but pure cost. The observability needs of those two orgs were phenomenally different, because the products were different, the revenue models were different, the needs of the users were different, etc. Also this notion that "all you need is wide events" is the answer seems ... really shallow. A data point is an unordered set of key-value pairs? That's how ... a LOT of logging, metrics, and tracing infra expresses things at the level of an individual record/event. The difference is in the relationships between the keys and values, the relationships between the individual records, etc. and "stop sampling" is just a bizarre marketing angle. If you have 1 million records or 10 million records and you get the same squiggly line out of analyzing it, congrats you have inflated the size of the data that nobody ever looks at. There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline are incentivized to tell you that sampling is bad: if you are sampling, you are sending and storing and querying fewer data points, so they are charging you less money. They are getting paid to tell you that sampling is bad. Sampling is not good or bad, sampling is sampling. The reality is that in a lot of these systems, the vast majority of the information will never, ever be looked at or used. Whether or not that matters is entirely context dependent.
4ndrewl 3y ago
Wide events are fine until someone puts personally identifiable information (PII) in them. Then you're in a bit of a mess as you've presumably taken PII out of an environment with one set of access controls, and into a separate, different environment, with access controls that are for a different purpose than required by the data.
- benwaffle 3y ago
  Logging and tracing have this problem too.
  - goosejuice 3y ago
    Wide events described in this article seem to equal structured logging but a more loose dumping ground. So yeah to an extent it has this problem, just more so. How does tracing? Are folks adding PII to spans? I suppose you could but I'm not sure why.
karmakaze 3y ago
What's the difference between a Wide Event and a structured log?
- deathanatos 3y ago
  Nothing; what the author is calling a wide event is a structured log. > Structured Log which is pretty much the Wide Event The OP's point is that you can just capture anything that one might call a metric/span/trace in a structured log. So, it's a bit more specific, in some senses: a "metric" is a structured log bearing some value. E.g., { "@timestamp": "…", "query_kind": "list_people_in_org", "query_time_s": .051 } Is a structured log. It's also a single datapoint along a timeseries. The metric might be something like "query time", with a dimension "query_kind". ELK stacks let you visualize such structured logs into graphs with only moderate pain. A "span" is a structured log with start/end timestamps. A trace is just a tree of spans, built out of many structured logs. In a prior life, we'd have, { "span_id": "root_span_id/subspan_a/subsub_span_b", … } You can pull out whole traces by doing a "span_id startswith 'root_span_id'" query in whatever your query language is. You can pull out sub-traces similarly. It works even across microservice boundaries, if you coordinate the span IDs appropriately. I think where I depart from the OP: it takes too much space. The OP advocates sampling; I'd rather a specialized data structure for metrics that can store a data point efficiently. I.e., at least in sizeof(timestamp || f32) bytes, ideally better, and then I can just not worry about sampling. (In a sense; if you're watching a continuous value, you're always sampling. But for discrete events that you're recording, I can just record all the events. But I'll only get a number. Sometimes, that's okay. I feel like this is a bit spiritually different than the sampling the OP is discussing.)
  - karmakaze 3y ago
    Thanks. I don't know why new unpopular names are used for well-known things. Googling for "Wide Events" give millions of results for "City-wide events" etc. Probably just clickbaiting--what's a Wide Event?
    - phillipcarter 3y ago
jrockway 3y ago
I like logs. Unlike most people selling and using observability platforms, most of the software I write is run by other people. That means it can't send me traces and I can't scrape it for metrics, but I still have to figure out and fix their problems. To me, logs are the answer. Logs are easy to pass around, and you can put whatever you want in there. I have libraries for metrics and traces, and just parse them out of the logs when that sort of presentation would be useful. (Yes, we do sampling as well.) I keep hearing that this doesn't scale. When I worked at Google, we used this sort of system to monitor our Google Fiber devices. They just uploaded their logs every minute (stored in memory, held in memory after a warm reboot thanks to a custom linux kernel with printk_persist), and then my software processed them into metrics for the "fast query" monitoring systems. The most important metrics fed into alerts, but it didn't take very much time to just re-read all the logs if you wanted to add something new. Amazingly, the first version of this system ran on a single machine... 1 Go program handling 10,000qps of log uploads and analysis. I eventually distributed it to survive machine and datacenter failures, but it ultimately isn't that computationally intensive. The point is, it kind of scales OK. Up to 10s of terabytes a day, it's something you don't even have to think about except for the storage cost. At some point it does make sense to move things into better databases than logs; you want to be alerted by your monitoring system that 99%-ile latency is high, then look in Jaeger for long-running traces, then take the trace ID and search your logs for it. If you start with logs, you have that capability. If you start with something else, then you just have "the program is broken, good luck" and you have to guess what the problem is whenever you debug. Ideally, your program would just tell you what's broken. That's what logs are. One place where people get burned with logs is not being careful about what to log. Logs are the primary user interface for operators of your software (i.e. you during your oncall week), and that task deserves the attention that any other user interface task demands. People often start by logging too much, then get tired of "spam", and end up not logging enough. Then a problem occurs and the logs are outright misleading. (My favorite is event failures that are retried, but the retry isn't logged anywhere. You end up seeing "ERROR foobar attempt 1/3 failed" and have no idea of knowing that attempt 2/3 succeeded a millisecond after that log line.) For the gophers around, here's what I do for traces: https://github.com/pachyderm/pachyderm/blob/master/src/internal/log/span.go#L130 https://github.com/pachyderm/pachyderm/blob/master/src/inter... and metrics: https://github.com/pachyderm/pachyderm/blob/master/src/internal/meters/meters.go https://github.com/pachyderm/pachyderm/blob/master/src/inter.... If you have a pipeline for storing and retrieving logs (which is exactly the case for this particular piece of software), now you have metrics and traces. It's great! I just need to write the thing to turn a set of log files into a UI that looks like Jaeger and Prometheus ;) My favorite part is that I don't need to care about the cardinality of metrics; every RPC gets its own set of metrics. So I can write a quick jq program to figure out how much bandwidth the entire system is using, or I can look at how much bandwidth one request is using. (meters logs every X bytes, and log entries have timestamps.) I think since we've added this capability to our system, incidents are most often resolved with "that's fixed in the next patch release" instead of multiple iterations "can you try this custom build and take another debug dump". Very enjoyable.
renewiltord 3y ago
Has anyone built an open-source version of this and have a blog post around it? Curious about implementation to see how you keep storage tight and querying still fast.
- isburmistrov 3y ago
  I think ClickHouse is becoming a default storage for observability nowdays: https://clickhouse.com/use-cases/logging-and-metrics https://clickhouse.com/use-cases/logging-and-metrics And there are quite a few solutions on top of it. A couple of examples that seem to be interesting (however I didn't use them in real life): https://coroot.com/ https://coroot.com/ https://qryn.metrico.in/#/ https://qryn.metrico.in/#/
  - renewiltord 3y ago
    Thank you for sharing!
  - AndyNemmity 3y ago
    We are currently using Elasticsearch, and can only store a tiny amount of the data because of how much we use. 5 days at most. ClickHouse is replacing Elasticsearch in this context, and is providing the same data storage, but with better compression, and not holding data in memory? Is that correct?
    - isburmistrov 3y ago
      I think the storage architecture in ClickHouse and Elastic are very different. And I think compression in ClickHouse can be really damn good. Don't have a good comparison at hands though, but a few random links on the topic: https://engineeringat.axis.com/schema-changes-clickhouse/ https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-elasticsearch-to-clickhouse https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migration-from-elasticsearch-to-clickhouse https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-platform-using-clickhouse-for-petabyte-scale https://blog.zomato.com/building-a-cost-effective-logging-pl...
Scotrix 3y ago
Using the ELK stack for almost a decade to have somewhat wide events + no sampling, on not Meta scale and a few GB/day make it absolutely affordable and super fast. Unfortunately Kibana was a bit better/easier in the old versions than nowadays but it’s still pretty straight forward to get everything out of it.
zug_zug 3y ago
This person is simply misinformed. I worked at meta and used scuba, and it's like 6/10 (which makes it one of meta's best tools). A tool like splunk can do everything scuba can do and a million things it can't. Sumologic can too. The reason that splunk/sumologic are so much better than scuba is that they have open-ended query languages rather than this on-rails "only ever do one group-by". Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo. I could write a whole essay on the topic really, but the gist of it is you need a full-scaled open-ended language for advanced querying because 1% of the time you need to do weird stuff like count-by -> a second count-by. What I will agree with is that traces/metrics do not inherently give you this ability, but absolutely traces could if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)
- isburmistrov 3y ago
  I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely. Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.
  - adamgordonbell 3y ago
    Honeycomb was in fact built to originally be a replacement for scuba, by people who found it so valuable and missed it.
  - zug_zug 3y ago
    It looks like a "wide-event" is just a structured log, you can send any log containing json to sumo/splunk at it'll parse it out as fields for you. So as I'm understanding it you're advocating for structured logs (which is fine, those are great). If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.
mdavidn 3y ago
The SQL-equivalent in the sampling rate example should sum the inversion: SELECT SUM(1 / samplingRate) FROM AdImpressions WHERE IsTest = False
fnordpiglet 3y ago
This is essentially Amazon Coral’s service log format except service logs include cumulative metrics between log events. This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain People point to Splunk and ELK but they fail to realize that inverted index based solutions algorithmically can’t scale to arbitrary sizes. I would rather point people to Grafana Loki and CloudWatch Logs Insights and the compromises they entail as not just the right model for “wide events” or structured logging based events and metrics. Their architectures allow you to scale at low costs to PB or even exabyte scale monitoring.
- flaminHotSpeedo 3y ago
  > This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain I don't have any experience with scuba besides this article, but I think you've missed the point. Wide events, based on my understanding, are a combination of traditional logs and something akin to service logs. This provides two crucial improvements. The first is flexible, arbitrary associations as a first-class feature. As I interpret it, wide events give you the ability to associate a free-form traditional log message with additional dimensions, which is similar to what service logs offer but more flexible. E.g. if you log "caught unhandled FooException, returning ServerException" but only emit a metric for ServerException=1, service logs can't help you. The other major benefit that you seem to have overlooked is a good UI to explore those events. I think most people would agree that the cloud watch UI is somewhere between bad to mediocre, but the monitor portal UI is nothing short of an unmitigated disaster. And neither give you the ability described in this article, to point and click graph events that match certain criteria. As I read it, it's the equivalent functionality to simple insights queries, except it doesn't require any typing, searching for the right dimension names, or writing stats queries to get graphs.
- jcgrillo 3y ago
  One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. This post (from 1996!) puts us all to shame: https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000... Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings). It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough. Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.
gorlami 3y ago
So, what is the closest thing in the open source world to what the author describes? (Setting aside the question of is it right for you, which, of course, depends.)
- mdavidn 3y ago
  Any OLAP database that accepts unstructured data can be used in this manner. The ELK stack is a popular choice, albeit with a focus on search rather than OLAP. If SaaS is an option, a simple staring point in AWS might be Data Firehose into S3 with Athena. Snowflake can load and query the data too. All of these tools have multiple frontend options with a proportional relationship between cost and user-friendliness. I honestly just do this in PostgreSQL until my project outgrows it. Create a table with a JSONB column and as few indexes as possible to improve write throughput. Cover a timestamp column with a BRIN index to filter by date range.
  - johnthescott 3y ago
    brin is your friend for logs, for sure.
- viraptor 3y ago
  Still early days, but Grafana's tracing is getting there. https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/traces/ https://grafana.com/docs/grafana/latest/panels-visualization...
  - MuffinFlavored 3y ago
    What format/library/protocol does your app speak to get the traces into Grafana Traces? Tempo?
    - viraptor 3y ago
      https://grafana.com/oss/tempo/ https://grafana.com/oss/tempo/ > Tempo can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.
wiseguyeh 3y ago
While I don't have an opinion on wide events (AKA spans) replacing logs, there are benefits to metrics that warrant their existence: 1. They're incredibly cheap to store. In Prometheus, is may cost you as little as 1 byte per sample (ignoring series overheads). Because they're cheap, you can keep them for much longer and use them for long-term analysis of traffic, resource use, performance, etc. Most tracing vendors seem to cap storage at 1-3 months while metric vendors can offer multi-year storage. 2. They're far more accurate that metrics derived from wide events in higher-throughput scenarios. While wide events are incredibly flexible, their higher storage cost means there's an upper limit on the sample rate. The sampled nature of wide events means that deriving accurate counts is far more difficult- metrics really shine in this role (unless you're operating over datasets with very high cardinality). The problem only gets worse when you combine tail sampling into the mix and add bias towards errors/ slow requests in your data.
- fl0ki 3y ago
  On top of that, metrics can have exemplars, which give you more (and dynamic) dimensions for buckets without increasing the cardinality of the metric vectors themselves. It's pretty much a wide event, with the sampling rate on this extra information just being the scrape interval you were already using anyway. Not every library or tool supports exemplars, but they're a big part of the Prometheus & Grafana value proposition that many users entirely overlook.
- phillipcarter 3y ago
  For point (2), you can derive accurate counts from sampled data if the sampling rate is captured as metadata on every sampled event. Some tools do support this (I work for Honeycomb, and our sampling proxy + backend work like this, can't speak for others). The issue is there are still limits to that, though. I can still get a count of events, or a AVG(duration_ms). But if I have a custom tag I can't get accurate counts of that. And if I want to get distinct counts of values, I'm out of luck. Estimating that is an active machine learning research problem.
  - taion 3y ago
    You also lose accuracy because of sampling noise.
nemo44x 3y ago
The best part of this post is where they quote a failed SaSS trying to explain why successful SaSS is wrong. Anything for an edge even if it’s not useful.
- jldugger 3y ago
  Is honeycomb failed?
  - phillipcarter 3y ago
    It’d be news to me and my paycheck.
blinded 3y ago
otel is working on their events spec https://github.com/open-telemetry/community/issues/1688 https://github.com/open-telemetry/community/issues/1688
- phillipcarter 3y ago
  Heh, this is an unfortunate consequence of naming. OTel Events are just OTel logs with a name. An OTel log is a log body with a trace ID, span ID, severity, etc. But that's also an event as per the author's definition. In OTel, a span is just a structured log, which is also an event (as per the author's definition of event). So is a Span Event, which is a log-like entity that you can produce in the context of a span. And OTel metrics produce what are called "metric events", which are also events. IMO it's one of those things that's horribly confusing until one day it's not, and then everything starts looking like how the author described.
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
mikpanko 3y ago
It took the world decades to develop widely accepted standards for working with relational data and SQL. I believe we are at the early stages of doing the same with event data and sequence analytics. It is starting to simultaneously emerge in many different fields: - eng observability (traces at Datadog, Sumologic, etc) - operational research (process mining at Celonis) - product analytics (funnels at Amplitude, Mixpanel) As with every new field, there are a lot of different and overlapping terms being suggested and explored at the same time. We are trying to contribute to the field with a deep fundamental approach at Motif Analytics, including a purpose-built set of core sequence operations, rich flow visualizations, a pattern matching query engine, and foundational AI models on event sequences [1]. Fun fact: creators of Scuba turned it into a startup Interana (acquired by Twitter), who we took a lot of inspiration from for Motif's query engine. [1] https://motifanalytics.com https://motifanalytics.com
- isburmistrov 3y ago
  Thanks for sharing, didn't know about Motif
jhardy54 3y ago
Is this not just structured logging? I’m wondering whether the author has used tracing tools much, or whether they’re truly trying to understand modern observability through OpenTelemetry documentation.
goosejuice 3y ago
This sounds like a privacy nightmare as described if there aren't guardrails. 'Dump everything' Can pretty easily achieve this with structured logging in GCP with their metrics explorer. Pretty cheaply I might add. Sentry can also do a bit of this if you're on something like fly.io (they offer a year free). I don't think either would completely replace tracing in a complex system for me. At least not in the context Ive worked.
treflop 3y ago
We use wide events at work (or really “structured logging” or really “your log system has fields”) and they are great. But they aren’t a replacement for metrics because metrics are so god damn cheap. And while I’ve never used a log system with traces, every logging setup I’ve ever used has had request/correlation IDs to generate a trace because sometimes you just wanna lookup a flow and see it without spending a time digging through wide events/your log system. If you aren’t looking up logs very often, then yeah it seems browsing through structured logs isn’t that bad but then do it often and it’s just annoying…
TeeWEE 3y ago
This is basically a metric with tags. Only difference is that a metric has a main unit it measures. In the end anything can be represented as a structured log. A span is NOT what the OP calls a “system wide event”. A span has a begin and end time. What he/she describes doesn’t have that. In the end giving different kind of instrumentation instruments a name makes sense, mainly for processing them / rendering them / altering on them.
infogulch 3y ago
This thread has a lot of discussion about Wide Events / Structured Logs (same thing) being too big at scale, and you should use metrics instead. Why does it have to be an either/or thing? Couldn't you hook up a metrics extractor to the event stream and convert your structured logs to compact metrics in-process before expensive serde/encoding? With this your choice doesn't have to affect the code, just write slogs all the time; if you want structured logs then then output them, but if you only want metrics then switch to the metrics extractor slog handler. Futher, has nobody tried writing structured logs to parquet files and shipping out 1MB blocks at once? Way less serde/encoding overhead, and column oriented layout compresses like crazy with built-in dictionary and delta encodings.
- pat2man 3y ago
  The open telemetry collector does just that. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/spanmetricsprocessor/README.md https://github.com/open-telemetry/opentelemetry-collector-co...
  - pingiun 3y ago
    seems cool but the top of the page says the thing you suggest is now deprecated
    - nijave 3y ago
      Span Metrics Processor is replaced with the very similar Span Metrics Connector which is still supported
      - paulddraper 3y ago
        The Span Metrics Processor being replaced by the Span Metrics Connector is very, very OpenTelemetry.
- lowbloodsugar 3y ago
  >too big at scale Bigger than meta? o_0
jupp0r 3y ago
This looks like structured logging and piping those logs to Splunk or am I missing something?
tlarkworthy 3y ago
FWIW I think x-ray has everything you need, its just that AWs tooling does not give you much ability to aggregate over x-ray bundles. I wrote a tool to help bulk load x-ray samples into a local browser duckdb and then slice and dicing in realtime interactive visualisations. It also includes the ability to generate a flamegraph over the selected traces. All this great data is already in an AWS, account and we just need better tools to make use of it. https://observablehq.com/@tomlarkworthy/x-ray-slurper https://observablehq.com/@tomlarkworthy/x-ray-slurper
pranabgohain 3y ago
You could also use a unified, OTel native platform like https://www.kloudmate.com https://www.kloudmate.com instead of setting up Grafana, Prometheus, Loki, separately.
npalli 3y ago
Man, this whole "All you need is XYZ" framing is turning out to the as irritating and overused as "XYZ considered harmful" from back in the day.
abhisgup 3y ago
New Relic supports this in the form of custom events. I have used it and it works but is very expensive. An alternative is to use ClickhouseDB directly.
- isburmistrov 3y ago
  Does it support "native sampling" described in the article? This is really important to keep the cost low.

267 points by talboren 3y ago

[dead]

This seems like event sourcing with a nice tool to inspect, filter and visualize the event stream. The sampling rate idea is a decent tactic I hadn't heard of.

veeralpatel979 3y ago

Great article, here is a Python notebook I created earlier to show you how you can capture such wide events: https://colab.research.google.com/drive/1Y65qXXogoDgOnXFBDyFsW2EPsJRUf8_J?usp=sharing https://colab.research.google.com/drive/1Y65qXXogoDgOnXFBDyF...

timthelion 3y ago

At the company I work for we send json to kafka and subsiquently to Elastic search with great effect. That's basically 'wide events'. The magical thing about hooking up a bunch of pipelines with kafka is that all of a sudden your observability/metrics system becomes an amazing API for extending systems with aditional automations. Want to do something when a router connects to a network? Just subscribe to this kafka topic here. It doesn't matter that the topic was origionally intended just to log some events. We even created an open source library for writing and running these,pipelines in jupyter. Here's a super simple example https://github.com/bitswan-space/BitSwan/blob/master/examples/Jupyter/Kafka2Kafka/main.ipynb https://github.com/bitswan-space/BitSwan/blob/master/example... People tend to think kafka is hard, but as you can see from the example, it can be extremely easy.

jeffbee 3y ago
I'm glad that works for you but to me it sounds really expensive. At small scale you can do this any way you want but if you build an observability system with linear cost and a high coefficient it will become an issue if you run into some success.
- timthelion 3y ago
  The only expensive part is the hardwarevfor the elastic servers. Kafka is cheap to run. We have an on prem elastic db pulling in tens of thousands of events per second. On prem servers aren't that expensive. It's really just 6 servers with 20tb each and another 40tb for backups. And it's not like you have to store everything forever... Compare that data flow to everyonevwatching youtube all the time. It's really nothing...
  - ricardobeat 3y ago
    I can name a single company in my area that runs their own servers, and they've been in the middle of a migration to the cloud for the past five years.
    - markhahn 3y ago
      it's really down to company culture: how many MBAs and PMs it normalizes.
  - kiitos 3y ago
    > We have an on prem elastic db pulling in tens of thousands of events per second. Many kinds of systems can work great at this scale, which is non-trivial but ultimately not particularly large. > It's really just 6 servers with 20tb each and another 40tb for backups. Hopefully you can ingest and store and query O(10k) RPS with an order of magnitude less resources than this?
hibikir 3y ago
This works well for a while. But eventually you get big, and have little to no idea of what is in your downstream. Then every single format change in any event you write must be treated like open heart surgery, because tracing your data dependencies is unreliable. Sometimes it seems that it's fixable by 'just having a list of people listening', and then you look and all that some of them do is mildly transform your data and pass it along. It doesn't take long before people realize that. 'just logging some events' is making future promises to other teams you don't know about, and people start being terrified of emitting anything. This is a story I've seen in at least 4 places in my career. Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database, and you'd see yearlong projects to do some mild changes to a data model, which was originally designed in 5 minutes. As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.
- meandmycode 3y ago
  I think this is the crux of it, if something works for awhile then actually that's fine, as an industry we over index and scare new developers towards complexity. The counter is true too, what works at scale doesn't at non scale - not because of tech, but because holistically your asking for a lot, a lot of knowledge, a lot of complex tech to be deployed by a small team.
- lmm 3y ago
  > This works well for a while. But eventually you get big, and have little to no idea of what is in your downstream. Then every single format change in any event you write must be treated like open heart surgery, because tracing your data dependencies is unreliable. Yeah, I'd always use protobuf or similar rather than JSON for that reason, and if you need a truly breaking change I'd emit a new version of the events to a new topic rather than trying to migrate the existing one in place. It's not actually so costly to keep writing events to an old topic (and if you really want you can move that part into a separate adapter process that reads your new topic and writes to your old one). Or you can do the whole avro/schema-registry stuff if you prefer. > Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database It should be significantly less scary: it's impossible to mutate data in-place, foreign key issues are something you go back and fix and reprocess rather than something that takes down your OLTP system, schema changes are better-understood and less big-bang, event streams that are generated by transforming another event stream are completely indistinguishable from "original" event streams as opposed to views being sort-of-like-tables but having all sorts of caveats and gotchas. > As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy. There are hard parts but also parts that are easier than a traditional database. There's no query planner, no MVCC, no locks, no deadlocks, no isolation levels, indices are not magic, ...
dgellow 3y ago
Zookeeper in production can be really a pain to maintain…
- rockwotj 3y ago
  Check out Redpanda if you don’t like Zookeeper. Raft based, no JVM, lower cost and simpler to run Kafka compatible system
- KptMarchewa 3y ago
  You don't need zookeeper for Kafka for a while now.
  - dgellow 3y ago
    good to hear, I haven’t dealt with Kafka for years. Zookeeper was a pain to deal with, glad to see there are alternatives

jeffbee 3y ago

The isomorphism of traces and logs is clear. You can flatten a trace to a log and you can perfectly reconstruct the trace graph from such a log. I don't see the unifying theme that brings metrics into this framework, though. Metrics feels fundamentally different, as a way to inspect the internal state of your program, not necessarily driven by exogenous events. But I definitely agree with the theme of the article that leaving a big company can feel like you got your memory erased in a time machine mishap. Inside a FANG you might become normalized to logging hundreds of thousands of informational statements, per second, per core. You might have got used to every endpoint exposing thirty million metric time series. As soon as you walk out the door some guy will chew you out about "cardinality" if you have 100 metrics.

HeyImAlex 3y ago
I think all metrics can be reconstructed as “wide events” since they’re just a bunch of arbitrary data? Counts, gauges, and histograms at least seem pretty straight forward to me. It seems like the main motivation for metrics is that sending + storing + querying wide events for everything is cost prohibitive and/or performance intensive. If you can afford it and it works well, wide events is definitely more flexible. A metric is kinda just a pre-aggregation on the event stream.
growse 3y ago
If you think of a metric as an event representing the act of measuring (along with the result of that measurement), then it becomes the same as any other event.
- jeffbee 3y ago
  True. I guess the thing that I normally want from metrics is I want to have a huge number of them that exist in a way that I can look at them when I want. But I don't want to have to pay for collecting and aggregating them all the time. So in the scenario where they are just events then I need some other control system that can trigger the collection of events that aren't normally emitted
  - 3y ago

Solvency 3y ago

[flagged]

deleted 3y ago
[deleted]

Osmose 3y ago

This isn't an unknown idea outside of Meta, it's just really expensive, especially if you're using a vendor and not building your own tooling. Prohibitively so, even with sampling.

staticautomatic 3y ago
Maybe this is a dumb question but why wouldn’t it be cost effective to pre-aggregate counts occasionally and sample on the fly?
ricardobeat 3y ago
Exactly. When they say > Unlike with prometheus, however, with Wide Events approach we don’t need to worry about cardinality This is hinting at the hidden reason why not everyone does it. You have to 'worry' about cardinality because Prometheus is pre-aggregating data so you can visualize it fast, and optimizing storage. If you want the same speed on a massive PB-scale data lake, with an infinite amount of unstructured data, and in the cloud instead of your own datacenters, it's gonna cost you a lot, and for most companies it is not a sensible expense. It does work at smaller scale though, we once had an in-house system like this that worked well. Eventually user events were moved to MixPanel, and everything else to Datadog, metrics/logs/traces + a migration to OpenTel. It took months and added 2-digit monthly bills, and in the end debugging or resolving incidents wasn't much improved over having instant access to events and business metrics. Whoever figures out a system that can do "wide events" in a cost-effective way from startup to unicorn scale will absolutely make a killing.
- hosh 3y ago
  Not everything emits wide events. Maybe you can get the entire application layer like that, but there is also value in logs and metrics emitted from the rest of the infra stack. To be fair, you could probably store and represent everything as wide events and build visualization tools out of that that can combine everything together, even if they are sourced from something else.
  - lelandbatey 3y ago

Veserv 3y ago

[flagged]

deleted 3y ago
[deleted]

rekwah 3y ago

> just put it there, it might be useful later > Also note that we have never mentioned anything about cardinality. Because it doesn’t matter - any field can be of any cardinality. Scuba works with raw events and doesn’t pre-aggregate anything, and so cardinality is not an issue. This is how we end up with very large, very expensive data swamps.

_visgean 3y ago
that depends on the sampling rate no? I would much rather have a rich log record sampled at 1% than more records that dont contain enough info to debug..
- growse 3y ago
  The people feeling the pain of (and paying for) the expensive data swamp are often not the same people who are yolo'ing the sample rate to 100% in their apps, because why wouldn't you want to store every event? Put another way, you're in charge of a large telemetry event sink. How do you incentivise the correct sampling behaviour by your users?
  - Spivak 3y ago
    Don't let the user pick the sampling rate. In Honeycomb land this is called the EMA Dynamic Sampler. https://docs.honeycomb.io/manage-data-volume/refinery/sampling-methods/#deterministic-sampler https://docs.honeycomb.io/manage-data-volume/refinery/sampli...
  - kiitos 3y ago
    You should never need to sample telemetry data.
- gtirloni 3y ago
  Metrics sample rate yes but logging sample? When an end-to-end transaction for a very important task breaks, do I get *some* breadcrumbs to debug it?

ojkelly 3y ago

Observability as a shared concept has followed Agile and DevOps. Something with a real meaning that is enables a step-change is development practices. Adoption is organic initially because the pain it solves is very real. But as awareness of the idea grows it threatens established institutions and vendors, who must co-opt the concept and redefine it such that they are included. If they can’t be explicitly included (logs, metrics, traces)[0], then they at least make sure the definition is becomes so vague and confused that they are not explicitly excluded[1]. Wide events and a good means to query them covers everything, but not if you as a vendor cannot store and query wide events. [0] as the article notes, one of these is not like the other. [1] Is Scrum Agile? What do you mean a standup can’t go for an hour? See also DevOps as a role.

deleted 3y ago
[deleted]

hacker_newz 3y ago

[flagged]

deleted 3y ago
[deleted]

guhcampos 3y ago

Incredible what you can do with infinite money! For everyone else, more specific data structures, sampling and careful consideration of what to record are essential.

munchbunny 3y ago
The key is that you pay for the bandwidth, sampling, cardinality, and schema considerations somewhere in the system. Depending on the problem, you may be able to get away with dealing with those issues later vs. earlier, but at some point you start dealing with required fields, aggregation, etc. I own a system in one of the tech giants where sampling is counterproductive to our problem domain, and where in the pipeline we deal with each of those issues is our bread and butter problem that can sometimes swing costs by $millions.
strken 3y ago
Hang on, you don't need infinite money if you've got a sampling rate, do you? Drop the sampling rate (mentioned in the article: they do use sampling) from 0.01 to 0.001 and you've reduced the data ingress by a factor of ten.
deleted 3y ago
[deleted]

zemo 3y ago

begging people to recognize that a person who sells a solution is going to view these problems through the lens of being rewarded for applying their solution to your problem, even if it's not appropriate. > Yet, per my own experience it’s still extremely hard to explain what does Charity meant by “logs are thrash”, let alone the fact that logs and traces are essentially the same things. Why is everyone so confused? Charity is not confused, Charity is incentivized. What she means by "logs are trash" is "I do not sell a logging product". (and, to be clear, I'm only naming Charity individually here because that's who the author named in their article.) > When I was working at Meta, I wasn’t aware that I was privileged to be using the best observability system ever. The observability system that is appropriate for Meta is not necessarily appropriate for your project. Those tools are cool but also require a pretty serious investment to build and operate correctly. It's very easy to wade into a cardinality explosion problem when tagging and indexing everything you can imagine, it's very easy to wade into problems regarding mixed retention policies when some events are important and others are less-important, it's very easy to wade into a latency-sensitivity issue if you're building a log/event collection infra that you don't allow to ever lose data, etc. As it turns out, observability is a large topic. The idea that there's one "best" way to do observability is a little ridiculous. Like when I worked at Etsy some of the data was literally money, when I worked at Jackbox Games we made fart joke games (Quiplash, Drawful, Fibbage, You Don't Know Jack, etc) and the infrastructure was nothing but pure cost. The observability needs of those two orgs were phenomenally different, because the products were different, the revenue models were different, the needs of the users were different, etc. Also this notion that "all you need is wide events" is the answer seems ... really shallow. A data point is an unordered set of key-value pairs? That's how ... a LOT of logging, metrics, and tracing infra expresses things at the level of an individual record/event. The difference is in the relationships between the keys and values, the relationships between the individual records, etc. and "stop sampling" is just a bizarre marketing angle. If you have 1 million records or 10 million records and you get the same squiggly line out of analyzing it, congrats you have inflated the size of the data that nobody ever looks at. There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline are incentivized to tell you that sampling is bad: if you are sampling, you are sending and storing and querying fewer data points, so they are charging you less money. They are getting paid to tell you that sampling is bad. Sampling is not good or bad, sampling is sampling. The reality is that in a lot of these systems, the vast majority of the information will never, ever be looked at or used. Whether or not that matters is entirely context dependent.

4ndrewl 3y ago

Wide events are fine until someone puts personally identifiable information (PII) in them. Then you're in a bit of a mess as you've presumably taken PII out of an environment with one set of access controls, and into a separate, different environment, with access controls that are for a different purpose than required by the data.

benwaffle 3y ago
Logging and tracing have this problem too.
- goosejuice 3y ago
  Wide events described in this article seem to equal structured logging but a more loose dumping ground. So yeah to an extent it has this problem, just more so. How does tracing? Are folks adding PII to spans? I suppose you could but I'm not sure why.

karmakaze 3y ago

What's the difference between a Wide Event and a structured log?

deathanatos 3y ago
Nothing; what the author is calling a wide event is a structured log. > Structured Log which is pretty much the Wide Event The OP's point is that you can just capture anything that one might call a metric/span/trace in a structured log. So, it's a bit more specific, in some senses: a "metric" is a structured log bearing some value. E.g., { "@timestamp": "…", "query_kind": "list_people_in_org", "query_time_s": .051 } Is a structured log. It's also a single datapoint along a timeseries. The metric might be something like "query time", with a dimension "query_kind". ELK stacks let you visualize such structured logs into graphs with only moderate pain. A "span" is a structured log with start/end timestamps. A trace is just a tree of spans, built out of many structured logs. In a prior life, we'd have, { "span_id": "root_span_id/subspan_a/subsub_span_b", … } You can pull out whole traces by doing a "span_id startswith 'root_span_id'" query in whatever your query language is. You can pull out sub-traces similarly. It works even across microservice boundaries, if you coordinate the span IDs appropriately. I think where I depart from the OP: it takes too much space. The OP advocates sampling; I'd rather a specialized data structure for metrics that can store a data point efficiently. I.e., at least in sizeof(timestamp || f32) bytes, ideally better, and then I can just not worry about sampling. (In a sense; if you're watching a continuous value, you're always sampling. But for discrete events that you're recording, I can just record all the events. But I'll only get a number. Sometimes, that's okay. I feel like this is a bit spiritually different than the sampling the OP is discussing.)
- karmakaze 3y ago
  Thanks. I don't know why new unpopular names are used for well-known things. Googling for "Wide Events" give millions of results for "City-wide events" etc. Probably just clickbaiting--what's a Wide Event?
  - phillipcarter 3y ago

jrockway 3y ago

I like logs. Unlike most people selling and using observability platforms, most of the software I write is run by other people. That means it can't send me traces and I can't scrape it for metrics, but I still have to figure out and fix their problems. To me, logs are the answer. Logs are easy to pass around, and you can put whatever you want in there. I have libraries for metrics and traces, and just parse them out of the logs when that sort of presentation would be useful. (Yes, we do sampling as well.) I keep hearing that this doesn't scale. When I worked at Google, we used this sort of system to monitor our Google Fiber devices. They just uploaded their logs every minute (stored in memory, held in memory after a warm reboot thanks to a custom linux kernel with printk_persist), and then my software processed them into metrics for the "fast query" monitoring systems. The most important metrics fed into alerts, but it didn't take very much time to just re-read all the logs if you wanted to add something new. Amazingly, the first version of this system ran on a single machine... 1 Go program handling 10,000qps of log uploads and analysis. I eventually distributed it to survive machine and datacenter failures, but it ultimately isn't that computationally intensive. The point is, it kind of scales OK. Up to 10s of terabytes a day, it's something you don't even have to think about except for the storage cost. At some point it does make sense to move things into better databases than logs; you want to be alerted by your monitoring system that 99%-ile latency is high, then look in Jaeger for long-running traces, then take the trace ID and search your logs for it. If you start with logs, you have that capability. If you start with something else, then you just have "the program is broken, good luck" and you have to guess what the problem is whenever you debug. Ideally, your program would just tell you what's broken. That's what logs are. One place where people get burned with logs is not being careful about what to log. Logs are the primary user interface for operators of your software (i.e. you during your oncall week), and that task deserves the attention that any other user interface task demands. People often start by logging too much, then get tired of "spam", and end up not logging enough. Then a problem occurs and the logs are outright misleading. (My favorite is event failures that are retried, but the retry isn't logged anywhere. You end up seeing "ERROR foobar attempt 1/3 failed" and have no idea of knowing that attempt 2/3 succeeded a millisecond after that log line.) For the gophers around, here's what I do for traces: https://github.com/pachyderm/pachyderm/blob/master/src/internal/log/span.go#L130 https://github.com/pachyderm/pachyderm/blob/master/src/inter... and metrics: https://github.com/pachyderm/pachyderm/blob/master/src/internal/meters/meters.go https://github.com/pachyderm/pachyderm/blob/master/src/inter.... If you have a pipeline for storing and retrieving logs (which is exactly the case for this particular piece of software), now you have metrics and traces. It's great! I just need to write the thing to turn a set of log files into a UI that looks like Jaeger and Prometheus ;) My favorite part is that I don't need to care about the cardinality of metrics; every RPC gets its own set of metrics. So I can write a quick jq program to figure out how much bandwidth the entire system is using, or I can look at how much bandwidth one request is using. (meters logs every X bytes, and log entries have timestamps.) I think since we've added this capability to our system, incidents are most often resolved with "that's fixed in the next patch release" instead of multiple iterations "can you try this custom build and take another debug dump". Very enjoyable.

renewiltord 3y ago

Has anyone built an open-source version of this and have a blog post around it? Curious about implementation to see how you keep storage tight and querying still fast.

isburmistrov 3y ago
I think ClickHouse is becoming a default storage for observability nowdays: https://clickhouse.com/use-cases/logging-and-metrics https://clickhouse.com/use-cases/logging-and-metrics And there are quite a few solutions on top of it. A couple of examples that seem to be interesting (however I didn't use them in real life): https://coroot.com/ https://coroot.com/ https://qryn.metrico.in/#/ https://qryn.metrico.in/#/
- renewiltord 3y ago
  Thank you for sharing!
- AndyNemmity 3y ago
  We are currently using Elasticsearch, and can only store a tiny amount of the data because of how much we use. 5 days at most. ClickHouse is replacing Elasticsearch in this context, and is providing the same data storage, but with better compression, and not holding data in memory? Is that correct?
  - isburmistrov 3y ago
    I think the storage architecture in ClickHouse and Elastic are very different. And I think compression in ClickHouse can be really damn good. Don't have a good comparison at hands though, but a few random links on the topic: https://engineeringat.axis.com/schema-changes-clickhouse/ https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-elasticsearch-to-clickhouse https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migration-from-elasticsearch-to-clickhouse https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-platform-using-clickhouse-for-petabyte-scale https://blog.zomato.com/building-a-cost-effective-logging-pl...

Scotrix 3y ago

Using the ELK stack for almost a decade to have somewhat wide events + no sampling, on not Meta scale and a few GB/day make it absolutely affordable and super fast. Unfortunately Kibana was a bit better/easier in the old versions than nowadays but it’s still pretty straight forward to get everything out of it.

zug_zug 3y ago

This person is simply misinformed. I worked at meta and used scuba, and it's like 6/10 (which makes it one of meta's best tools). A tool like splunk can do everything scuba can do and a million things it can't. Sumologic can too. The reason that splunk/sumologic are so much better than scuba is that they have open-ended query languages rather than this on-rails "only ever do one group-by". Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo. I could write a whole essay on the topic really, but the gist of it is you need a full-scaled open-ended language for advanced querying because 1% of the time you need to do weird stuff like count-by -> a second count-by. What I will agree with is that traces/metrics do not inherently give you this ability, but absolutely traces could if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)

isburmistrov 3y ago
I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely. Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.
- adamgordonbell 3y ago
  Honeycomb was in fact built to originally be a replacement for scuba, by people who found it so valuable and missed it.
- zug_zug 3y ago
  It looks like a "wide-event" is just a structured log, you can send any log containing json to sumo/splunk at it'll parse it out as fields for you. So as I'm understanding it you're advocating for structured logs (which is fine, those are great). If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.

mdavidn 3y ago

The SQL-equivalent in the sampling rate example should sum the inversion: SELECT SUM(1 / samplingRate) FROM AdImpressions WHERE IsTest = False

fnordpiglet 3y ago

This is essentially Amazon Coral’s service log format except service logs include cumulative metrics between log events. This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain People point to Splunk and ELK but they fail to realize that inverted index based solutions algorithmically can’t scale to arbitrary sizes. I would rather point people to Grafana Loki and CloudWatch Logs Insights and the compromises they entail as not just the right model for “wide events” or structured logging based events and metrics. Their architectures allow you to scale at low costs to PB or even exabyte scale monitoring.

flaminHotSpeedo 3y ago
> This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain I don't have any experience with scuba besides this article, but I think you've missed the point. Wide events, based on my understanding, are a combination of traditional logs and something akin to service logs. This provides two crucial improvements. The first is flexible, arbitrary associations as a first-class feature. As I interpret it, wide events give you the ability to associate a free-form traditional log message with additional dimensions, which is similar to what service logs offer but more flexible. E.g. if you log "caught unhandled FooException, returning ServerException" but only emit a metric for ServerException=1, service logs can't help you. The other major benefit that you seem to have overlooked is a good UI to explore those events. I think most people would agree that the cloud watch UI is somewhere between bad to mediocre, but the monitor portal UI is nothing short of an unmitigated disaster. And neither give you the ability described in this article, to point and click graph events that match certain criteria. As I read it, it's the equivalent functionality to simple insights queries, except it doesn't require any typing, searching for the right dimension names, or writing stats queries to get graphs.
jcgrillo 3y ago
One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. This post (from 1996!) puts us all to shame: https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000... Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings). It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough. Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.

gorlami 3y ago

So, what is the closest thing in the open source world to what the author describes? (Setting aside the question of is it right for you, which, of course, depends.)

mdavidn 3y ago
Any OLAP database that accepts unstructured data can be used in this manner. The ELK stack is a popular choice, albeit with a focus on search rather than OLAP. If SaaS is an option, a simple staring point in AWS might be Data Firehose into S3 with Athena. Snowflake can load and query the data too. All of these tools have multiple frontend options with a proportional relationship between cost and user-friendliness. I honestly just do this in PostgreSQL until my project outgrows it. Create a table with a JSONB column and as few indexes as possible to improve write throughput. Cover a timestamp column with a BRIN index to filter by date range.
- johnthescott 3y ago
  brin is your friend for logs, for sure.
viraptor 3y ago
Still early days, but Grafana's tracing is getting there. https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/traces/ https://grafana.com/docs/grafana/latest/panels-visualization...
- MuffinFlavored 3y ago
  What format/library/protocol does your app speak to get the traces into Grafana Traces? Tempo?
  - viraptor 3y ago
    https://grafana.com/oss/tempo/ https://grafana.com/oss/tempo/ > Tempo can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.

wiseguyeh 3y ago

While I don't have an opinion on wide events (AKA spans) replacing logs, there are benefits to metrics that warrant their existence: 1. They're incredibly cheap to store. In Prometheus, is may cost you as little as 1 byte per sample (ignoring series overheads). Because they're cheap, you can keep them for much longer and use them for long-term analysis of traffic, resource use, performance, etc. Most tracing vendors seem to cap storage at 1-3 months while metric vendors can offer multi-year storage. 2. They're far more accurate that metrics derived from wide events in higher-throughput scenarios. While wide events are incredibly flexible, their higher storage cost means there's an upper limit on the sample rate. The sampled nature of wide events means that deriving accurate counts is far more difficult- metrics really shine in this role (unless you're operating over datasets with very high cardinality). The problem only gets worse when you combine tail sampling into the mix and add bias towards errors/ slow requests in your data.

fl0ki 3y ago
On top of that, metrics can have exemplars, which give you more (and dynamic) dimensions for buckets without increasing the cardinality of the metric vectors themselves. It's pretty much a wide event, with the sampling rate on this extra information just being the scrape interval you were already using anyway. Not every library or tool supports exemplars, but they're a big part of the Prometheus & Grafana value proposition that many users entirely overlook.
phillipcarter 3y ago
For point (2), you can derive accurate counts from sampled data if the sampling rate is captured as metadata on every sampled event. Some tools do support this (I work for Honeycomb, and our sampling proxy + backend work like this, can't speak for others). The issue is there are still limits to that, though. I can still get a count of events, or a AVG(duration_ms). But if I have a custom tag I can't get accurate counts of that. And if I want to get distinct counts of values, I'm out of luck. Estimating that is an active machine learning research problem.
- taion 3y ago
  You also lose accuracy because of sampling noise.

nemo44x 3y ago

The best part of this post is where they quote a failed SaSS trying to explain why successful SaSS is wrong. Anything for an edge even if it’s not useful.

jldugger 3y ago
Is honeycomb failed?
- phillipcarter 3y ago
  It’d be news to me and my paycheck.

blinded 3y ago

otel is working on their events spec https://github.com/open-telemetry/community/issues/1688 https://github.com/open-telemetry/community/issues/1688

phillipcarter 3y ago
Heh, this is an unfortunate consequence of naming. OTel Events are just OTel logs with a name. An OTel log is a log body with a trace ID, span ID, severity, etc. But that's also an event as per the author's definition. In OTel, a span is just a structured log, which is also an event (as per the author's definition of event). So is a Span Event, which is a log-like entity that you can produce in the context of a span. And OTel metrics produce what are called "metric events", which are also events. IMO it's one of those things that's horribly confusing until one day it's not, and then everything starts looking like how the author described.

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]

mikpanko 3y ago

It took the world decades to develop widely accepted standards for working with relational data and SQL. I believe we are at the early stages of doing the same with event data and sequence analytics. It is starting to simultaneously emerge in many different fields: - eng observability (traces at Datadog, Sumologic, etc) - operational research (process mining at Celonis) - product analytics (funnels at Amplitude, Mixpanel) As with every new field, there are a lot of different and overlapping terms being suggested and explored at the same time. We are trying to contribute to the field with a deep fundamental approach at Motif Analytics, including a purpose-built set of core sequence operations, rich flow visualizations, a pattern matching query engine, and foundational AI models on event sequences [1]. Fun fact: creators of Scuba turned it into a startup Interana (acquired by Twitter), who we took a lot of inspiration from for Motif's query engine. [1] https://motifanalytics.com https://motifanalytics.com

isburmistrov 3y ago
Thanks for sharing, didn't know about Motif

jhardy54 3y ago

Is this not just structured logging? I’m wondering whether the author has used tracing tools much, or whether they’re truly trying to understand modern observability through OpenTelemetry documentation.

goosejuice 3y ago

This sounds like a privacy nightmare as described if there aren't guardrails. 'Dump everything' Can pretty easily achieve this with structured logging in GCP with their metrics explorer. Pretty cheaply I might add. Sentry can also do a bit of this if you're on something like fly.io (they offer a year free). I don't think either would completely replace tracing in a complex system for me. At least not in the context Ive worked.

treflop 3y ago

We use wide events at work (or really “structured logging” or really “your log system has fields”) and they are great. But they aren’t a replacement for metrics because metrics are so god damn cheap. And while I’ve never used a log system with traces, every logging setup I’ve ever used has had request/correlation IDs to generate a trace because sometimes you just wanna lookup a flow and see it without spending a time digging through wide events/your log system. If you aren’t looking up logs very often, then yeah it seems browsing through structured logs isn’t that bad but then do it often and it’s just annoying…

TeeWEE 3y ago

This is basically a metric with tags. Only difference is that a metric has a main unit it measures. In the end anything can be represented as a structured log. A span is NOT what the OP calls a “system wide event”. A span has a begin and end time. What he/she describes doesn’t have that. In the end giving different kind of instrumentation instruments a name makes sense, mainly for processing them / rendering them / altering on them.

infogulch 3y ago

This thread has a lot of discussion about Wide Events / Structured Logs (same thing) being too big at scale, and you should use metrics instead. Why does it have to be an either/or thing? Couldn't you hook up a metrics extractor to the event stream and convert your structured logs to compact metrics in-process before expensive serde/encoding? With this your choice doesn't have to affect the code, just write slogs all the time; if you want structured logs then then output them, but if you only want metrics then switch to the metrics extractor slog handler. Futher, has nobody tried writing structured logs to parquet files and shipping out 1MB blocks at once? Way less serde/encoding overhead, and column oriented layout compresses like crazy with built-in dictionary and delta encodings.

pat2man 3y ago
The open telemetry collector does just that. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/spanmetricsprocessor/README.md https://github.com/open-telemetry/opentelemetry-collector-co...
- pingiun 3y ago
  seems cool but the top of the page says the thing you suggest is now deprecated
  - nijave 3y ago
    Span Metrics Processor is replaced with the very similar Span Metrics Connector which is still supported
    - paulddraper 3y ago
      The Span Metrics Processor being replaced by the Span Metrics Connector is very, very OpenTelemetry.
lowbloodsugar 3y ago
>too big at scale Bigger than meta? o_0

jupp0r 3y ago

This looks like structured logging and piping those logs to Splunk or am I missing something?

tlarkworthy 3y ago

FWIW I think x-ray has everything you need, its just that AWs tooling does not give you much ability to aggregate over x-ray bundles. I wrote a tool to help bulk load x-ray samples into a local browser duckdb and then slice and dicing in realtime interactive visualisations. It also includes the ability to generate a flamegraph over the selected traces. All this great data is already in an AWS, account and we just need better tools to make use of it. https://observablehq.com/@tomlarkworthy/x-ray-slurper https://observablehq.com/@tomlarkworthy/x-ray-slurper

pranabgohain 3y ago

You could also use a unified, OTel native platform like https://www.kloudmate.com https://www.kloudmate.com instead of setting up Grafana, Prometheus, Loki, separately.

npalli 3y ago

Man, this whole "All you need is XYZ" framing is turning out to the as irritating and overused as "XYZ considered harmful" from back in the day.

abhisgup 3y ago

New Relic supports this in the form of custom events. I have used it and it works but is very expensive. An alternative is to use ClickhouseDB directly.

isburmistrov 3y ago
Does it support "native sampling" described in the article? This is really important to keep the cost low.