4 ms·

This is essentially Amazon Coral’s service log format except service logs include cumulative metrics between log events. This surfaces in cloudwatch logs as met

This is essentially Amazon Coral’s service log format except service logs include cumulative metrics between log events. This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain People point to Splunk and ELK but they fail to realize that inverted index based solutions algorithmically can’t scale to arbitrary sizes. I would rather point people to Grafana Loki and CloudWatch Logs Insights and the compromises they entail as not just the right model for “wide events” or structured logging based events and metrics. Their architectures allow you to scale at low costs to PB or even exabyte scale monitoring.

flaminHotSpeedo 3y ago
> This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain I don't have any experience with scuba besides this article, but I think you've missed the point. Wide events, based on my understanding, are a combination of traditional logs and something akin to service logs. This provides two crucial improvements. The first is flexible, arbitrary associations as a first-class feature. As I interpret it, wide events give you the ability to associate a free-form traditional log message with additional dimensions, which is similar to what service logs offer but more flexible. E.g. if you log "caught unhandled FooException, returning ServerException" but only emit a metric for ServerException=1, service logs can't help you. The other major benefit that you seem to have overlooked is a good UI to explore those events. I think most people would agree that the cloud watch UI is somewhere between bad to mediocre, but the monitor portal UI is nothing short of an unmitigated disaster. And neither give you the ability described in this article, to point and click graph events that match certain criteria. As I read it, it's the equivalent functionality to simple insights queries, except it doesn't require any typing, searching for the right dimension names, or writing stats queries to get graphs.
jcgrillo 3y ago
One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. This post (from 1996!) puts us all to shame: https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000... Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings). It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough. Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.
- heyoni 3y ago
  That second part of only storing reference ids sounds like db normalizing but for logs. Seems reasonable though!
- fnordpiglet 3y ago
  In my analysis of some exabyte scale log analytic systems the top two compute spend buckets was json serde and utf8 conversions.
- ablob 3y ago
  Storing by delta can bite you quite hard in the event of data corruption. Instead of 1 data point being affected it would cascade down. Selecting specific ranges where the concrete bottom/top as in "give me everything between 1-2 pm from last Saturday" might also become problematic. I'm sure there's a tradeoff to be had here; Weaving data-dependencies throughout your file certainly leaves a redundancy hole not everyone is willing to have.
  - jcgrillo 3y ago
    I think we could limit the blast radius by working in reasonably sized chunks--like O(10-100MB)--and possibly replicating (which becomes much more attractive when the data set gets a lot smaller). But you're right, it's a good point that redundancy can be a feature.
riku_iki 3y ago
> inverted index based solutions algorithmically can’t scale to arbitrary sizes curious why do you think so? Inverted index can be sharded and built/updated/queried in parallel, so scale linearly.
- jcgrillo 3y ago
  I've seen situations where the cost of indexing all the logs (which were otherwise just written to a hierarchical structure in HDFS and queried with mapreduce jobs) in ES would have been highly significant--think like an uncomfortable fraction of total infrastructure spend. So, sure, you can make it scale linearly by adding enough nodes to keep up with write volume but that doesn't mean it's affordable. And then consider what that's actually accomplishing for those dollars. You're optimizing for quick search queries on data which you'll mostly never query. Worth it? EDIT: as a user, being able to just run mapreduce jobs over logs is a heck of a lot better experience IMO than trying to torture Kibana into giving me the answers I want.
  - riku_iki 3y ago
    it could be shortcoming of ES specifically, since JVM is not ideal platform for heavy data crunching.
  - fnordpiglet 3y ago
    Loki basically works like a nice UI on mapreducing logs. The rest is exactly my point about ELK.
- fnordpiglet 3y ago
  A few issues come up. First inverted indices can be sharded but the index insert patterns aren’t uniformly distributed but instead have a zipf distribution, which means your sharding scales proportional to the frequency of the most common token in the log. There are patches but in the end it sort of boils down to this. Another issue is indexing up front is crazy expensive vs doing absolutely nothing but packing and time indexing, maybe some bloom indices. This is really important because the vast majority of log and event and telemetry in general is never accessed. Like 99.99% of it or more. The technique of something like Loki is to batch data into micro batches and index them within the batches into a columnar store (like parquet of orc) and time index the micro batches. The query path is highly parallel and fairly expensive, but given the cost savings up front it’s a lot cheaper than up front indexing. You can turn the fan out knob on queries to any size and similar to MPP scale out databases such as Snowflake there’s not really much of an upper limit. Effectively everything from ingestion to query scales out linearly without uneven heat problems like you see in a sharded index.
deschutes 3y ago
As far as design and ergonomics go, I'd compare servicelogs to a pile of trash that may yet grow massive enough to accrete into a planetoid. A text based format whose sole virtue is descending from a system that was composed mainly of bugs that had coalesced into perl scripts. It's not the basis of something you could even give away, let alone have people willingly pay you for their agony. Cloudwatch being rather alike in this regard.
rmbyrro 3y ago
Which compromises in CloudWatch Log Insights makes it not the right model for "wide events"? I have the impression it does a good job providing visibility tools (search, filter, aggregation...) over structured logs. Ergonomics is bad, though, with the custom query language and low processing speed, depending on the amount of data you're processing during an investigation.
- fnordpiglet 3y ago
  I think you misunderstood. It does.

flaminHotSpeedo 3y ago

> This surfaces in cloudwatch logs as metrics extraction and Logs Insights as structured log queries. The meta scuba is like a janky imitation of that tool chain I don't have any experience with scuba besides this article, but I think you've missed the point. Wide events, based on my understanding, are a combination of traditional logs and something akin to service logs. This provides two crucial improvements. The first is flexible, arbitrary associations as a first-class feature. As I interpret it, wide events give you the ability to associate a free-form traditional log message with additional dimensions, which is similar to what service logs offer but more flexible. E.g. if you log "caught unhandled FooException, returning ServerException" but only emit a metric for ServerException=1, service logs can't help you. The other major benefit that you seem to have overlooked is a good UI to explore those events. I think most people would agree that the cloud watch UI is somewhere between bad to mediocre, but the monitor portal UI is nothing short of an unmitigated disaster. And neither give you the ability described in this article, to point and click graph events that match certain criteria. As I read it, it's the equivalent functionality to simple insights queries, except it doesn't require any typing, searching for the right dimension names, or writing stats queries to get graphs.

jcgrillo 3y ago

One thing that really gets under my skin when I think about observability data is the abject waste we incur by shipping all this crap around as UTF-8 bytes. This post (from 1996!) puts us all to shame: https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000... Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings). It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough. Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.

heyoni 3y ago
That second part of only storing reference ids sounds like db normalizing but for logs. Seems reasonable though!
fnordpiglet 3y ago
In my analysis of some exabyte scale log analytic systems the top two compute spend buckets was json serde and utf8 conversions.
ablob 3y ago
Storing by delta can bite you quite hard in the event of data corruption. Instead of 1 data point being affected it would cascade down. Selecting specific ranges where the concrete bottom/top as in "give me everything between 1-2 pm from last Saturday" might also become problematic. I'm sure there's a tradeoff to be had here; Weaving data-dependencies throughout your file certainly leaves a redundancy hole not everyone is willing to have.
- jcgrillo 3y ago
  I think we could limit the blast radius by working in reasonably sized chunks--like O(10-100MB)--and possibly replicating (which becomes much more attractive when the data set gets a lot smaller). But you're right, it's a good point that redundancy can be a feature.

riku_iki 3y ago

> inverted index based solutions algorithmically can’t scale to arbitrary sizes curious why do you think so? Inverted index can be sharded and built/updated/queried in parallel, so scale linearly.

jcgrillo 3y ago
I've seen situations where the cost of indexing all the logs (which were otherwise just written to a hierarchical structure in HDFS and queried with mapreduce jobs) in ES would have been highly significant--think like an uncomfortable fraction of total infrastructure spend. So, sure, you can make it scale linearly by adding enough nodes to keep up with write volume but that doesn't mean it's affordable. And then consider what that's actually accomplishing for those dollars. You're optimizing for quick search queries on data which you'll mostly never query. Worth it? EDIT: as a user, being able to just run mapreduce jobs over logs is a heck of a lot better experience IMO than trying to torture Kibana into giving me the answers I want.
- riku_iki 3y ago
  it could be shortcoming of ES specifically, since JVM is not ideal platform for heavy data crunching.
- fnordpiglet 3y ago
  Loki basically works like a nice UI on mapreducing logs. The rest is exactly my point about ELK.
fnordpiglet 3y ago
A few issues come up. First inverted indices can be sharded but the index insert patterns aren’t uniformly distributed but instead have a zipf distribution, which means your sharding scales proportional to the frequency of the most common token in the log. There are patches but in the end it sort of boils down to this. Another issue is indexing up front is crazy expensive vs doing absolutely nothing but packing and time indexing, maybe some bloom indices. This is really important because the vast majority of log and event and telemetry in general is never accessed. Like 99.99% of it or more. The technique of something like Loki is to batch data into micro batches and index them within the batches into a columnar store (like parquet of orc) and time index the micro batches. The query path is highly parallel and fairly expensive, but given the cost savings up front it’s a lot cheaper than up front indexing. You can turn the fan out knob on queries to any size and similar to MPP scale out databases such as Snowflake there’s not really much of an upper limit. Effectively everything from ingestion to query scales out linearly without uneven heat problems like you see in a sharded index.

deschutes 3y ago

As far as design and ergonomics go, I'd compare servicelogs to a pile of trash that may yet grow massive enough to accrete into a planetoid. A text based format whose sole virtue is descending from a system that was composed mainly of bugs that had coalesced into perl scripts. It's not the basis of something you could even give away, let alone have people willingly pay you for their agony. Cloudwatch being rather alike in this regard.

rmbyrro 3y ago

Which compromises in CloudWatch Log Insights makes it not the right model for "wide events"? I have the impression it does a good job providing visibility tools (search, filter, aggregation...) over structured logs. Ergonomics is bad, though, with the custom query language and low processing speed, depending on the amount of data you're processing during an investigation.

fnordpiglet 3y ago
I think you misunderstood. It does.