6 ms·

Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data

140 points by benshumaker 2y ago

ericyd 2y ago
Many commenters seem unenthusiastic about this approach, but it seems like it solves some useful problems for my typical use cases. Regarding dropped timestamps: if log order is preserved within an aggregation, that would be sufficient for me. I'm less concerned with millisecond precision and more interested in sequence. I'm making a big assumption that aggregation would be limited to relatively short time horizons on the order of seconds, not minutes. Grouping related logs sounds like a big win to me in general. Certainly I can perform the same kind of filtering manually in a log viewer, but having logs grouped by common properties by default sounds like it would make logs easier to scan. I've never been involved in billing for logging services so I can't comment on the efficiency gains of log aggregation vs zstd or gzip as proposed by some other commenters.
corytheboyd 2y ago
You know what would actually kill for saving on log data? Being forced to use an efficient format. Instead of serializing string literals to text and sending all those bytes on each log, require that log message templates be registered in a schema, and then a byte or two can replace the text part of the message. Log message template parameters would have to be fully represented in the message, it would be way too much work to register the arbitrary values that get thrown in there. Next logical step is to serialize to something more efficient than JSON— should be dead simple, it’s a template followed by N values to sub into the template, could be a proprietary format, or just use something like protobuf. It’s better than compression, because the data that was being compressed (full text of log message template) is just not even present. You could still see gains from compressing the entire message, in case it has text template values that would benefit from it. I get it, we lost human readability, which may be too big of a compromise for some, but we accomplish the main goal of “make logs smaller” without losing data (individually timestamped events). Besides, this could be made up for with a really nice log viewer client. I’m sure this all exists already to some degree and I just look dumb, but look dumb I will.
- onionisafruit 2y ago
  Don’t worry about human readability. When you have an issue with log size, you are already logging more than a human can read.
  - corytheboyd 2y ago
    Agreed. At that point you need specialized tools anyway.
  - nepthar 2y ago
    I think this is a really good point. A logging system could theoretically toggle "text" mode on and off, giving human readable logs in development and small scale deployments. In fact, I'm going to build a toy one in python!
    - Izkata 2y ago
      > In fact, I'm going to build a toy one in python! I suggest building it as a normal python logging handler instead of totally custom, that way you don't need a "text" toggle and it can be used without changing any existing standard python logging code. Only requires one tweak to the idea: Rather than a template table at the start of the file, have two types of log entries and add each template the first time it's used. Drawback is having to parse the whole file to find all the templates, but you could also do something like putting the templates into a separate file to avoid that...
  - theamk 2y ago
    not really. I am writing code on my machine, running one query at a time. I can easily view the logs and spend a lot of time looking at them. I am running test suites, running thousands of queries. It's harder, but I will still view the logs around failures. Then I am taking the very code, and pushing it to prod. Should my logs be suddenly completely different in this case? (The right answer of course is to have "log-to-text formatter", either running in-process or as a separate post-processing step. But it better produce nice-looking, human-readable logs, or every format will be "Message: %s" )
dantodor 2y ago
FWIW, I've had something very similar in my head like 4 years ago :) https://medium.com/@dantodor/collapsible-logging-567aa2abf829 https://medium.com/@dantodor/collapsible-logging-567aa2abf82...
aaaronic 2y ago
The multi-line case can usually be fixed with simple configuration changes to a structured log format. The other cases are more interesting, and pre-aggregation of all logs related to a correlation ID can be really helpful when debugging a specific incident, but it does seem like this proposal is the same basic trade-off around size and performance as with virtually any form of compression.
XorNot 2y ago
I have a persistent mental nag that the way we do logging always seems so inefficient: that theoretically no application should ever be outputting actual text messages, because logs are basically fixed strings + formatting data. So in theory we could uniquely identify all logs as a much more compact binary representation + a lookup table we ship with the executable.
- drivebyhooting 2y ago
  Compression will basically achieve that kind of encoding without tightly coupling your logging with an external enum or schema.
  - vouwfietsman 2y ago
    Compression is only ever as good as the information it is given, if given text, it can do an OK job using general entropy encoding. However, most data is not text, even log data is not text. Log data is timestamps, numbers, enums, indices, guids, text, etc. Each of those categories, combined with how they are layed out, can be compressed in different optimal ways, resulting in different optimal compression ratios based on your specific log format. As a simple example, compressing 3D meshes as meshes is an order of magnitude better than compressing 3D meshes as text. Sure, zipping your logs gives you a LOT, and you should do it, but if the result disappoints it is not the end of the road, at all.
    - jumping_frog 2y ago
      Isn't compression the best general purpose solution with theoretical guarantees? I mean a simple huffman coding could easily extract the key-values (where the repeated keys will be candidates for compression) relationship and compress it. But if you want to extract more juice, then that implies knowing the datatypes of strings themselves. It would be like types for logs and not messing those up.
- RHSeeger 2y ago
  It depends on your use case. If you're looking to have a set of data that you can search through, sure. If you're looking to tail the log while working on things, having it output in plain text is very handy.
ThrowawayTestr 2y ago
Logs seem like they'd be easily compressible, no?
- efitz 2y ago
  All the LZ variants work well on logs, but I had the best luck so far with zstd. YMMV.
- kikimora 2y ago
  They are, but log analysis vendors charge for the amount of uncompressed logs which quickly amounts to a large bill.
iampims 2y ago
Or sampling :)
- craigching 2y ago
  Sampling is lossy though
  - iampims 2y ago
    lossy and simpler. IME, I've found sampling simpler to reason about, and with the sampling rate part of the message, deriving metrics from logs works pretty well. The example in the article is a little contrived. Healthchecks often originate from multiple hosts and/or logs contain the remote address+port, leading to each log message being effectively unique. So sure, one could parse the remote address into remote_address=192.168.12.23 remote_port=64780 and then decide to drop the port in the aggregation, but is it worth the squeeze?
    - kiitos 2y ago
      If a service emits a log event, then that log event should be visible in your logging system. Basic stuff. Sampling fails this table-stakes requirement.
      - eru 2y ago
        Typically, you store your most recent logs in full, and you can move to sampling for older logs (if you don't want to delete them outright).
        kiitos 2y ago
        It's reasonable to drop logs beyond some window of time -- a year, say -- but I'm not sure why you'd ever sample log events. Metric samples, maybe! Log data, no point. But, in general, I think we agree -- all good!
        2y ago
1oooqooq 2y ago
always assumed this was a given on all the log aggregators SaaS of the last decade.
spenczar5 2y ago
But… this does drop data? Only the start and end timestamp are preserved; the middle ones have no time. How can this be called lossless? Genuinely lossless compression algorithms like gzip work pretty well.
- corytheboyd 2y ago
  Exactly my thoughts, the order of these events by timestamp is itself necessary for debugging. If I want something like per-transaction rollup of events into one log message, I build it and use it explicitly.
- efitz 2y ago
  Was going to point out the same thing - the original article's solution is losing timestamps and possibly ordering. They also are losing some compressibility by converting to a structured format (JSON). And if they actually include a lot of UUIDs (their diagram is vague on what transaction IDs look like), then good luck - those don't compress very well. I worked at a magnificent 7 company that compressed a lot of logs; we found that zstd actually did the best all-around job back in 2021 after a lot of testing.
  - greggyb 2y ago
    We have a process monitor that basically polls ps output and writes it to JSON. We see ~30:1 compression using zstd on a ZFS dataset that stores these logs. I laugh every time I see it.
  - eru 2y ago
    Agreed. If you used something like sequential IDs (even in some UUID format) it can compress pretty well.
    - willvarfar 2y ago
      As a member of the UUIDv7 cheering squad let me say 'rah rah'! :D
  - pdimitar 2y ago
eastern 2y ago
The obvious answer is a relational structure. In the given example, host, status, path and target should be separate relations. They'll all be tiny ones, a few rows each. Of course, performance etc are a separate story but as far as the shape of the data goes, that's what the solution is
- LAC-Tech 2y ago
  Then you're locking your data into a relational model. A log is universal, and can be projected to relational, document etc. Which is why most relational databases are built on top of them.
KaiserPro 2y ago
Shipping and diving logs is a bad idea for anything other than last line debug defence. If you're going to agregate your logs, you're much better off converting them to metrics _on device_. it makes comparison much easier, and storage and pensioning trivial.
- _kb 2y ago
  It doesn't need to happen on device, just upstream of storage (and as close to the source as possible to minimise transport overheads). Most of the OTel collectors are good at this, but IMO Grafana Alloy is particularly neat. This works for when you cannot change the log source too (e.g. third party component or even legacy hardware that may be syslog only).
- marklar423 2y ago
  I think a good compromise is having metrics that are always on, with the ability to enable more verbose debugging logs as needed.
- speedgoose 2y ago
  I agree. I much prefer Prometheus and Sentry over logs.
  - VBprogrammer 2y ago
    Metrics are useless in my experience for figuring out a problem. For the most part they only tell you that you have a problem. Being able to slice and dice logs that you have faith in is critical in my experience.
    - intelVISA 2y ago
      problem_counter: 1
      - _kb 2y ago
        days_since_problem compresses better if you have long streams of '0'.
vdm 2y ago
https://github.com/y-scope/clp https://github.com/y-scope/clp
koliber 2y ago
I'm wondering how this would compare to compressing the logs with Broli and a custom dictionary. I would imagine the size reduction would be much better than the 40% Kevin showed with his approach. As for volume, I don't know if volume is a problem. With logs I either need to look at one log entry to get it's details, a small handful to see a pattern, or aggregate statistics. I can do all of those things wether the volume is 1x, 100x, or 100000x. In other words, size matters to me, but I don't care about volume. On the other hand, for cases when we use tools that charge for uncompressed size or message count, then Kevin's approach could be a useful cost-saving measure.
- ericyd 2y ago
  I interpretted the article's argument as saying that volume can be a major business cost, not that it creates operational challenges.
ajuc 2y ago
My problem with this is handling malformed logs (which is exactly the logs you are the most interested in). If you process logs in a smart way you have to have assumptions about them. When the assumptions are broken the data might get missing (or you can lose the correlation between logs that are close together). That's why I prefer raw unprocessed logs. For example let's imagine you have a bug with stale transaction id in the first example. You probably won't notice when it's grouped by transactionId. If it's raw logs you might notice the order of the logs is wrong. Or maybe the grouping key is missing and your throw NPE and crash the app or skip that line of logs. I've seen both happen when I got too fancy with logging (we had async logger that logged by inserting into a db).

140 points by benshumaker 2y ago

Many commenters seem unenthusiastic about this approach, but it seems like it solves some useful problems for my typical use cases. Regarding dropped timestamps: if log order is preserved within an aggregation, that would be sufficient for me. I'm less concerned with millisecond precision and more interested in sequence. I'm making a big assumption that aggregation would be limited to relatively short time horizons on the order of seconds, not minutes. Grouping related logs sounds like a big win to me in general. Certainly I can perform the same kind of filtering manually in a log viewer, but having logs grouped by common properties by default sounds like it would make logs easier to scan. I've never been involved in billing for logging services so I can't comment on the efficiency gains of log aggregation vs zstd or gzip as proposed by some other commenters.

corytheboyd 2y ago

You know what would actually kill for saving on log data? Being forced to use an efficient format. Instead of serializing string literals to text and sending all those bytes on each log, require that log message templates be registered in a schema, and then a byte or two can replace the text part of the message. Log message template parameters would have to be fully represented in the message, it would be way too much work to register the arbitrary values that get thrown in there. Next logical step is to serialize to something more efficient than JSON— should be dead simple, it’s a template followed by N values to sub into the template, could be a proprietary format, or just use something like protobuf. It’s better than compression, because the data that was being compressed (full text of log message template) is just not even present. You could still see gains from compressing the entire message, in case it has text template values that would benefit from it. I get it, we lost human readability, which may be too big of a compromise for some, but we accomplish the main goal of “make logs smaller” without losing data (individually timestamped events). Besides, this could be made up for with a really nice log viewer client. I’m sure this all exists already to some degree and I just look dumb, but look dumb I will.

onionisafruit 2y ago
Don’t worry about human readability. When you have an issue with log size, you are already logging more than a human can read.
- corytheboyd 2y ago
  Agreed. At that point you need specialized tools anyway.
- nepthar 2y ago
  I think this is a really good point. A logging system could theoretically toggle "text" mode on and off, giving human readable logs in development and small scale deployments. In fact, I'm going to build a toy one in python!
  - Izkata 2y ago
    > In fact, I'm going to build a toy one in python! I suggest building it as a normal python logging handler instead of totally custom, that way you don't need a "text" toggle and it can be used without changing any existing standard python logging code. Only requires one tweak to the idea: Rather than a template table at the start of the file, have two types of log entries and add each template the first time it's used. Drawback is having to parse the whole file to find all the templates, but you could also do something like putting the templates into a separate file to avoid that...
- theamk 2y ago
  not really. I am writing code on my machine, running one query at a time. I can easily view the logs and spend a lot of time looking at them. I am running test suites, running thousands of queries. It's harder, but I will still view the logs around failures. Then I am taking the very code, and pushing it to prod. Should my logs be suddenly completely different in this case? (The right answer of course is to have "log-to-text formatter", either running in-process or as a separate post-processing step. But it better produce nice-looking, human-readable logs, or every format will be "Message: %s" )

dantodor 2y ago

FWIW, I've had something very similar in my head like 4 years ago :) https://medium.com/@dantodor/collapsible-logging-567aa2abf829 https://medium.com/@dantodor/collapsible-logging-567aa2abf82...

aaaronic 2y ago

The multi-line case can usually be fixed with simple configuration changes to a structured log format. The other cases are more interesting, and pre-aggregation of all logs related to a correlation ID can be really helpful when debugging a specific incident, but it does seem like this proposal is the same basic trade-off around size and performance as with virtually any form of compression.

XorNot 2y ago

I have a persistent mental nag that the way we do logging always seems so inefficient: that theoretically no application should ever be outputting actual text messages, because logs are basically fixed strings + formatting data. So in theory we could uniquely identify all logs as a much more compact binary representation + a lookup table we ship with the executable.

drivebyhooting 2y ago
Compression will basically achieve that kind of encoding without tightly coupling your logging with an external enum or schema.
- vouwfietsman 2y ago
  Compression is only ever as good as the information it is given, if given text, it can do an OK job using general entropy encoding. However, most data is not text, even log data is not text. Log data is timestamps, numbers, enums, indices, guids, text, etc. Each of those categories, combined with how they are layed out, can be compressed in different optimal ways, resulting in different optimal compression ratios based on your specific log format. As a simple example, compressing 3D meshes as meshes is an order of magnitude better than compressing 3D meshes as text. Sure, zipping your logs gives you a LOT, and you should do it, but if the result disappoints it is not the end of the road, at all.
  - jumping_frog 2y ago
    Isn't compression the best general purpose solution with theoretical guarantees? I mean a simple huffman coding could easily extract the key-values (where the repeated keys will be candidates for compression) relationship and compress it. But if you want to extract more juice, then that implies knowing the datatypes of strings themselves. It would be like types for logs and not messing those up.
RHSeeger 2y ago
It depends on your use case. If you're looking to have a set of data that you can search through, sure. If you're looking to tail the log while working on things, having it output in plain text is very handy.

ThrowawayTestr 2y ago

Logs seem like they'd be easily compressible, no?

efitz 2y ago
All the LZ variants work well on logs, but I had the best luck so far with zstd. YMMV.
kikimora 2y ago
They are, but log analysis vendors charge for the amount of uncompressed logs which quickly amounts to a large bill.

iampims 2y ago

Or sampling :)

craigching 2y ago
Sampling is lossy though
- iampims 2y ago
  lossy and simpler. IME, I've found sampling simpler to reason about, and with the sampling rate part of the message, deriving metrics from logs works pretty well. The example in the article is a little contrived. Healthchecks often originate from multiple hosts and/or logs contain the remote address+port, leading to each log message being effectively unique. So sure, one could parse the remote address into remote_address=192.168.12.23 remote_port=64780 and then decide to drop the port in the aggregation, but is it worth the squeeze?
  - kiitos 2y ago
    If a service emits a log event, then that log event should be visible in your logging system. Basic stuff. Sampling fails this table-stakes requirement.
    - eru 2y ago
      Typically, you store your most recent logs in full, and you can move to sampling for older logs (if you don't want to delete them outright).
      - kiitos 2y ago
        It's reasonable to drop logs beyond some window of time -- a year, say -- but I'm not sure why you'd ever sample log events. Metric samples, maybe! Log data, no point. But, in general, I think we agree -- all good!
        2y ago

1oooqooq 2y ago

always assumed this was a given on all the log aggregators SaaS of the last decade.

spenczar5 2y ago

But… this does drop data? Only the start and end timestamp are preserved; the middle ones have no time. How can this be called lossless? Genuinely lossless compression algorithms like gzip work pretty well.

corytheboyd 2y ago
Exactly my thoughts, the order of these events by timestamp is itself necessary for debugging. If I want something like per-transaction rollup of events into one log message, I build it and use it explicitly.
efitz 2y ago
Was going to point out the same thing - the original article's solution is losing timestamps and possibly ordering. They also are losing some compressibility by converting to a structured format (JSON). And if they actually include a lot of UUIDs (their diagram is vague on what transaction IDs look like), then good luck - those don't compress very well. I worked at a magnificent 7 company that compressed a lot of logs; we found that zstd actually did the best all-around job back in 2021 after a lot of testing.
- greggyb 2y ago
  We have a process monitor that basically polls ps output and writes it to JSON. We see ~30:1 compression using zstd on a ZFS dataset that stores these logs. I laugh every time I see it.
- eru 2y ago
  Agreed. If you used something like sequential IDs (even in some UUID format) it can compress pretty well.
  - willvarfar 2y ago
    As a member of the UUIDv7 cheering squad let me say 'rah rah'! :D
- pdimitar 2y ago

eastern 2y ago

The obvious answer is a relational structure. In the given example, host, status, path and target should be separate relations. They'll all be tiny ones, a few rows each. Of course, performance etc are a separate story but as far as the shape of the data goes, that's what the solution is

LAC-Tech 2y ago
Then you're locking your data into a relational model. A log is universal, and can be projected to relational, document etc. Which is why most relational databases are built on top of them.

KaiserPro 2y ago

Shipping and diving logs is a bad idea for anything other than last line debug defence. If you're going to agregate your logs, you're much better off converting them to metrics _on device_. it makes comparison much easier, and storage and pensioning trivial.

_kb 2y ago
It doesn't need to happen on device, just upstream of storage (and as close to the source as possible to minimise transport overheads). Most of the OTel collectors are good at this, but IMO Grafana Alloy is particularly neat. This works for when you cannot change the log source too (e.g. third party component or even legacy hardware that may be syslog only).
marklar423 2y ago
I think a good compromise is having metrics that are always on, with the ability to enable more verbose debugging logs as needed.
speedgoose 2y ago
I agree. I much prefer Prometheus and Sentry over logs.
- VBprogrammer 2y ago
  Metrics are useless in my experience for figuring out a problem. For the most part they only tell you that you have a problem. Being able to slice and dice logs that you have faith in is critical in my experience.
  - intelVISA 2y ago
    problem_counter: 1
    - _kb 2y ago
      days_since_problem compresses better if you have long streams of '0'.

vdm 2y ago

https://github.com/y-scope/clp https://github.com/y-scope/clp

koliber 2y ago

I'm wondering how this would compare to compressing the logs with Broli and a custom dictionary. I would imagine the size reduction would be much better than the 40% Kevin showed with his approach. As for volume, I don't know if volume is a problem. With logs I either need to look at one log entry to get it's details, a small handful to see a pattern, or aggregate statistics. I can do all of those things wether the volume is 1x, 100x, or 100000x. In other words, size matters to me, but I don't care about volume. On the other hand, for cases when we use tools that charge for uncompressed size or message count, then Kevin's approach could be a useful cost-saving measure.

ericyd 2y ago
I interpretted the article's argument as saying that volume can be a major business cost, not that it creates operational challenges.

ajuc 2y ago

My problem with this is handling malformed logs (which is exactly the logs you are the most interested in). If you process logs in a smart way you have to have assumptions about them. When the assumptions are broken the data might get missing (or you can lose the correlation between logs that are close together). That's why I prefer raw unprocessed logs. For example let's imagine you have a bug with stale transaction id in the first example. You probably won't notice when it's grouped by transactionId. If it's raw logs you might notice the order of the logs is wrong. Or maybe the grouping key is missing and your throw NPE and crash the app or skip that line of logs. I've seen both happen when I got too fancy with logging (we had async logger that logged by inserting into a db).