12 ms·

Monitoring your logs is mostly a tarpit

119 points by signa11 3y ago

laserbeam 3y ago
I generally feel that way regarding the log systems I've worked with, specifically since they were some of the first I helped implement and/or design, and thus I'd not consider them high quality. However, what would be better than monitoring logs? Any sort of monitoring I know of looks at logs produced by one system or another... I'm not sure if there is an alternative.
- Xylakant 3y ago
  Tracking specific metrics and monitoring those is an option - for example you could track service restarts and alert on unusually high numbers. The tricky part - just as with logging - is to figure out what metric make sense and give you signal and what metrics are mostly noise. Sometimes you can even extract metrics from logs by looking for specific log lines if you know what to expect.
  - fluential 3y ago
    You often create metrics based on logs indicating certain events, not much progress here in that approach
  - kubanczyk 3y ago
    > Sometimes you can even extract metrics from logs by looking for specific log lines This looks backwards from the app dev's perspective. - Metrics are strictly structured, documented data. They are a first-class output. - "Strictly structured logs" is contradiction in terms. These are just events to be sent to a pub/sub, also a first-class output. - Logs are unstructured, unknown, bleeding-edge, grey-area data, which can potentially become either metrics, or events, or bug tickets. If I output data as metrics, I don't output it again as logs - that'd be just noise. But of course other apps/systems can have a wholly different approach and force people to invest a lot into sorting out that dumpster fire. The leverage is on the producer side.
- gjvc 3y ago
  if we are talking about the "log files" aspect then one certainly needs to get away from that and have all log events in a single searchable interface.
- thom 3y ago
  Start treating reporting and monitoring as first class parts of your domain instead of some text. Do the same modelling/event storming exercises you’d do for everything else with these operational requirements.
acumenical 3y ago
This seems backwards to me in 2023. Logs seem like valuable data to train your AI on in order to automate as much of the basic inference away as possible.
- darkclouds 3y ago
  What if the logs were missing entries though? Would you train your AI on missing data? I found a bug in rsyslog a few years ago, where in certain conditions, log entries were missing. If you know the pattern for what log entries were going missing, then you could craft a very stealthy attack on the system and remain undetected in the logs.
- eesmith 3y ago
  Consider this point: > there's no guarantee that the message you're monitoring for won't change. Maybe someday the Linux kernel developers will decide to put more information in their MCE messages and change the format. How often do the messages change? How often do you need to re-train the AI? Is it worthwhile?
  - cobertos 3y ago
    A properly trained one shouldn't overfit. If it was trainer to just detect error lines and flag them for human review I think that'd be very beneficial Not as accurate as string matching though. And more time/compute intensive
    - eesmith 3y ago
      That doesn't seem to address my question. How much work does it take to train and maintain a property trained AI, given the three problems mentioned in the essay?
skissane 3y ago
> The fundamental problem with general log monitoring is that logs are functionally unstructured. JSON format logs? One pattern I’ve used before - in a Java app with an RDBMS, create an error table, and every unexpected Java exception gets logged to that table. Then have a system to match problems in that table to known errors, or flag new ones for analysis. (I just clone the table from the production DB into a non-production DB, then have a bunch of views I wrote to analyse them - the fact it is cloned to the non-production DB means I don’t need release management approval to update the definitions of my views.) At a certain scale this may lead to problems with performance or data volumes, but generally not if the app is small to medium scale (as in practice many apps are), and especially if exceptions are sufficiently infrequent (one hopes so!)
- jstanley 3y ago
  In my experience JSON format logs are not super useful if they boil down to: {"date":"2023-08-08 10:14:35.129532","log":"... normal unstructured log text ..."}
  - kubanczyk 3y ago
    Agreed, JSON is just a wire protocol. What TFA complains about is lack of contract on the part of logs' producers. (Schema would be a crucial part of such contract.)
  - taneliv 3y ago
    There are a few more that are fairly common to be logged somewhat separately: severity, host and component/service, but even if so, the structure is lacking in universality (i.e. lot of logging systems support this, but in their own ways). So, yeah, it's not much. In my limited experience the amount of lines logged correlated with expected or actual activity of the system(s) is perhaps useful for monitoring. Looking into and analyzing the actual log events and their text is then the next step if there seems to be too much or too little information logged.
  - magicalhippo 3y ago
    In .Net we're using Serilog which supports CLEF[1], and I'm in the process of changing our non-.Net code to do structured logging to CLEF as well (I made an internal library to support structured logging). Since Serilog supports consuming CLEF as well, this makes it trivial to upload the non-.Net logs to Azure Application Insights for example. Might be other options as well, I didn't look much further as this fit our needs well. edit: This doesn't completely solve the main point in the article of course, as the variables, error codes etc in the structured log message can change willy-nilly in a general system setting. [1]: https://clef-json.org/ https://clef-json.org/
dopidopHN 3y ago
That does not match recent experience with datadog or the GCP log aggregator We derived values out of that ( meaning : OPS having a playbook to troubleshot, alarms & prod support in general )
dncornholio 3y ago
It seems to me OP cannot see the difference between noise and information. Looking at logs more and you will build a feel of what is important. Better yet, write your own logs.
tudorw 3y ago
Is there an AI for this, asking for a friend?
Ensorceled 3y ago
This reminds me of the time the VP of engineering wanted to “just open telnet up” to solve a short term problem (this was a while ago) so I showed him the firewall logs of all the probes to the telnet port. Then we went down the rabbit hole … “what else is being probed”, “how do we report these people”, “block all those IP addresses”, etc. etc. Logging can be useful AND a rabbit hole. If you have the horsepower to manage the firehose, knock yourself out.
- sumtechguy 3y ago
  AH the 'i didnt know' side of logging. Once you start logging you probably will find all sorts of things going on you did not know about. Things you probably now need to address. I have also made the mistake of logging because I could. Logging without a plan I have found to not be worth it. If you are not going to take actions on those logs. Why are you keeping/making them? The flip side is when something breaks you will know you want those logs. Hopefully, you will have what you need in them (usually not). But then you turn on more and more as you need it and eventually find a happy medium.
- khiqxj 3y ago
  [dead]
- deathanatos 3y ago
  I've seen similar panic+rabbit holes on people seeing the HTTP request logs. They contain a constant stream of requests for WP vulns., phpMyAdmin vulns/cred stuffing, etc. (We ran none of this, so it shouldn't've been a concern.) It is amazing how many people (even ostensibly security people…) don't know that, if you're on the Internet, you're being subjected to a never ending torrent of it.
dale_glass 3y ago
Most logging is insanity in action. Behold: 1.2.3.4 - - [08/Aug/2023:12:48:11 +0200] "GET /wp-config.php.bak HTTP/1.1" 404 196 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2503.0 Safari/537.36" 1. Apache has a perfectly coherent and structured view of the situation: Host 1.2.3.4 made a GET request of this particular URL, with this particular UA, at this particular date. 2. Apache proceeds to shove all this into a string, destroying the structure, and probably some of the data (it likely knows the timestamp with >1s precision) 3. Then to do some sort of analysis we need to manually reverse this process: write some convoluted regex to match this kind of line, turn an IP address back into a number, turn a timestamp into a number, find the delimiters. We're now undoing the damage Apache did. Why!? 4. For extra fun this adds extra problems to deal with log rotation, log compression, newlines and special characters in fields, and so on. All kinds of problems that are ultimately unproductive to solve and get in the way of actual useful work. Logging should be structured by nature. I shouldn't be writing weird regexps. I should have every message structured from the start, where I can search by "ip_address is in 1.2.3/24", or "request == 'GET'", or such. Every bit of data should be logged in its pristine form with zero ambiguity, with its original type, and analyzable accordingly. I think systemd has the beginning of a good idea in here, in that you can actually do this if you care to, and can send arbitrary chunks of data (even binary) to the log if needed. People who complain about journald not logging in plaintext honestly baffle me, because come on, whose idea of fun is it to do log parsing? And why are we spending CPU cycles on converting timestamps to text and parsing them back?
- ranting-moth 3y ago
  You can customize the log format from Apache. I highly recommend it!
  - dale_glass 3y ago
    And it luckily has a journald module, which actually can log some fields in a structured manner. But even that doesn't seem to go far enough according to the docs. Eg, maybe I want structured logging for the HTTP headers. No way to get that at present, as far as I can tell.
nolist_policy 3y ago
> The first problem with monitoring this way is that there's no guarantee that the message you're monitoring for won't change. This is a solved problem (at least for the linux kernel) with printk indexing[0]. So you can query the running kernel what log messages it can output. With that your monitoring tool can check when an expected message was changed and you get the messages in printf format so you can parse them if you want. [0] https://docs.kernel.org/core-api/printk-index.html https://docs.kernel.org/core-api/printk-index.html
ranting-moth 3y ago
Monitoring logs is the last line of defence. If you're advocating against monitoring logs, then well, that's your opinion and certainly not mine.
sethammons 3y ago
Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including, where appropriate, copy-pastable curl commands to replicate an error to remote api endpoints!) and log aggregators like Splunk can group and aggregate and find relationships. If I see a spike in errors, I can, in real time, count them by user id, group by region, filter out by some other field, then have the table of events generate a live line chart easily showing the error rates over time, and pin pointing the exact moment an error started. Less than five to ten minutes after detection or report and I can be talking with another team about a change they released at time $x, including a list of affected users for product teams to reach out to if needed.
- junon 3y ago
  Doesn't the curl replication leak secrets, though?
  - sethammons 3y ago
    Depends on what you are doing. We mask auth stuff and some known params like an auth header might log as "authorization: bearer $APIKEY". Some things can't be logged. But as a general rule, it is an immense time saver.
  - cogman10 3y ago
    Yeah, structured logging has a tendency to do that as well. Logs are often a treasure trove for system invaders.
- hallman76 3y ago
  This sounds awesome - pretty disk-heavy but worth it. How long do you retain these logs? What sort of user/request volume do you deal with? Do you have any resources/blog posts/keywords for learning more?
  - jonhohle 3y ago
    For compressed logs, it doesn’t have to be significant. If you can get structured logs into a TSDB you really only need to retain logs in cold storage or for as long as more detailed views or correlation might be necessary.
alecco 3y ago
Just as it happens with backups, if you overlook the quality and utility of logs it will come back to haunt you, often costing 10x more time and effort.
- gjvc 3y ago
  this is an excellent notion.
zemo 3y ago
structured logging has been a thing for a long time, so I don’t really buy this argument. the problem with monitoring and alerting derived from logs is that log ingestion pipelines usually have the property that when they get saturated, they get behind in time and catch up later. That’s more or less the opposite of what you want for an alerting pipeline.
bdcravens 3y ago
I like monitoring my logs, but I hate making it an added cost and piece of infrastructure. While it lacks a ton of features that ideal solutions have, CloudWatch Log Insights is "good enough" for those times I need to comb, requires no additional infrastructure, and is dirt-cheap.
khiqxj 3y ago
[dead]
karmakaze 3y ago
What does the title intended to mean? It's summarized at the end. > But monitoring your logs looks productive and certainly sounds good and proper. You can write some positive matches to find known problems, you can write some negative matches to discard noise, you can 'look for anomalies' and then refine your filtering, and so on. That's what makes it a tarpit; it's quite easy to thoroughly mire yourself in progressively more complex log monitoring. Nothing surprising here. Spend only the time that's worthwhile and has a point. I don't think I've done any bottom-up (just because) log analysis other than counting logged exception types/sites.
h2odragon 3y ago
Compressor model log watcher. When a log entry has unusually high entropy score it higher. Add that to your regexp tables of "things to watch for" and even a very simple parser can give you a better first pass log filter than an untrained intern. "structured or not" is a valid discussion; but irrelevant to the point. We're not good at watching our logs for the things we wanted to see; nor are we good at emitting the data we want in the logs in the first place. Too many conflicting purposes in the same place.
- feisuzhu 3y ago
  Wow, that's really a brilliant idea. Is there any ready to use tools to implement that?
  - h2odragon 3y ago
    not that I know of. I had a lovely system that had audio alerts for all sorts of things and multiple time windows for comparison and statistics etc. long ago now, big mass of python script tuned for that environment. Made my office sound like a jungle: bird chirps for firewall probes, coin clatters and cash register "Kaching!" for sales page hits and actual "someone gave us money" sales, etc. It was fun.
LunaSea 3y ago
We used to split logs into two buckets at work, one containing parsable log lines that used the expected format and schema and then the rest in the second one. That way we could centralise all logs and detect different kinds of failures, like unexpected log lines appearing.
dkarl 3y ago
Request logging should be done by an application gateway; it will do a more honest job of it. In applications, don't log anything except: - application startup and shutdown (at INFO level) - occurrences that indicate a bug in the system logging (at ERROR level) - occurrences that indicate a bug in another of your systems (at WARN level) The last two require a lot of education and discipline to stop people from logging rare but valid circumstances, bad input originating outside your software, and "important" events. The answer to "but don't we want to know when..." arguments should always be, "If it's that important, emit it as a metric or store it in an appropriate datastore." Anything logged at WARN or ERROR should be something that can be addressed by fixing code. If you stick to this discipline, monitoring logs is valuable and reasonably simple. You can alert on any WARN and ERROR level logging. There's nothing like looking at a month of logs and seeing only actionable information and a handful of application lifecycle events. (There's also nothing like looking at six months of logs from a stable system and only seeing lifecycle events.) It is hard to stick to this discipline. You need a manager who believes in it and will back it up when engineers want to depart from it, and who will always prioritize fixing violations. Why is it so hard to make engineers follow these rules? Because initially, it makes them feel incredibly anxious. It feels irresponsible to throw away so much "important" information instead of logging it. After working this way for a while, though, they realize that it actually forces a higher level of responsibility. Logging information in noisy logs might feel better than throwing it away, but it's not. It creates an illusion of having handled the information responsibly. By taking away the illusory option, you force people to make a real decision between throwing it away and saving it in an actionable way, by emitting it as a metric or storing it in a datastore to be processed.
- rdoherty 3y ago
  I've been here with this and having logs also emitted by application/web servers is critical too. If you only have it at the gateway, and it emits a 503 or 504, did the request make it to the web server? Maybe? You have no signal. Seeing a timeout at the gateway/load-balancer and the web server log that it serviced the request successfully but took 45s tells you very critical information. If you didn't have the web log, you have a missing signal that tells you critical information. Despite having to managed TBs of logs per day and sift through them at times, I'd rather have too much logs. However, we did not alert on any of them. All alerting was symptoms based via SLOs (error rate, latency). Logs were only used for debugging.
emadda 3y ago
I recently released ‘lilo’, a CLI that downloads your GCP logs to a SQLite DB. It works best when you are logging structured data in the form of JSON lines though, so you can query the JSON structure. https://github.com/emadda/lilo https://github.com/emadda/lilo

119 points by signa11 3y ago

I generally feel that way regarding the log systems I've worked with, specifically since they were some of the first I helped implement and/or design, and thus I'd not consider them high quality. However, what would be better than monitoring logs? Any sort of monitoring I know of looks at logs produced by one system or another... I'm not sure if there is an alternative.

Xylakant 3y ago
Tracking specific metrics and monitoring those is an option - for example you could track service restarts and alert on unusually high numbers. The tricky part - just as with logging - is to figure out what metric make sense and give you signal and what metrics are mostly noise. Sometimes you can even extract metrics from logs by looking for specific log lines if you know what to expect.
- fluential 3y ago
  You often create metrics based on logs indicating certain events, not much progress here in that approach
- kubanczyk 3y ago
  > Sometimes you can even extract metrics from logs by looking for specific log lines This looks backwards from the app dev's perspective. - Metrics are strictly structured, documented data. They are a first-class output. - "Strictly structured logs" is contradiction in terms. These are just events to be sent to a pub/sub, also a first-class output. - Logs are unstructured, unknown, bleeding-edge, grey-area data, which can potentially become either metrics, or events, or bug tickets. If I output data as metrics, I don't output it again as logs - that'd be just noise. But of course other apps/systems can have a wholly different approach and force people to invest a lot into sorting out that dumpster fire. The leverage is on the producer side.
gjvc 3y ago
if we are talking about the "log files" aspect then one certainly needs to get away from that and have all log events in a single searchable interface.
thom 3y ago
Start treating reporting and monitoring as first class parts of your domain instead of some text. Do the same modelling/event storming exercises you’d do for everything else with these operational requirements.

acumenical 3y ago

This seems backwards to me in 2023. Logs seem like valuable data to train your AI on in order to automate as much of the basic inference away as possible.

darkclouds 3y ago
What if the logs were missing entries though? Would you train your AI on missing data? I found a bug in rsyslog a few years ago, where in certain conditions, log entries were missing. If you know the pattern for what log entries were going missing, then you could craft a very stealthy attack on the system and remain undetected in the logs.
eesmith 3y ago
Consider this point: > there's no guarantee that the message you're monitoring for won't change. Maybe someday the Linux kernel developers will decide to put more information in their MCE messages and change the format. How often do the messages change? How often do you need to re-train the AI? Is it worthwhile?
- cobertos 3y ago
  A properly trained one shouldn't overfit. If it was trainer to just detect error lines and flag them for human review I think that'd be very beneficial Not as accurate as string matching though. And more time/compute intensive
  - eesmith 3y ago
    That doesn't seem to address my question. How much work does it take to train and maintain a property trained AI, given the three problems mentioned in the essay?

skissane 3y ago

> The fundamental problem with general log monitoring is that logs are functionally unstructured. JSON format logs? One pattern I’ve used before - in a Java app with an RDBMS, create an error table, and every unexpected Java exception gets logged to that table. Then have a system to match problems in that table to known errors, or flag new ones for analysis. (I just clone the table from the production DB into a non-production DB, then have a bunch of views I wrote to analyse them - the fact it is cloned to the non-production DB means I don’t need release management approval to update the definitions of my views.) At a certain scale this may lead to problems with performance or data volumes, but generally not if the app is small to medium scale (as in practice many apps are), and especially if exceptions are sufficiently infrequent (one hopes so!)

jstanley 3y ago
In my experience JSON format logs are not super useful if they boil down to: {"date":"2023-08-08 10:14:35.129532","log":"... normal unstructured log text ..."}
- kubanczyk 3y ago
  Agreed, JSON is just a wire protocol. What TFA complains about is lack of contract on the part of logs' producers. (Schema would be a crucial part of such contract.)
- taneliv 3y ago
  There are a few more that are fairly common to be logged somewhat separately: severity, host and component/service, but even if so, the structure is lacking in universality (i.e. lot of logging systems support this, but in their own ways). So, yeah, it's not much. In my limited experience the amount of lines logged correlated with expected or actual activity of the system(s) is perhaps useful for monitoring. Looking into and analyzing the actual log events and their text is then the next step if there seems to be too much or too little information logged.
- magicalhippo 3y ago
  In .Net we're using Serilog which supports CLEF[1], and I'm in the process of changing our non-.Net code to do structured logging to CLEF as well (I made an internal library to support structured logging). Since Serilog supports consuming CLEF as well, this makes it trivial to upload the non-.Net logs to Azure Application Insights for example. Might be other options as well, I didn't look much further as this fit our needs well. edit: This doesn't completely solve the main point in the article of course, as the variables, error codes etc in the structured log message can change willy-nilly in a general system setting. [1]: https://clef-json.org/ https://clef-json.org/

dopidopHN 3y ago

That does not match recent experience with datadog or the GCP log aggregator We derived values out of that ( meaning : OPS having a playbook to troubleshot, alarms & prod support in general )

dncornholio 3y ago

It seems to me OP cannot see the difference between noise and information. Looking at logs more and you will build a feel of what is important. Better yet, write your own logs.

tudorw 3y ago

Is there an AI for this, asking for a friend?

Ensorceled 3y ago

This reminds me of the time the VP of engineering wanted to “just open telnet up” to solve a short term problem (this was a while ago) so I showed him the firewall logs of all the probes to the telnet port. Then we went down the rabbit hole … “what else is being probed”, “how do we report these people”, “block all those IP addresses”, etc. etc. Logging can be useful AND a rabbit hole. If you have the horsepower to manage the firehose, knock yourself out.

sumtechguy 3y ago
AH the 'i didnt know' side of logging. Once you start logging you probably will find all sorts of things going on you did not know about. Things you probably now need to address. I have also made the mistake of logging because I could. Logging without a plan I have found to not be worth it. If you are not going to take actions on those logs. Why are you keeping/making them? The flip side is when something breaks you will know you want those logs. Hopefully, you will have what you need in them (usually not). But then you turn on more and more as you need it and eventually find a happy medium.
khiqxj 3y ago
[dead]
deathanatos 3y ago
I've seen similar panic+rabbit holes on people seeing the HTTP request logs. They contain a constant stream of requests for WP vulns., phpMyAdmin vulns/cred stuffing, etc. (We ran none of this, so it shouldn't've been a concern.) It is amazing how many people (even ostensibly security people…) don't know that, if you're on the Internet, you're being subjected to a never ending torrent of it.

dale_glass 3y ago

Most logging is insanity in action. Behold: 1.2.3.4 - - [08/Aug/2023:12:48:11 +0200] "GET /wp-config.php.bak HTTP/1.1" 404 196 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2503.0 Safari/537.36" 1. Apache has a perfectly coherent and structured view of the situation: Host 1.2.3.4 made a GET request of this particular URL, with this particular UA, at this particular date. 2. Apache proceeds to shove all this into a string, destroying the structure, and probably some of the data (it likely knows the timestamp with >1s precision) 3. Then to do some sort of analysis we need to manually reverse this process: write some convoluted regex to match this kind of line, turn an IP address back into a number, turn a timestamp into a number, find the delimiters. We're now undoing the damage Apache did. Why!? 4. For extra fun this adds extra problems to deal with log rotation, log compression, newlines and special characters in fields, and so on. All kinds of problems that are ultimately unproductive to solve and get in the way of actual useful work. Logging should be structured by nature. I shouldn't be writing weird regexps. I should have every message structured from the start, where I can search by "ip_address is in 1.2.3/24", or "request == 'GET'", or such. Every bit of data should be logged in its pristine form with zero ambiguity, with its original type, and analyzable accordingly. I think systemd has the beginning of a good idea in here, in that you can actually do this if you care to, and can send arbitrary chunks of data (even binary) to the log if needed. People who complain about journald not logging in plaintext honestly baffle me, because come on, whose idea of fun is it to do log parsing? And why are we spending CPU cycles on converting timestamps to text and parsing them back?

ranting-moth 3y ago
You can customize the log format from Apache. I highly recommend it!
- dale_glass 3y ago
  And it luckily has a journald module, which actually can log some fields in a structured manner. But even that doesn't seem to go far enough according to the docs. Eg, maybe I want structured logging for the HTTP headers. No way to get that at present, as far as I can tell.

nolist_policy 3y ago

> The first problem with monitoring this way is that there's no guarantee that the message you're monitoring for won't change. This is a solved problem (at least for the linux kernel) with printk indexing[0]. So you can query the running kernel what log messages it can output. With that your monitoring tool can check when an expected message was changed and you get the messages in printf format so you can parse them if you want. [0] https://docs.kernel.org/core-api/printk-index.html https://docs.kernel.org/core-api/printk-index.html

ranting-moth 3y ago

Monitoring logs is the last line of defence. If you're advocating against monitoring logs, then well, that's your opinion and certainly not mine.

sethammons 3y ago

Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including, where appropriate, copy-pastable curl commands to replicate an error to remote api endpoints!) and log aggregators like Splunk can group and aggregate and find relationships. If I see a spike in errors, I can, in real time, count them by user id, group by region, filter out by some other field, then have the table of events generate a live line chart easily showing the error rates over time, and pin pointing the exact moment an error started. Less than five to ten minutes after detection or report and I can be talking with another team about a change they released at time $x, including a list of affected users for product teams to reach out to if needed.

junon 3y ago
Doesn't the curl replication leak secrets, though?
- sethammons 3y ago
  Depends on what you are doing. We mask auth stuff and some known params like an auth header might log as "authorization: bearer $APIKEY". Some things can't be logged. But as a general rule, it is an immense time saver.
- cogman10 3y ago
  Yeah, structured logging has a tendency to do that as well. Logs are often a treasure trove for system invaders.
hallman76 3y ago
This sounds awesome - pretty disk-heavy but worth it. How long do you retain these logs? What sort of user/request volume do you deal with? Do you have any resources/blog posts/keywords for learning more?
- jonhohle 3y ago
  For compressed logs, it doesn’t have to be significant. If you can get structured logs into a TSDB you really only need to retain logs in cold storage or for as long as more detailed views or correlation might be necessary.

alecco 3y ago

Just as it happens with backups, if you overlook the quality and utility of logs it will come back to haunt you, often costing 10x more time and effort.

gjvc 3y ago
this is an excellent notion.

zemo 3y ago

structured logging has been a thing for a long time, so I don’t really buy this argument. the problem with monitoring and alerting derived from logs is that log ingestion pipelines usually have the property that when they get saturated, they get behind in time and catch up later. That’s more or less the opposite of what you want for an alerting pipeline.

bdcravens 3y ago

I like monitoring my logs, but I hate making it an added cost and piece of infrastructure. While it lacks a ton of features that ideal solutions have, CloudWatch Log Insights is "good enough" for those times I need to comb, requires no additional infrastructure, and is dirt-cheap.

khiqxj 3y ago

[dead]

karmakaze 3y ago

What does the title intended to mean? It's summarized at the end. > But monitoring your logs looks productive and certainly sounds good and proper. You can write some positive matches to find known problems, you can write some negative matches to discard noise, you can 'look for anomalies' and then refine your filtering, and so on. That's what makes it a tarpit; it's quite easy to thoroughly mire yourself in progressively more complex log monitoring. Nothing surprising here. Spend only the time that's worthwhile and has a point. I don't think I've done any bottom-up (just because) log analysis other than counting logged exception types/sites.

h2odragon 3y ago

Compressor model log watcher. When a log entry has unusually high entropy score it higher. Add that to your regexp tables of "things to watch for" and even a very simple parser can give you a better first pass log filter than an untrained intern. "structured or not" is a valid discussion; but irrelevant to the point. We're not good at watching our logs for the things we wanted to see; nor are we good at emitting the data we want in the logs in the first place. Too many conflicting purposes in the same place.

feisuzhu 3y ago
Wow, that's really a brilliant idea. Is there any ready to use tools to implement that?
- h2odragon 3y ago
  not that I know of. I had a lovely system that had audio alerts for all sorts of things and multiple time windows for comparison and statistics etc. long ago now, big mass of python script tuned for that environment. Made my office sound like a jungle: bird chirps for firewall probes, coin clatters and cash register "Kaching!" for sales page hits and actual "someone gave us money" sales, etc. It was fun.

LunaSea 3y ago

We used to split logs into two buckets at work, one containing parsable log lines that used the expected format and schema and then the rest in the second one. That way we could centralise all logs and detect different kinds of failures, like unexpected log lines appearing.

dkarl 3y ago

Request logging should be done by an application gateway; it will do a more honest job of it. In applications, don't log anything except: - application startup and shutdown (at INFO level) - occurrences that indicate a bug in the system logging (at ERROR level) - occurrences that indicate a bug in another of your systems (at WARN level) The last two require a lot of education and discipline to stop people from logging rare but valid circumstances, bad input originating outside your software, and "important" events. The answer to "but don't we want to know when..." arguments should always be, "If it's that important, emit it as a metric or store it in an appropriate datastore." Anything logged at WARN or ERROR should be something that can be addressed by fixing code. If you stick to this discipline, monitoring logs is valuable and reasonably simple. You can alert on any WARN and ERROR level logging. There's nothing like looking at a month of logs and seeing only actionable information and a handful of application lifecycle events. (There's also nothing like looking at six months of logs from a stable system and only seeing lifecycle events.) It is hard to stick to this discipline. You need a manager who believes in it and will back it up when engineers want to depart from it, and who will always prioritize fixing violations. Why is it so hard to make engineers follow these rules? Because initially, it makes them feel incredibly anxious. It feels irresponsible to throw away so much "important" information instead of logging it. After working this way for a while, though, they realize that it actually forces a higher level of responsibility. Logging information in noisy logs might feel better than throwing it away, but it's not. It creates an illusion of having handled the information responsibly. By taking away the illusory option, you force people to make a real decision between throwing it away and saving it in an actionable way, by emitting it as a metric or storing it in a datastore to be processed.

rdoherty 3y ago
I've been here with this and having logs also emitted by application/web servers is critical too. If you only have it at the gateway, and it emits a 503 or 504, did the request make it to the web server? Maybe? You have no signal. Seeing a timeout at the gateway/load-balancer and the web server log that it serviced the request successfully but took 45s tells you very critical information. If you didn't have the web log, you have a missing signal that tells you critical information. Despite having to managed TBs of logs per day and sift through them at times, I'd rather have too much logs. However, we did not alert on any of them. All alerting was symptoms based via SLOs (error rate, latency). Logs were only used for debugging.

emadda 3y ago

I recently released ‘lilo’, a CLI that downloads your GCP logs to a SQLite DB. It works best when you are logging structured data in the form of JSON lines though, so you can query the JSON structure. https://github.com/emadda/lilo https://github.com/emadda/lilo