4 ms·
I have a long rant about logging, but the TL;DR is: aggregated logging in general is useless don't bother, everything you need to know should be in your monitor
by jedberg 11y ago
I have a long rant about logging, but the TL;DR is: aggregated logging in general is useless don't bother, everything you need to know should be in your monitoring system. Basically, if something is broken and it's a problem, it will happen again, so you just start logging when something breaks. How do you know something breaks? Log your application metrics. Log your customer experience in the form of metrics and monitoring.
But don't aggregate every code exception and warning. If they aren't affecting your customer experience, who cares? And if they are affecting it, then it should be expressed as an application metric in your monitoring system.
Edit: To make my point a little clearer, I believe that central logging and aggregating should be available, just not turned on all the time.
- teacup50 11y agoWhat? The only way to find many issues is by looking at logs of unexpected (and especially unhandled) exceptions. If you have so many of those that they're impossible to dig through, FIX YOUR CODE.
- jedberg 11y agoYou can keep the logs on the local box and take a peek at them once in a while if you want to clean up the code (which you should), but there is no reason to spend a ton of money and time aggregating the logs.
- seiji 11y agoCentralized logging is critical to managing security. No real world production systems should be logging locally.
- jedberg 11y agoThe longer version of my rant stipulates that you have a good application firewall that is looking at the incoming traffic and you are using immutable infrastructure for the fronted so an attacker can't do much damage if they do get in. Keep in mind I consider the existence of exceptions to be an application metric that should be logged, so if there is a security issue causing exceptions that should show up in the monitoring, and then you can look at the exceptions that happen going forward. If the box is compromised, your local IDS should catch that (assuming you are allowing writes on the box at all)
- seiji 11y agoyou have a good application firewall that is looking at the incoming traffic I will concede if you have a perfectly configured all-knowing security oracle in front of your application then you don't need proper logging. :) immutable infrastructure for the fronted so an attacker can't do much damage if they do get in. I've seen attackers get access then just dump the site databases or copy all the site's code sitting on the compromised servers. Immutable infrastructure doesn't protect against reading things over the network. The class of security problems is larger and less clearly defined than what can be pre-programmed into static monitoring or analysis tools. It's always good to store logs for after-the-fact forensics if something does go wrong. How often do we see attacks, but then the attacked company doesn't even know what data was compromised or exfiltrated due to lack of logging? It happens daily. If you do happen to have a magic all-knowing security oracle, you should probably productize it and make a trillion dollars.
- jedberg 11y ago> I will concede if you have a perfectly configured all-knowing security oracle in front of your application then you don't need proper logging. Heh, that's not quite was I saying. :) I was just saying that any analysis you'll do on application logs you can also do with an application firewall. > How often do we see attacks, but then the attacked company doesn't even know what data was compromised or exfiltrated due to lack of logging? It happens daily. Again, I would contend that you won't find this information in application logs anyway. You'd find it in your IDS logs that are monitoring outbound network traffic.
- seiji 11y agoyou won't find this information in application logs anyway. Anecdote: I've previously caught an in-progress exploit by seeing mysql errors logged from the application because the exploits were doing dumb things like SELECT * against a table with tens or hundreds of millions of rows. Sometimes the little things let you know. IDS is nice in theory, but it's not really de rigueur for non-specialized platforms these days. It's about as practical as requesting companies keep complete netflow logs: great in theory, but almost nobody does it. If these security approaches were deployable in a complete drop-in fashion, we could push for industry wide adoption, but right now everything is custom tailored to individual architectures and environments. ain't nobody got time for that when there's hustlin' to be done and worse is better and perfect is the enemy of good and fail fast and be lean and flaunt your tail feathers towards all the VCs.
- ceejayoz 11y ago> You can keep the logs on the local box and take a peek at them once in a while... That doesn't work very well if you're using stuff like AWS Lambda functions, short-lived VMs, Docker containers that come and go regularly, etc. Plus, off-server logs are invaluable if the server is compromised.
- jedberg 11y ago> That doesn't work very well if you're using stuff like AWS Lambda functions, short-lived VMs, Docker containers that come and go regularly, etc. Actually that's the use case where it works best. If you have constantly changing infrastructure, how useful are those logs anyway? Unless its something that is happening across the fleet, in which case it should get picked up by your application metrics. Also this is the perfect case for stream processing of the logs where you look for things in real time and then throw away the actual logs (no need to keep them around after you've processed them). > Plus, off-server logs are invaluable if the server is compromised. Only if your systems are mutable. If they are immutable then you're much better off looking at the incoming data through an application firewall or looking for unexpected data changes in the data store. The application server should just be a conduit for the user to interact with the data store.
- jacques_chester 11y agoProposing the ideal world as a solution to the problems of the real world is an engineering antipattern with a long, storied and unhappy history.
- Zombieball 11y agoI personally don't understand this. Say your team owns 2 or 3 interconnected services. A customer reports an issue. How do you go about tracing the route cause? You don't know which server services a request so now you need to fetch logs from all your servers. So now you go and retrieve local logs from all of your servers to your local dev box (which could be considerable size given the size of your infrastructure and request volume) and then try and grep them? What if you need low level wire and request logs to debug the issue? After doing this enough times you are probably going to wish you had centralized and searchable log infrastructure.
- EvanAnderson 11y agoThis perspective completely ignores the use case of off-the-shelf software where you have limited or no capability to add "application metrics". Logs are invaluable in that scenario, which is arguably the majority of software deployments.
- jedberg 11y agoYou can extract application metrics from the local logs as they are output (and then redirected to /dev/null).
- shanemhansen 11y agoI've done this for Cassandra. While there are tons of jmx metrics exposed for Cassandra, last I checked the recommended way of logging compaction times was to tail the log. I even have a little project that tails logs and sends timestamps and values to graphite based on regexp: https://github.com/shanemhansen/graphitetailer https://github.com/shanemhansen/graphitetailer
- jacques_chester 11y agoLogging is for events with identity, metrics are for events without identity. They are not the same thing. Sometimes I care about the bulk behaviour of my application. I hit 2757 RPS and suddenly find a catastrophic gap in the performance landscape of my application -- that's useful. I don't care much about the particulars of the requests. What matters is the statistical behaviour. But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details. Metrics are warnings that smokers die younger, on average, than non-smokers. Logs are a biopsy showing which kind of lung cancer you have. No sensible production system or PaaS treats them as identical, even if you can derive metrics from logs.
- jedberg 11y ago> But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details. But my question is, why do you care about the outliers if the customer experience is not harmed by them? And if it is harmed, then it should should up in your 99th percentile metrics, even as an outlier, at which point you can turn on aggregated logging to look for that one thing. And if that one thing doesn't happen again, then did it really matter?
- mostlyjason 11y agoImagine customer support where you really need a record of the individual transactions for that particular customer. A monitoring solution won't give you that level of detail.
- jedberg 11y agoIndividual transactions aren't found in application logs, they are found in the data store. Also, again, it's important to stress that central logging should be available, just not on all the time. If it's a specific sequence of events that triggers a problem, turn on logging and then ask them to do the sequence again. If they don't need to do the sequence again, then it wasn't really of consequence. And if it happens for multiple users, then customer service should be able to turn on let the developers know to try and find the cause which they can do by adding more metrics or perhaps some temporary central logging.
- mugsie 11y agoCouple of reasons - 1. You need to stick to compliance requirements. 2. Tracking individual transactions (user complains about $Problem - if you attach a request ID to each request, and pass it between all components, you can find the exact problem for that request) 3. Building retrospective graphs is much easier when you have the data to hand, which you can them opitimise into a metric. Logging is not the only solution, but neither is metrics / monitoring only.
- henrik_w 11y agoWhere I used to work, we had a great system for logging/tracing only sessions we were interested in, while not generating any output for all the other sessions. I wrote about it here: http://henrikwarne.com/2014/01/21/session-based-logging/ http://henrikwarne.com/2014/01/21/session-based-logging/
- mugsie 11y agoThat looks really cool! Will have a look at how we could implement it. One problem is that we run across multiple services, on multiple machines, so we would have to have some way of passing a customer ID into our AZ level logstash forwarders, and find a way of collating ERROR level logs, and sending those transactions up. We also have the problem of customers asking for retrospective issue debug, so having all the logs is useful.
- agopaul 11y agoI'm impressed by how only a few people mentioned what I think it's the main point of application/system logging: debugging problems. Especially in small projects when you don't work 'at scale and you don't have metrics. Am I the only one that stash GBs of low-level log messages in order to diagnose bugs that occurred in the past?
- kiyoto 11y agoI think there are two kinds of logging that's conflated into two in the industry: logging for devops and logging for analytics. For logging for devops, I 100% agree with you. Looking at application metrics rather than raw logs is far more productive, and the raw logs should only be consulted after you have triaged the situation based on the metrics monitored. However, there is another kind of logging, and that's for data science and analytics. Here, it's hugely helpful to have centralized logging. Hell, it is a must. The last thing you want is to have data scientists with a shaky Linux knowledge to ssh into your prod machines. At the same time, logs are the best source of customer behavior data to inform product insights, etc. By centralizing these logs and making them available on S3 or HDFS or something, you can point them there and have everyone win. Among Fluentd users, we definitely see both camps. As a matter of fact, one of the reasons that I think people like Fluentd is that because it enables both monitoring and log aggregation within it.
- a3voices 11y agoOne alternative is to replicate some of your data into a different DB that is safe for the data scientists to use. And that way, they have raw data to play with instead of logs.
- nostrademons 11y agoCurious what you mean by "some of your data"? What kinds of data? Usually, I think of the logs as the raw data, everything else is derived data & analysis.
- a3voices 11y agoWell it would be data from your databases, and anything they would find useful excluding private user information.
- jedberg 11y agoYou're absolutely right. I was only talking about DevOps logging (should have made that clearer). Logging for data science is a totally different ball game.
- ketralnis 11y agoUnless you think you can predict every single bug that you'll ever write, and also how it will affect your customers' experience, how do you know you can track it at all? And if you're not logging it, once you do figure out that 50% of your customers are leaving because your payment endpoint throws an exception on Amex cards, how do you know what's causing that? Now it's just a silly detective game for something you could have been just logging in the first place. And if you're only able to find out via your customer metrics, it's already too late. If you were just logging the exception, you'd have said "huh that's funny" when you deployed the code, rather than when you have enough data to realise that you've already lost the customers. Personally I'd rather just have the logs.
- jedberg 11y ago> If you were just logging the exception, you'd have said "huh that's funny" when you deployed the code, rather than when you have enough data to realise that you've already lost the customers. I'm assuming that you still have the local logs on the local server and you're looking at one instance as you deploy to find those exceptions. I'm also assuming that we're talking about a system too complex to actually watch logs go by in real time. If the log level is light enough that you can actually read the logs as you deploy, then by all means do so. But if you're aggregating them just to be able to search them because you can't read them as they go by, then I'm saying there are better things to be doing than storing all those logs.
- pbowyer 11y agoHaving just started to go down the 'log everything centrally' route (to keep track of when third party services are down, what actions User 1234 performed the day they reported an issue, and which transactional emails seem to have disappeared), can you explain how I'd change this from logging to metrics & monitoring? One driver has been the lack of information I've had for debugging production issues (e.g. user X not able to log in against authentication provider Y) and so storing extra info is needed (vs the standard web server logs)
- pbowyer 11y agoThinking further, I guess I'm conflating logging for devops and logging for analytics and logging for audit trails. Not sure how to separate it out without data duplication (the boundaries aren't clear cut?) and for a small system it prob doesn't matter - I'm nowhere near Netflix, PayPal et al for scale.
- jedberg 11y agoA good rule of thumb is that if you can send all your logs to one place and tail -f is still a reasonable way to consume them, just send them all to one place.