3 ms·
Log format is the least of your problems (FWIW: ElasticSearch is happy to to take JSON.) A more interesting question is "can you actually learn from your logs?
by itamarst 10y ago
Log format is the least of your problems (FWIW: ElasticSearch is happy to to take JSON.)
A more interesting question is "can you actually learn from your logs?" Most logs are just isolated data points in time. I've built a logging system called Eliot for Python (https://eliot.readthedocs.io https://eliot.readthedocs.io) that lets you generate logs with causality: it's a tree of actions where you can see which action caused which other action, and whether each action failed. Actions can span processes, so it's also a tracing system.
This makes debugging complex systems much easier because instead of "X happened. Y happened. Z happened." you can see that Z happened because of X, but Y was unrelated.
(Output is JSON, easy to push into ElasticSearch.)
I'm sure there are other logging systems out there with similar features, or other approaches that work (combination of service-specific logging and protocol tracing, say). But the basic point is you should be thinking about what your goals are for the logs and what you want to learn from them... formats are easy to change and transform, but lost information is often lost forever.
- deleted 10y ago[deleted]