9 ms·

Another Redis case: Centralized logging

26 points by theonlyroot 15y ago

Would like to know what logging strategy people have been following in a distributed environment.

andrewvc 15y ago
My only question: how is this better than syslog?
- theonlyroot 15y ago
  I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
  - trentonstrong 15y ago
    Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways. I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible. On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!
    - tptacek 15y ago
      Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs. The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.
  - tedjdziuba 15y ago
    Ahem. tail -n 1000
- tptacek 15y ago
  It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate. Have you noticed how none of the cool kids postprocess http log files anymore?
  - moe 15y ago
    I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.
    - tptacek 15y ago
      Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
      - moe 15y ago
        Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.
        tptacek 15y ago
        If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
        15y ago
mkelly 15y ago
Depends on how much you care about latency, right? The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on how often you rotate.
- StavrosK 15y ago
  I prefer sending the logs to some other computer over UDP.
  - rarrrrrr 15y ago
    This only works if you can tolerate loss of some log items.
    - StavrosK 15y ago
      Very true. If you can, then it's a great way to log things.
      - deleted 15y ago
        [deleted]
      - prodigal_erik 15y ago
        Yikes, losing messages? If anyone designs a system that knowingly makes itself harder to diagnose under stress by destroying evidence, I only hope it's themselves and not their successors who are there to endure the pain.
        StavrosK 15y ago
        As I said, it depends. If you're logging cycle completion times or web page visits, you don't care about losing a few.
aedocw 15y ago
A similar concept, though using MongoDB with a capped collection is http://graylog2.org http://graylog2.org.
- deleted 15y ago
  [deleted]
ajays 15y ago
The article was fairly devoid of the necessary details. One of the main ones being: how many events are we talking about? 10 events/sec? 100/sec? 10_000/sec ? And what is the size of these events? How many event emitters are connecting to the Redis server? With the details, it would be a much more interesting post.
tedjdziuba 15y ago
Holy balls, talk about going out of your way to avoid syslog.
- tptacek 15y ago
  Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a great way to store log information. * Goofy little UDP messages are not a good way to convey system events * The syslog PRI field dates back to when we exchanged messages with UUCP. I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
  - zbailey 15y ago
    We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog. Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
    - tptacek 15y ago
      I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.
  - tedjdziuba 15y ago
    OK Thomas. First of all: u mad bro? Second of all: maybe on the day when you build something substantive, useful, and valuable, you can lecture me. Until then, take a knee and lend your ear. You know what else is a "relic of the past"? The internal combustion engine. It's wildly inefficient, but somehow, people keep on using it because it solves the problem. Maybe not optimally, but satisfactory. More to the point: * freeform text is an _excellent_ way to track system events. Know why? It's human readable. Anything more is a micro-optimization that is perpetrated by people who have never had to debug a serious production system. * Periodic rotation is fine, if you are storing a lot of "chatter" type logs. If you have critical events that need more careful tracking, then wowie zowie, syslog can do that too. It's almost like those Unix guys knew what they were doing. * Modern syslog implementations can use TCP, if that's your thing. Dumb if you ask me, but whatever. * So? What problem does that create? If you want to chest-thump about technical masturbation, then you're in the right place. Hacker News Wantrepreneurs love that kind of stuff. However, the rest of us who have work to do will use the tools available that solve the problem. Nothing is perfect, but I've got better ways to spend my time than worry about making my own centralized logging system. I ain't even mad.
antoncohen 15y ago
Take a look at logstash [1], it's on GitHub [2]. I think it could replace or integrate with RedisLogHandler. logstash takes logs from various inputs (syslog, files, Redis, HTTP), filters/normalized the formats into JSON, and outputs various formats (ElasticSearch, Redis, MongoDB, Graylog2). There is a WebUI with search and graphs. It's designed to scale-out and run on multiple machines. [1] http://logstash.net/ http://logstash.net/ [2] https://github.com/logstash/logstash https://github.com/logstash/logstash
rbucker 15y ago
I'm working on a universal subscriber, however, it currently does a nice job with redis. http://sub-watcher.com http://sub-watcher.com It forwards messages back to redis and to syslog. And it has several filtering options.

26 points by theonlyroot 15y ago

Would like to know what logging strategy people have been following in a distributed environment.

andrewvc 15y ago

My only question: how is this better than syslog?

theonlyroot 15y ago
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
- trentonstrong 15y ago
  Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways. I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible. On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!
  - tptacek 15y ago
    Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs. The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.
- tedjdziuba 15y ago
  Ahem. tail -n 1000
tptacek 15y ago
It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate. Have you noticed how none of the cool kids postprocess http log files anymore?
- moe 15y ago
  I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.
  - tptacek 15y ago
    Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
    - moe 15y ago
      Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.
      - tptacek 15y ago
        If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
        15y ago

mkelly 15y ago

Depends on how much you care about latency, right? The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on how often you rotate.

StavrosK 15y ago
I prefer sending the logs to some other computer over UDP.
- rarrrrrr 15y ago
  This only works if you can tolerate loss of some log items.
  - StavrosK 15y ago
    Very true. If you can, then it's a great way to log things.
    - deleted 15y ago
      [deleted]
    - prodigal_erik 15y ago
      Yikes, losing messages? If anyone designs a system that knowingly makes itself harder to diagnose under stress by destroying evidence, I only hope it's themselves and not their successors who are there to endure the pain.
      - StavrosK 15y ago
        As I said, it depends. If you're logging cycle completion times or web page visits, you don't care about losing a few.

aedocw 15y ago

A similar concept, though using MongoDB with a capped collection is http://graylog2.org http://graylog2.org.

deleted 15y ago
[deleted]

ajays 15y ago

The article was fairly devoid of the necessary details. One of the main ones being: how many events are we talking about? 10 events/sec? 100/sec? 10_000/sec ? And what is the size of these events? How many event emitters are connecting to the Redis server? With the details, it would be a much more interesting post.

tedjdziuba 15y ago

Holy balls, talk about going out of your way to avoid syslog.

tptacek 15y ago
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a great way to store log information. * Goofy little UDP messages are not a good way to convey system events * The syslog PRI field dates back to when we exchanged messages with UUCP. I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
- zbailey 15y ago
  We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog. Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
  - tptacek 15y ago
    I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.
- tedjdziuba 15y ago
  OK Thomas. First of all: u mad bro? Second of all: maybe on the day when you build something substantive, useful, and valuable, you can lecture me. Until then, take a knee and lend your ear. You know what else is a "relic of the past"? The internal combustion engine. It's wildly inefficient, but somehow, people keep on using it because it solves the problem. Maybe not optimally, but satisfactory. More to the point: * freeform text is an _excellent_ way to track system events. Know why? It's human readable. Anything more is a micro-optimization that is perpetrated by people who have never had to debug a serious production system. * Periodic rotation is fine, if you are storing a lot of "chatter" type logs. If you have critical events that need more careful tracking, then wowie zowie, syslog can do that too. It's almost like those Unix guys knew what they were doing. * Modern syslog implementations can use TCP, if that's your thing. Dumb if you ask me, but whatever. * So? What problem does that create? If you want to chest-thump about technical masturbation, then you're in the right place. Hacker News Wantrepreneurs love that kind of stuff. However, the rest of us who have work to do will use the tools available that solve the problem. Nothing is perfect, but I've got better ways to spend my time than worry about making my own centralized logging system. I ain't even mad.

antoncohen 15y ago

Take a look at logstash [1], it's on GitHub [2]. I think it could replace or integrate with RedisLogHandler. logstash takes logs from various inputs (syslog, files, Redis, HTTP), filters/normalized the formats into JSON, and outputs various formats (ElasticSearch, Redis, MongoDB, Graylog2). There is a WebUI with search and graphs. It's designed to scale-out and run on multiple machines. [1] http://logstash.net/ http://logstash.net/ [2] https://github.com/logstash/logstash https://github.com/logstash/logstash

rbucker 15y ago

I'm working on a universal subscriber, however, it currently does a nice job with redis. http://sub-watcher.com http://sub-watcher.com It forwards messages back to redis and to syslog. And it has several filtering options.