4 ms·

My only question: how is this better than syslog?

theonlyroot 15y ago
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
- trentonstrong 15y ago
  Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways. I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible. On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!
  - tptacek 15y ago
    Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs. The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.
- tedjdziuba 15y ago
  Ahem. tail -n 1000
tptacek 15y ago
It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate. Have you noticed how none of the cool kids postprocess http log files anymore?
- moe 15y ago
  I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.
  - tptacek 15y ago
    Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
    - moe 15y ago
      Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.
      - tptacek 15y ago
        If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
        15y ago

I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.

trentonstrong 15y ago
Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways. I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible. On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!
- tptacek 15y ago
  Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs. The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.
tedjdziuba 15y ago
Ahem. tail -n 1000

tptacek 15y ago

It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate. Have you noticed how none of the cool kids postprocess http log files anymore?

moe 15y ago
I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.
- tptacek 15y ago
  Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
  - moe 15y ago
    Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.
    - tptacek 15y ago
      If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
      - 15y ago