5 ms·

Sir, I can't tell if you're brilliant or insane. For those who didn't read the post: he's formatting syslog messages as redis wire commands, telling syslog to

by seiji 13y ago

Sir, I can't tell if you're brilliant or insane. For those who didn't read the post: he's formatting syslog messages as redis wire commands, telling syslog to forward those messages to redis over the network (normally you're forwarding to another syslog client, but because of the custom message format here, it acts against a redis server), so syslog ends up as a write-only redis client. Insane. and brilliant.

jeffjose 13y ago
Hacker. In all senses of the word. Brilliant post!
bonekeeper 13y ago
Brilliantly lazy, I would say, since he used out-of-box software to do the piping, he didn't have to write any code. My first inclination would be to format nginx's logs to a JSON array like such: log_format json '["$time_local","$msec","$remote_addr","$remote_user","$http_host","$request","$status","$upstream_addr","$upstream_cache_status","$bytes_sent","$request_time","$upstream_response_time","$http_referer","$http_user_agent","$http_x_forwarded_for","$http_cookie","$upstream_http_x_session_key","$upstream_http_x_session_user"]'; (just copied and pasted from my own config, many fields are not relevant to this use case in particular). Then log to a named pipe into a program which would then RPUSH the entries into a LIST or maybe PUBLISH them to a channel, from where several different analytics programs would be running for several different metrics that depend on the same data, so that they could run in parallel.
- throwaway2048 13y ago
  Can you please break up the 3rd line of your post, it is badly breaking the comment page layout.
  - derefr 13y ago
    The HN stylesheet really needs a "word-wrap: break-word;" in there somewhere.
    - dingdingdang 13y ago
      This is what I have in my Stylish extension for HN: @namespace url(http://www.w3.org/1999/xhtml); http://www.w3.org/1999/xhtml); @-moz-document domain("news.ycombinator.com") { p { word-break: break-all; } }
      - rschmitty 13y ago
        Awesome, thanks!
spc476 13y ago
He's already patching nginx to support logging via syslog, so why didn't he just patch nginx to log directly to redis? One less place for things to go wrong.
- seiji 13y ago
  Some ad contracts require you keep server logs of hits and not post-processed versions in your DB (in case any discrepancies come up). Running through syslog allows you to centralize where your logs are being processed/sent since each log hit can branch out to multiple endpoints.
  - spc476 13y ago
    Okay, that I can buy.

by seiji 13y ago

jeffjose 13y ago

Hacker. In all senses of the word. Brilliant post!

bonekeeper 13y ago

Brilliantly lazy, I would say, since he used out-of-box software to do the piping, he didn't have to write any code. My first inclination would be to format nginx's logs to a JSON array like such: log_format json '["$time_local","$msec","$remote_addr","$remote_user","$http_host","$request","$status","$upstream_addr","$upstream_cache_status","$bytes_sent","$request_time","$upstream_response_time","$http_referer","$http_user_agent","$http_x_forwarded_for","$http_cookie","$upstream_http_x_session_key","$upstream_http_x_session_user"]'; (just copied and pasted from my own config, many fields are not relevant to this use case in particular). Then log to a named pipe into a program which would then RPUSH the entries into a LIST or maybe PUBLISH them to a channel, from where several different analytics programs would be running for several different metrics that depend on the same data, so that they could run in parallel.

throwaway2048 13y ago
Can you please break up the 3rd line of your post, it is badly breaking the comment page layout.
- derefr 13y ago
  The HN stylesheet really needs a "word-wrap: break-word;" in there somewhere.
  - dingdingdang 13y ago
    This is what I have in my Stylish extension for HN: @namespace url(http://www.w3.org/1999/xhtml); http://www.w3.org/1999/xhtml); @-moz-document domain("news.ycombinator.com") { p { word-break: break-all; } }
    - rschmitty 13y ago
      Awesome, thanks!

spc476 13y ago

He's already patching nginx to support logging via syslog, so why didn't he just patch nginx to log directly to redis? One less place for things to go wrong.

seiji 13y ago
Some ad contracts require you keep server logs of hits and not post-processed versions in your DB (in case any discrepancies come up). Running through syslog allows you to centralize where your logs are being processed/sent since each log hit can branch out to multiple endpoints.
- spc476 13y ago
  Okay, that I can buy.