3 ms·

> Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archiv

> Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archive old logs so they don't hog the disk space. How do most people manage activity logs? Currently what we have set up is the user id (if the user is logged in), IP address, URL they hit, user agent, and timestamp are all inserted into an activity logs table. For one particular site with an API that's being polled the size of the DB grew pretty large.

daigoba66 12y ago
Logging every hit will always require a lot of space. But there are some tricks you can use to "compress" it: hash long strings like the URL and user agent and store the hash as binary instead of a string. A 100+ byte string can compress to just 16 or 32 bytes depending the hash your pick. Store the hash lookup in a separate table.
- mandeepj 12y ago
  what is the benefit of your approach? The lookup table will still have data growth issues
  - tonyarkles 12y ago
    Presumably there will be many more hits than URLs, i.e. the URL table will grow much slower than the Hits table.
chrissnell 12y ago
Off-machine logging. There are commercial services (we're using Papertrail but there are tons of them), roll-your-own solutions (Elasticsearch-Logstash-Kibana), and simple solutions (syslog). For an easy and simple solution, spin up a second instance and send logs to it via rsyslog over a private network interface. Most mature frameworks provide a method to send logs over syslog. It's UDP and very lightweight. Another plus: if you are compromised, you have another server with your logs and that server isn't running your vulnerable app.
- clogston 12y ago
  We're running the Elasticsearch, Logstash, Kibana (ELK) stack with the recommended approach i.e.: logstash client \ logstash client --> redis -> logstash "server" process -> elasticsearch <- kibana / logstash client We have a high logging load (we log every request) due largely to IRS requirements. I've been really happy with it over the past 6 months but something that cannot be overstated is that you'll really need to become familiar with each one of the technologies used as each requires it's own setup and configuration. Not being familiar with any of them, it took me a solid 3 days to get to where the whole thing was usable and performant. Troubleshooting it is a breeze, and the whole system scales really easily, but a lot of that time was invested up front.
  - agib 12y ago
    Just curious about the redis-in-the-middle. Any references so I can dig deeper?
tonglil 12y ago
We keep the last 30 days and also send it out to Rollbar for notifications and analysis. It's working great!
oisinkim 12y ago
> Logging, logging, logging - I can't stress on this enough. There is no easier way to offload, view, filter, alert and search than logentries: http://www.logentries.com http://www.logentries.com

daigoba66 12y ago

Logging every hit will always require a lot of space. But there are some tricks you can use to "compress" it: hash long strings like the URL and user agent and store the hash as binary instead of a string. A 100+ byte string can compress to just 16 or 32 bytes depending the hash your pick. Store the hash lookup in a separate table.

mandeepj 12y ago
what is the benefit of your approach? The lookup table will still have data growth issues
- tonyarkles 12y ago
  Presumably there will be many more hits than URLs, i.e. the URL table will grow much slower than the Hits table.

chrissnell 12y ago

Off-machine logging. There are commercial services (we're using Papertrail but there are tons of them), roll-your-own solutions (Elasticsearch-Logstash-Kibana), and simple solutions (syslog). For an easy and simple solution, spin up a second instance and send logs to it via rsyslog over a private network interface. Most mature frameworks provide a method to send logs over syslog. It's UDP and very lightweight. Another plus: if you are compromised, you have another server with your logs and that server isn't running your vulnerable app.

clogston 12y ago
We're running the Elasticsearch, Logstash, Kibana (ELK) stack with the recommended approach i.e.: logstash client \ logstash client --> redis -> logstash "server" process -> elasticsearch <- kibana / logstash client We have a high logging load (we log every request) due largely to IRS requirements. I've been really happy with it over the past 6 months but something that cannot be overstated is that you'll really need to become familiar with each one of the technologies used as each requires it's own setup and configuration. Not being familiar with any of them, it took me a solid 3 days to get to where the whole thing was usable and performant. Troubleshooting it is a breeze, and the whole system scales really easily, but a lot of that time was invested up front.
- agib 12y ago
  Just curious about the redis-in-the-middle. Any references so I can dig deeper?

tonglil 12y ago

We keep the last 30 days and also send it out to Rollbar for notifications and analysis. It's working great!

oisinkim 12y ago

> Logging, logging, logging - I can't stress on this enough. There is no easier way to offload, view, filter, alert and search than logentries: http://www.logentries.com http://www.logentries.com