One thing that's an absolute must: Put an ISO-8601 timestamp at the very beginning of every line in your log. No apache format, no "sun" or "jan" or other words

One thing that's an absolute must: Put an ISO-8601 timestamp at the very beginning of every line in your log. No apache format, no "sun" or "jan" or other words. ISO only. Seriously. If the timestamp is in a weird format (or, god help you, multiple formats since some libraries log shit in a special way), it'll be just about impossible to tell when things actually happened instead of just when the logging server saw them. In a perfect world these would be milliseconds apart, but lots of bad stuff can happen. Your log-grepping-guy will thank you.