3 ms·

If you haven't read the chapter of 12factor on logging I highly recommend it https://www.12factor.net/logs https://www.12factor.net/logs This is coming from an

If you haven't read the chapter of 12factor on logging I highly recommend it https://www.12factor.net/logs https://www.12factor.net/logs This is coming from an ops person, do that and I'll be happy. Essentially the goal is to externalize all your log routing to stdout, then wrap tooling around your application to route it wherever you want it to go. It's geared toward heroku but same rules apply in docker land and more traditional VM environments. We either send logs ECS -> Cloudwatch for our AWS stuff or docker swarm -> fluentd -> s3 for an onsite appliance (also anything syslog ends up in Sumologic through a collector we host in the environment). From there the logs get consumed by our SIEM (in our case Sumologic, which is a great service). We keep logs hot for 90 days and ship archives back to s3 for a year. Set up the lifecycle management stuff, keeping log files forever is not only a waste but can actively hurt you if they ever get exposed in a breach. I highly recommend formatting your logs JSON (even if you have to do it by hand like in Apache). If you do that and go to Splunk, Sumologic, or ELK all your fields will be populated either automatically or with a single filter. Saves writing or buying your own and if you add a field there's no action for you to take. nginx/apache default logging is complete trash. Look at the variables they expose, there's a lot of stuff in there you'll want to add to the log format to make your life loads easier. I have a format I use I'd be willing to send you if you want it. I don't recommend logstash (the L in ELK), ever (except maybe if you're java across the board). It's way too damn heavy to run on a workload host, fluentd is much lighter (and not java, why would I deploy java for a system tool ever?). Maybe as a network collector you throw syslog at but that would be it. For your use case Sumologics free service would be great. You can get I think up to 200m a day with a weeks retention for free and you'll get exposed to what an SIEM can do for you (ingestion rate and retention period are typically how hosted solutions are billed, you'll need email with any non-free email domain to get a free account from them). IMO you have to get to some fairly insane log rates for me to ever recommend running ELK stack yourself, it has way too much care and feeding if you want to run it correctly with good security.

edoceo 7y ago
I'm interested in these webserver configs you got for logs, please
- whatsmyusername 7y ago
  This is what I previously used for apache 2.4 (this very out of date for what I use now since the company I'm with isn't apache but the gist is there). I don't have the nginx one readily in front of me. I snipped a few fields out of this that were 'me setup' specific so small chance the formatting is off. The field variables for nginx are a hell of a lot less obtuse than the apache ones. LogFormat "{ \"application_name\":\"%v\", \"application_canonical-port\":\"%p\", \"application_client-ip\":\"%a\", \"application_local-ip\":\"%A\", \"application_local-port\":\"%{local}p\", \"application_pid\":\"%P\", \"fastly-client-ip\":\"%{fastly-client-ip}i\", \"request_x-forwarded-for\":\"%{X-Forwarded-For}i\", \"request_x-tracer\":\"%{X-TRACER}i\", \"request_geo-ip\":\"%{GEOIP_ADDR}e\", \"request_geo-continent\":\"%{GEOIP_CONTINENT_CODE}e\", \"request_geo-country\":\"%{GEOIP_COUNTRY_CODE}e\", \"request_host\":\"%{Host}i\", \"request_auth-user\":\"%u\", \"request_content-type\":\"%{Content-Type}i\", \"request_timestamp\":\"%t\", \"request_uri\":\"%r\", \"request_referer\":\"%{Referer}i\", \"request_user-agent\":\"%{User-Agent}i\", \"response_code\":\"%>s\", \"response_bytes\":\"%b\", \"response_seconds\":\"%T\", \"response_microseconds\":\"%D\", \"response_content-type\":\"%{Content-Type}o\" }" extendedcombined It'll come out looking like this (thrown through jsonlint.com) { "application_name": "%v", # Server name from the vhost config (not necessarily the hostname if you have aliases and depending on how you handle bare ip requests and if you use named vhosts "application_canonical-port": "%p", # 80 or 443 depending on if you're tls or not, I never figured out the point of this and it's actively confusing "application_client-ip": "%a", # The client ip calling apache, not necessarily the users IP if you have lbs/reverse proxies in the mix "application_local-ip": "%A", # Host ip the application is running on "application_local-port": "%{local}p", # Actual port the application is running on "application_pid": "%P", # Apache pid that handled the request "fastly-client-ip": "%{fastly-client-ip}i", # This is a header that the fastly service will add to tell you the actual client ip, we don't use them anymore but they're good (expensive). They actively defend this field, I spent an hour or so poking at it to see if I could add false data without success (of course if you don't protect your origins someone could falsify the data there instead). "request_x-forwarded-for": "%{X-Forwarded-For}i", # X-forwarded-for, occasionally useful if I suspect the client was monkeying with request data, there's a similar header for the protocol the load balancer saw in AWS land "request_x-tracer": "%{X-TRACER}i", # I used to use this pattern for when we wanted to set an arbitrary header we could trace a request with, usually by QA "request_geo-ip": "%{GEOIP_ADDR}e", # Relevant to mod_geoip, I don't use this anymore "request_geo-continent": "%{GEOIP_CONTINENT_CODE}e", # Relevant to mod_geoip, I don't use this anymore "request_geo-country": "%{GEOIP_COUNTRY_CODE}e", # Relevant to mod_geoip, I don't use this anymore "request_host": "%{Host}i", # Hostname the client sent "request_auth-user": "%u", # Relevant if you're using basic auth, usually not relevant "request_content-type": "%{Content-Type}i", # Content-Type the client requested, comes up if we expect request monkeying "request_timestamp": "%t", # Request timestamp "request_uri": "%r", # Request URI, I don't remember if this logs the get fields or not "request_referer": "%{Referer}i", # Request referer header if there is one "request_user-agent": "%{User-Agent}i", # Request user agent "response_code": "%>s", # Response we got "response_bytes": "%b", # Response size "response_seconds": "%T", # Response time (not necessarily how long it took the client to get it) in seconds "response_microseconds": "%D", # Same thing in microseconds "response_content-type": "%{Content-Type}o" # Response returned content-type header, occasionally relevant if we expect monkeying } When this makes it's way into Splunk or ELK they will automatically parse out the header fields. Sumologic will do it if you pass a query through "| json auto nodrop" Sumologic will handle sub structures (if you're passing an object where one of the fields is a hash object, not relevant for apache/nginx), I don't know about the others. Sumologic will intelligently handle non-json data (like the timestamps and tagging rsyslog adds).
Havoc 7y ago
>I have a format I use I'd be willing to send you if you want it. Throw the nginx one on a paste in please

edoceo 7y ago

I'm interested in these webserver configs you got for logs, please

whatsmyusername 7y ago
This is what I previously used for apache 2.4 (this very out of date for what I use now since the company I'm with isn't apache but the gist is there). I don't have the nginx one readily in front of me. I snipped a few fields out of this that were 'me setup' specific so small chance the formatting is off. The field variables for nginx are a hell of a lot less obtuse than the apache ones. LogFormat "{ \"application_name\":\"%v\", \"application_canonical-port\":\"%p\", \"application_client-ip\":\"%a\", \"application_local-ip\":\"%A\", \"application_local-port\":\"%{local}p\", \"application_pid\":\"%P\", \"fastly-client-ip\":\"%{fastly-client-ip}i\", \"request_x-forwarded-for\":\"%{X-Forwarded-For}i\", \"request_x-tracer\":\"%{X-TRACER}i\", \"request_geo-ip\":\"%{GEOIP_ADDR}e\", \"request_geo-continent\":\"%{GEOIP_CONTINENT_CODE}e\", \"request_geo-country\":\"%{GEOIP_COUNTRY_CODE}e\", \"request_host\":\"%{Host}i\", \"request_auth-user\":\"%u\", \"request_content-type\":\"%{Content-Type}i\", \"request_timestamp\":\"%t\", \"request_uri\":\"%r\", \"request_referer\":\"%{Referer}i\", \"request_user-agent\":\"%{User-Agent}i\", \"response_code\":\"%>s\", \"response_bytes\":\"%b\", \"response_seconds\":\"%T\", \"response_microseconds\":\"%D\", \"response_content-type\":\"%{Content-Type}o\" }" extendedcombined It'll come out looking like this (thrown through jsonlint.com) { "application_name": "%v", # Server name from the vhost config (not necessarily the hostname if you have aliases and depending on how you handle bare ip requests and if you use named vhosts "application_canonical-port": "%p", # 80 or 443 depending on if you're tls or not, I never figured out the point of this and it's actively confusing "application_client-ip": "%a", # The client ip calling apache, not necessarily the users IP if you have lbs/reverse proxies in the mix "application_local-ip": "%A", # Host ip the application is running on "application_local-port": "%{local}p", # Actual port the application is running on "application_pid": "%P", # Apache pid that handled the request "fastly-client-ip": "%{fastly-client-ip}i", # This is a header that the fastly service will add to tell you the actual client ip, we don't use them anymore but they're good (expensive). They actively defend this field, I spent an hour or so poking at it to see if I could add false data without success (of course if you don't protect your origins someone could falsify the data there instead). "request_x-forwarded-for": "%{X-Forwarded-For}i", # X-forwarded-for, occasionally useful if I suspect the client was monkeying with request data, there's a similar header for the protocol the load balancer saw in AWS land "request_x-tracer": "%{X-TRACER}i", # I used to use this pattern for when we wanted to set an arbitrary header we could trace a request with, usually by QA "request_geo-ip": "%{GEOIP_ADDR}e", # Relevant to mod_geoip, I don't use this anymore "request_geo-continent": "%{GEOIP_CONTINENT_CODE}e", # Relevant to mod_geoip, I don't use this anymore "request_geo-country": "%{GEOIP_COUNTRY_CODE}e", # Relevant to mod_geoip, I don't use this anymore "request_host": "%{Host}i", # Hostname the client sent "request_auth-user": "%u", # Relevant if you're using basic auth, usually not relevant "request_content-type": "%{Content-Type}i", # Content-Type the client requested, comes up if we expect request monkeying "request_timestamp": "%t", # Request timestamp "request_uri": "%r", # Request URI, I don't remember if this logs the get fields or not "request_referer": "%{Referer}i", # Request referer header if there is one "request_user-agent": "%{User-Agent}i", # Request user agent "response_code": "%>s", # Response we got "response_bytes": "%b", # Response size "response_seconds": "%T", # Response time (not necessarily how long it took the client to get it) in seconds "response_microseconds": "%D", # Same thing in microseconds "response_content-type": "%{Content-Type}o" # Response returned content-type header, occasionally relevant if we expect monkeying } When this makes it's way into Splunk or ELK they will automatically parse out the header fields. Sumologic will do it if you pass a query through "| json auto nodrop" Sumologic will handle sub structures (if you're passing an object where one of the fields is a hash object, not relevant for apache/nginx), I don't know about the others. Sumologic will intelligently handle non-json data (like the timestamps and tagging rsyslog adds).

Havoc 7y ago

>I have a format I use I'd be willing to send you if you want it. Throw the nginx one on a paste in please