2 ms·
Glad you asked! I'm pretty proud of this hack :) uWSGI has a very flexible logging system. You can create a log with just requests, and completely customize th
by scrollaway 10y ago
Glad you asked! I'm pretty proud of this hack :)
uWSGI has a very flexible logging system. You can create a log with just requests, and completely customize the format of the line.
On top of that, uWSGI supports creating multiple log targets, and supports logging directly into a UDP socket.
If you make it match InfluxDB's line format (https://docs.influxdata.com/influxdb/v1.0/write_protocols/line/ https://docs.influxdata.com/influxdb/v1.0/write_protocols/li...), and you set up InfluxDB to accept connections from a matching UDP socket, then uWSGI can essentially log every request directly into influx.
Like this for example:
logformat = uwsgirequest,host="%(host)",status=%(status) msecs=%(msecs),size=%(size) %(epoch)000000000
logger = file:/var/logs/uwsgi/uwsgi.log
req-logger = file:/var/logs/uwsgi/requests.log
req-logger = socket:127.0.0.1:8083
You can also add 'path="%(var.PATH_INFO)"' to that but there is a potential data injection vulnerability if you don't re-parse the output.
With that, you can create nice graphs with Grafana to analyze your loading times, error rates, page sizes etc:
https://i.imgur.com/fzbXgUd.png https://i.imgur.com/fzbXgUd.png
You can also set up Kapacitor to identify when your page load times are abnormally increasing. We had the idea after a database connection leak increased our average loading times by 300%.
Do note that this doesn't batch inputs, so 1 request = 1 connection to influx. But it's pretty easy to put a middleware agent in between those to do validation and batching.
The nice thing about all this is it's application level timing metrics. There's not a lot of ways to get that, except with application middleware which can't always catch everything you want.
- chrissnell 10y agoSweet. Since it's UDP, it's connectionless so any penalty for not batching data points is likely minor. We're not a Python/uWSGI shop, but it looks like I might be able to emulate the Influx format as you have done using this nginx logging module: https://github.com/vkholodkov/nginx-udplog-module https://github.com/vkholodkov/nginx-udplog-module