3 ms·

Want to hear more about your HTTP analytics. Particularly interested in collecting selenium/sitespeed.io-style page timing metrics. There are a million commer

Want to hear more about your HTTP analytics. Particularly interested in collecting selenium/sitespeed.io-style page timing metrics. There are a million commercial solutions for this but so few open source options.

scrollaway 10y ago
Glad you asked! I'm pretty proud of this hack :) uWSGI has a very flexible logging system. You can create a log with just requests, and completely customize the format of the line. On top of that, uWSGI supports creating multiple log targets, and supports logging directly into a UDP socket. If you make it match InfluxDB's line format (https://docs.influxdata.com/influxdb/v1.0/write_protocols/line/ https://docs.influxdata.com/influxdb/v1.0/write_protocols/li...), and you set up InfluxDB to accept connections from a matching UDP socket, then uWSGI can essentially log every request directly into influx. Like this for example: logformat = uwsgirequest,host="%(host)",status=%(status) msecs=%(msecs),size=%(size) %(epoch)000000000 logger = file:/var/logs/uwsgi/uwsgi.log req-logger = file:/var/logs/uwsgi/requests.log req-logger = socket:127.0.0.1:8083 You can also add 'path="%(var.PATH_INFO)"' to that but there is a potential data injection vulnerability if you don't re-parse the output. With that, you can create nice graphs with Grafana to analyze your loading times, error rates, page sizes etc: https://i.imgur.com/fzbXgUd.png https://i.imgur.com/fzbXgUd.png You can also set up Kapacitor to identify when your page load times are abnormally increasing. We had the idea after a database connection leak increased our average loading times by 300%. Do note that this doesn't batch inputs, so 1 request = 1 connection to influx. But it's pretty easy to put a middleware agent in between those to do validation and batching. The nice thing about all this is it's application level timing metrics. There's not a lot of ways to get that, except with application middleware which can't always catch everything you want.
- chrissnell 10y ago
  Sweet. Since it's UDP, it's connectionless so any penalty for not batching data points is likely minor. We're not a Python/uWSGI shop, but it looks like I might be able to emulate the Influx format as you have done using this nginx logging module: https://github.com/vkholodkov/nginx-udplog-module https://github.com/vkholodkov/nginx-udplog-module

scrollaway 10y ago

Glad you asked! I'm pretty proud of this hack :) uWSGI has a very flexible logging system. You can create a log with just requests, and completely customize the format of the line. On top of that, uWSGI supports creating multiple log targets, and supports logging directly into a UDP socket. If you make it match InfluxDB's line format (https://docs.influxdata.com/influxdb/v1.0/write_protocols/line/ https://docs.influxdata.com/influxdb/v1.0/write_protocols/li...), and you set up InfluxDB to accept connections from a matching UDP socket, then uWSGI can essentially log every request directly into influx. Like this for example: logformat = uwsgirequest,host="%(host)",status=%(status) msecs=%(msecs),size=%(size) %(epoch)000000000 logger = file:/var/logs/uwsgi/uwsgi.log req-logger = file:/var/logs/uwsgi/requests.log req-logger = socket:127.0.0.1:8083 You can also add 'path="%(var.PATH_INFO)"' to that but there is a potential data injection vulnerability if you don't re-parse the output. With that, you can create nice graphs with Grafana to analyze your loading times, error rates, page sizes etc: https://i.imgur.com/fzbXgUd.png https://i.imgur.com/fzbXgUd.png You can also set up Kapacitor to identify when your page load times are abnormally increasing. We had the idea after a database connection leak increased our average loading times by 300%. Do note that this doesn't batch inputs, so 1 request = 1 connection to influx. But it's pretty easy to put a middleware agent in between those to do validation and batching. The nice thing about all this is it's application level timing metrics. There's not a lot of ways to get that, except with application middleware which can't always catch everything you want.

chrissnell 10y ago
Sweet. Since it's UDP, it's connectionless so any penalty for not batching data points is likely minor. We're not a Python/uWSGI shop, but it looks like I might be able to emulate the Influx format as you have done using this nginx logging module: https://github.com/vkholodkov/nginx-udplog-module https://github.com/vkholodkov/nginx-udplog-module