8 ms·

scales: Greplin's new open source Python server metrics library

83 points by rwalker 15y ago

rarrrrrr 15y ago
We open sourced something similar called StatGrabber awhile back, including Perl and Python client libraries. We tend to avoid threads, and so it instead uses non blocking UDP (guaranteed delivery on localhost) to a collector daemon, which aggregates and then delivers the info to Ganglia for graphing. https://spideroak.com/code https://spideroak.com/code if anyone is curious. The client modules simply emit non-blocking UDP packets and get on with their business, avoiding slowing down their response time. You can graph 4 types of stats: counters (ex: transactions) averages (ex: size of transactions), accumulators (ex: bandwidth used) and elapsed time (ex: time per transaction) It's pretty nice to see Ganglia graphing system metrics along side of all the stats we have the backend emit. For example, one of the stats graphed are revenue events. There's clear relationship between network health issues and revenue.
- gnubardt 15y ago
  We wrote something similar at Brightcove to collect system statistics and publish them graphite. It runs as an independent service and focuses on OS (not application) level metrics. https://github.com/BrightcoveOS/Diamond https://github.com/BrightcoveOS/Diamond
  - pjscott 15y ago
    That looks really convenient and useful.
  - deleted 15y ago
    [deleted]
  - sciurus 15y ago
    That looks like nice work, but I'm curious why you needed to develop your own system statistics gathering system rather than use an existing one like http://collectd.org/ http://collectd.org/ (which can be connected to graphite via https://github.com/joemiller/collectd-graphite https://github.com/joemiller/collectd-graphite)
- SkyMarshal 15y ago
  <3 SpiderOak. Any reason you guys don't host or mirror your code on Github? Makes it much easier for us curious tinkerers to keep your stuff organized and 'top of mind' among the many other things we fork, grok, and hack at.
  - simonw 15y ago
    I'll second that - it was actually a bit of a surprise seeing a recent open source project where I had to download and extract an archive rather than browsing the source code online.
    - rarrrrrr 15y ago
      patio11 explained it best here, I think. http://www.kalzumeus.com/2010/01/24/startup-seo/ http://www.kalzumeus.com/2010/01/24/startup-seo/ That said, we really need to at least put up the git browser so you can browse on the site. Thanks for the reminder.
      - SkyMarshal 15y ago
        Good writeup but long. Are you saying that keeping the code on your own site improves SEO?
        StavrosK 15y ago
        It gets them backlinks.
- moreati 15y ago
  > UDP (guaranteed delivery on localhost) That's not something I've heard before. Is it generally true of localhost-UDP? OS Specific? Particular to your usage?
  - rarrrrrr 15y ago
    I've tried to find where I originally read that. IIRC, it's a particularity of the Linux implementation, and of course only applies in the case that there's actually a process listening. I'll keep looking and post back if I run across a proper reference.
    - eis 15y ago
      I doubt non-blocking anything has guaranteed delivery on localhost if you don't account for full buffers. As long as there is a buffer, sure, why would UDP lose packets on localhost? In order to not block your program, you'll have to do it like one always has with non-blocking sockets: have a buffer and poll for the "write-able" event. Unix sockets would involve less overhead and provide more security if you only send to localhost. I really don't see a reason to use UDP on localhost at least if you are not running on Windows.
- beagle3 15y ago
  You could use non blocking TCP, and not worry about where the listening process, and have some idea if there's problem. Now, you're just ignoring the possibility of a problem, which might be fine and dandy, but you're not actually getting anything in return (well, about 5 lines of C code to set up the TCP connection in non-blocking mode, and tear it down if there's an error).
deleted 15y ago
[deleted]
99tokens 15y ago
that reminds me to put 2 of three puzzle solutions online....
mattlong 15y ago
Correct me if I'm mistaken, but it seems like this wouldn't really work with a web server that has several worker processes like uwsgi in preforking mode. Each worker process would be sandboxed to its own STATS object and attempting to serve the HTTP/Graphite server in its own background thread; all of which would be trying to access the same port.
- pjscott 15y ago
  Correct. This is intended more for daemons with one or two processes per server, or for handy debugging of a single instance on a development machine. In order to monitor something like preforking uwsgi, you would need a way to aggregate stats from all the processes, which this library doesn't do. (You can change the port the HTTP server listens on, though. If you have ten workers, you can have them listen on ten different ports.)
deleted 15y ago
[deleted]

83 points by rwalker 15y ago

We open sourced something similar called StatGrabber awhile back, including Perl and Python client libraries. We tend to avoid threads, and so it instead uses non blocking UDP (guaranteed delivery on localhost) to a collector daemon, which aggregates and then delivers the info to Ganglia for graphing. https://spideroak.com/code https://spideroak.com/code if anyone is curious. The client modules simply emit non-blocking UDP packets and get on with their business, avoiding slowing down their response time. You can graph 4 types of stats: counters (ex: transactions) averages (ex: size of transactions), accumulators (ex: bandwidth used) and elapsed time (ex: time per transaction) It's pretty nice to see Ganglia graphing system metrics along side of all the stats we have the backend emit. For example, one of the stats graphed are revenue events. There's clear relationship between network health issues and revenue.

gnubardt 15y ago
We wrote something similar at Brightcove to collect system statistics and publish them graphite. It runs as an independent service and focuses on OS (not application) level metrics. https://github.com/BrightcoveOS/Diamond https://github.com/BrightcoveOS/Diamond
- pjscott 15y ago
  That looks really convenient and useful.
- deleted 15y ago
  [deleted]
- sciurus 15y ago
  That looks like nice work, but I'm curious why you needed to develop your own system statistics gathering system rather than use an existing one like http://collectd.org/ http://collectd.org/ (which can be connected to graphite via https://github.com/joemiller/collectd-graphite https://github.com/joemiller/collectd-graphite)
SkyMarshal 15y ago
<3 SpiderOak. Any reason you guys don't host or mirror your code on Github? Makes it much easier for us curious tinkerers to keep your stuff organized and 'top of mind' among the many other things we fork, grok, and hack at.
- simonw 15y ago
  I'll second that - it was actually a bit of a surprise seeing a recent open source project where I had to download and extract an archive rather than browsing the source code online.
  - rarrrrrr 15y ago
    patio11 explained it best here, I think. http://www.kalzumeus.com/2010/01/24/startup-seo/ http://www.kalzumeus.com/2010/01/24/startup-seo/ That said, we really need to at least put up the git browser so you can browse on the site. Thanks for the reminder.
    - SkyMarshal 15y ago
      Good writeup but long. Are you saying that keeping the code on your own site improves SEO?
      - StavrosK 15y ago
        It gets them backlinks.
moreati 15y ago
> UDP (guaranteed delivery on localhost) That's not something I've heard before. Is it generally true of localhost-UDP? OS Specific? Particular to your usage?
- rarrrrrr 15y ago
  I've tried to find where I originally read that. IIRC, it's a particularity of the Linux implementation, and of course only applies in the case that there's actually a process listening. I'll keep looking and post back if I run across a proper reference.
  - eis 15y ago
    I doubt non-blocking anything has guaranteed delivery on localhost if you don't account for full buffers. As long as there is a buffer, sure, why would UDP lose packets on localhost? In order to not block your program, you'll have to do it like one always has with non-blocking sockets: have a buffer and poll for the "write-able" event. Unix sockets would involve less overhead and provide more security if you only send to localhost. I really don't see a reason to use UDP on localhost at least if you are not running on Windows.
beagle3 15y ago
You could use non blocking TCP, and not worry about where the listening process, and have some idea if there's problem. Now, you're just ignoring the possibility of a problem, which might be fine and dandy, but you're not actually getting anything in return (well, about 5 lines of C code to set up the TCP connection in non-blocking mode, and tear it down if there's an error).

deleted 15y ago

[deleted]

99tokens 15y ago

that reminds me to put 2 of three puzzle solutions online....

mattlong 15y ago

Correct me if I'm mistaken, but it seems like this wouldn't really work with a web server that has several worker processes like uwsgi in preforking mode. Each worker process would be sandboxed to its own STATS object and attempting to serve the HTTP/Graphite server in its own background thread; all of which would be trying to access the same port.

pjscott 15y ago
Correct. This is intended more for daemons with one or two processes per server, or for handy debugging of a single instance on a development machine. In order to monitor something like preforking uwsgi, you would need a way to aggregate stats from all the processes, which this library doesn't do. (You can change the port the HTTP server listens on, though. If you have ten workers, you can have them listen on ten different ports.)

deleted 15y ago

[deleted]