9 ms·
Introducing Vector: Netflix's On-Host Performance Monitoring Tool
- Thaxll 12y agoNot sure what the difference is between that and Collect + Graphite / InfluxDB.
- Florin_Andrei 12y agoMake that Sensu + Graphite and you get fault detection and alerts too.
- easytiger 12y agoYea, but i guess they connect directly to the daemon on each host so no intermediate db. Which sounds like a terrible idea
- nathan_scott 12y agoThere are many, many differences - there's some books about PCP that might help clarify PCP design points, see: http://pcp.io/documentation.html http://pcp.io/documentation.html
- capkutay 12y agoThe charts look like they were built with nvd3. Can anyone confirm/deny?
- mspier 12y agoThat's right. Any suggestion of better reusable charts?
- capkutay 12y agoIMO nvd3 is one of the better high-level charting libraries in terms of number of charts, out-of-the box styles and interactivity.
- forrestthewoods 12y agoHonestly? Just write your own SVG code. If you inspect the elements the output data is super simple and easy to understand. NVD3 just wraps D3.js which just wraps utilities that output relatively basic data. Well, d3js is a data-binding system that's way too god damned complicated if all you want to do is make some simple charts which is how it's used 99% of the time. I've spent more time trying to manipulate chart libraries into doing almost the same thing but just different enough to cause pain and suffering. Output your own path data and it's a million times easier. For reference here's what I made: http://forrestthewoods.com/unbalanced-design-of-super-smash-brothers-part-3/ http://forrestthewoods.com/unbalanced-design-of-super-smash-...
- SEJeff 12y agoIt isn't d3 based (uses canvas instead of svg), but flot is fantastic as well: http://flotcharts.org http://flotcharts.org
- rubiquity 12y agoAm I wrong in saying this is a web interface that wraps PCP? So you can't really compare it to inspeqtor or collectd since those actually do the metrics collection.
- oimaz 12y agoWhere can I find deb packages for pcp version 3.10 or higher?
- lscotte 12y agoSee http://pcp.io/docs/installation.html#deb http://pcp.io/docs/installation.html#deb or direct link at ftp://ftp.pcp.io/projects/pcp/download/deb/ - but I only see packages for i386 arch, so you may have to compile from source if on amd64 or other architecture.
- jmedefind 12y agoThe git repo has a MakePkg script that will generate deb packages for you. I found it really easy to use.
- xbryanx 12y agoI'm curious how this compares to Zabbix's agent and server. Does PCP give you finer grained details, or is it possibly more lightweight?
- nathan_scott 12y agoPCP is much finer-grained than Zabbix in terms of the metrics it makes available (esp. from the Linux kernel); not sure on Zabbix costs but PCP is quite light on all resources (mem, cpu, net) and very robust. I've worked on production systems where everything else was failing (hardware, kernel, applications) but PCP kept chugging along, recording and telling the sad story to anyone that would listen.
- tdicola 12y agoWow this looks great, thanks for releasing it! Nice to see even 'simple' stuff like this that can help people who aren't running at Netflix scale is still released.
- CWuestefeld 12y agoIt's enormously frustrating to me that the Windows platform is so far behind. I've been researching for a rebuild of our ecommerce site with the idea of a modern microservice architecture. Team skillset and some other considerations dictate a .Net environment. Netflix and other companies have created a really rich platform, between logging and monitoring technologies like this; message queueing; deployment; and so on. In the Windows environment there's precious little to compare to. One bright spot is the news story also on HN today [1] about MS announcing Docker-related Hyper-V technology - but in the next version of Windows server. It might be that to satisfy those .Net compatibility wishes, we just go to Mono and do everything else in Linux. [1] https://news.ycombinator.com/item?id=9342369 https://news.ycombinator.com/item?id=9342369
- voltagex_ 12y agoI suspect that you could extend PCP (the monitoring component of this dashboard) to support Windows' performance counters. The profiling capabilities are certainly there.
- nathan_scott 12y agoPCP can export many hundreds of Windows-specific counters (which it accesses via the PDH APIs). The Windows build does need some love though, as the OP mentioned - its out of date and needs a regular contributor to bring it up to date with the latest PCP sources. Then Vector should Just Work (tm) for Windows servers too.
- tonyarkles 12y ago> Team skillset and some other considerations dictate a .Net environment. Not to be a dick but... that doesn't sound like a Windows limitation.
- dkhenry 12y agoSo I haven't looked into all the metrics that they are getting off of PCP but from an initial glance it doesn't look like they are getting a lot more information then you get off net-snmp. In this one case the windows platform isn't going to hold you back too much. On top of that anything the .net stack on windows gives you access to WMI which can give you really simple really deep data from your application. Don't buy into too much hype on what some of the big guys are doing on monitoring, most of their challenges seem to be around the scale they are monitoring at not with the depth of metrics.
- vizzah 12y agoCould anyone who tried this monitoring tool compare it to Munin?
- fapjacks 12y agoI just tried installing this on a test VPS. PCP went well, but whenever I tried to input a "hostname" in Vector's UI to get stats, it just kept telling me it couldn't connect and to check hostname, regardless of what I put in the box. PCP was available and of course ports open and whatnot. I'm not sure what the problem was, but that was a bit frustrating. It looks like a slick product otherwise, especially for a first version! Thanks for releasing projects like this! I'll be trying again in the future, for sure.
- mspier 12y agoCan you open an issue on GitHub and post more details? Any errors on the JavaScript console?
- victorhooi 12y agoThis is a slight tangent...but does anybody know what UI toolkit Netflix is using for this? Or if it's in-house, any info on whether they have, or might release it? I see bootstrap-submenu.css mentioned, but not Bootstrap itself: https://github.com/Netflix/vector/tree/master/app/css https://github.com/Netflix/vector/tree/master/app/css
- mspier 12y agoIt's bootstrap (see bower.json dependencies), but with our own layer on top of that.
- samstave 12y agoAfter using Stackdriver for the last 18 months, I would never go back to rolling my own monitoring infra if I could avoid it. I had nagios, cacti, munin, graphite all running and had two ops guys pretty much 80% of their time managing it. Stackdriver with pagerduty and I have 250,000 custom metrics being published and hundreds and hundreds of graphs on dozens of dashboards. Although, I am looking at SignalFX to give even better version of this, but I manage nearly 1,000 machines with only a staff of four.
- makeitsuckless 12y agoNever heard of it, so checked out Stackdriver. Seemed very interesting, especially since it's geared at AWS (which we use). Until I noticed the big red flag on the front page "Stackdriver is now part of Google." Surely Google is going to kill this product and use the knowledge to offer something similar for Google's cloud services.
- samstave 12y agoWe were concerned when.Google bought them and we had many meetings with them about this, I do not believe that will happen, but if that is still an issue for you, use signalFX.
- debaserab2 12y agoWhat does this have that CloudWatch enhanced metrics doesn't? From the screenshots, the metrics look pretty similar. Not a slight at all against this project (it looks awesome), I'm just curious if your infrastructure is already AWS based what would cause you to choose a non-CloudWatch option.
- thieving_magpie 12y agoIt looks like it has some metrics that aren't present in cloudwatch without a lot of scripting, such as Memory utilization. I'm new to cloud watch though, so someone please correct me if that is wrong.
- toomuchtodo 12y agoCloudWatch charges you per instance if you want 1 minute metrics instead of the standard (free) 5 minute metrics. Any tool that collects the data for you gets you out of that $3.50/instance/month detailed monitoring charge [1]. [1] http://aws.amazon.com/cloudwatch/pricing/ http://aws.amazon.com/cloudwatch/pricing/
- brendangregg 12y agoIt's one second metrics, and has been designed to allow us to add other metric sources and visualizations (using, say, ftrace, perf_events, SystemTap, etc), although those are not in the initial release.
- strunz 12y agoAnyone care to compare this to something like collectd? https://collectd.org/ https://collectd.org/
- mspier 12y agoGoal is a bit different. Vector doesn't collect and persist metrics. We needed something that had as little overhead as possible so it could be deployed to all our hosts and simplify the process of analyzing those metrics.
- falcolas 12y agoNeither does collectd, it just gathers and transmits the data to an endpoint you specify. It's also quite lightweight (once you address the memory leaks, at least).
- toomuchtodo 12y agoIf its not collecting and persisting metrics, is it more of a glorified htop?
- davidu 12y agoWait, really? Why not? Storage is cheap. Do you use something else to get historical visibility into metrics?
- toomuchtodo 12y agoGrafana, Graphite, StatsD, Skyline.
- zantana 12y agoI think the reason is they have a bunch of ephemeral hosts which they don't want to put in a central web interface or collect statistics on. They are only interested in getting some insight as to what went wrong with this host so they can fix it in the future.
- 12y ago
- philsnow 12y agoI got down to you should be able to install PCP from binary packages made available by the PCP development team on: ftp.pcp.io and that threw up a red flag in my brain. Then I noticed that techblog.netflix.com doesn't redirect to https (and indeed can't serve https) (I use https://www.eff.org/HTTPS-EVERYWHERE https://www.eff.org/HTTPS-EVERYWHERE and did not get content over https). The directions _I_ saw for building from source looked pretty innocuous, but you might see a different set of directions if you're being MITMed. Observe an appropriate amount of caution.
- mspier 12y agoHappy to discuss your concerns. PCP 3.10 should be available on Ubuntu's official repo pretty soon too.
- harshreality 12y agoHis concerns seem plain to me. Unauthenticated channels for software distribution or software installation instructions are bad. The techblog isn't using SSL, and the git pull url for PCP is using the git protocol which is also unauthenticated, rather than the authenticated https transport (ssh is only an option when user accounts make sense). Someone's at a conference and follows the link over public wifi. They get the same page but with "here's how to get PCP: ftp evil.io or git clone git://git.evil.io/pcp" Even if the webpage were ssl-enabled so that an attacker can't rewrite the pcp.io links, an attacker or evil network operator could MITM git.pcp.io or ftp.pcp.io. (FTP?!) Being in Ubuntu's repo doesn't make it safe if Ubuntu's maintainers have no (semi-)trustworthy way of getting the code.