6 ms·
Monitoring Cloudflare's edge network with Prometheus
- bebop 9y agoHere is the video if you want to watch the full talk: https://www.youtube.com/watch?v=lHtY7TUsLzk https://www.youtube.com/watch?v=lHtY7TUsLzk
- MetalMatze 9y agoThe talk originally given at PromCon 2017: https://www.youtube.com/watch?v=ypWwvz5t_LE https://www.youtube.com/watch?v=ypWwvz5t_LE
- the_evacuator 9y agoPrometheus is an escaped implementation of Google’s borgmon, which is seen inside Google as a kind of horror show, and alternatives have been developed. It is kind of frightening that it has got out in the wild and people like it.
- orf 9y agoSo you have any reason to not like it?
- the_evacuator 9y agoNot really. I just think it’s funny to see it described by one groups as a reasonable or even state-of-the-art system, while another group describes it as brain damage from ten years ago.
- delroth 9y agoBorgmon being "brain damage" or some kind of "horror show" is more a meme than a serious opinion held by people who have used Borgmon. Personally I'm very happy that the open source world is adopting something derived from Borgmon rather than something derived from its supposed "replacement".
- bogomipz 9y agoI have heard unflattering things about it from a few different ex-Google SREs, specifically about the macro system and it being cumbersome to use.
- SuperQue 9y agoYup, I've had more than one current Google SRE state "You can have my Borgmon when you pry it from my cold dead hands.". Borgmon may be dead in the eyes of some people, but I know for a fact that it's still the only thing monitoring core and critical systems. Most of the problem with Borgmon, IMO, is the cruft that has built up over the decade+, and neglect due to the Google pattern of "The new thing that doesn't work, and the old thing that is deprecated.". The difficulty at Google is that developers are rewarded for writing new and shiny from scratch, rather than fix the old but working systems. This isn't always a problem, as some good things can come out of starting from scratch. But sometimes they throw out too many of the good ideas, in an attempt to be fancy and new.
- user5994461 9y agoIt's pretty normal when you consider that google is 10 years ahead of almost every other company when it comes to infrastructure.
- toomuchtodo 9y ago> is 10 years ahead of almost every other company when it comes to infrastructure. For Google-scale orgs or infrastructure needs. Most everyone else in the world does not need Google scale tools.
- user5994461 9y agoGoogle needs are extremely common. Take a look at any Fortune 500 and and it could usually benefit greatly from a lot of the infrastructure that powers google. Most of them do run their own datacenters, sometimes in numerous locations, they have massive and extremely complex IT systems in place.
- deleted 9y ago[deleted]
- clhodapp 9y agoWhy is borgmon considered a horror show? Is something fundamentally flawed in the model? How do the alternatives differ from the original borgmon?
- adrianratnapala 9y agoBorgmon's language is weird and crusty. People can debate whether this is the main problem with Borgmon or whether more fundamental changes are necessary. I won't weight in on that debate. But you can think of Prometheus as an experiment to decide the issue: it is very similar to Borgmon, but has a cleaner language.
- kyrra 9y agoGoogle has an alternative that they gave a talk on back in December. Sadly there aren't any papers on it yet. It's called Monarch and it's what backs up Stackdriver. It's config language is less crazy (Python based) and operates globally. https://www.youtube.com/watch?v=LlvJdK1xsl4 https://www.youtube.com/watch?v=LlvJdK1xsl4 Edit: Monarch config isn't sane, it's just different and at least not in the crazy languages that borgmon uses.
- packetslave 9y ago"sane" is an interesting choice of words to describe Monarch configuration...
- alin-sinpalean 9y agoI will point out that borgmon's language (minus macros) is almost a 1:1 match with Prometheus. You can judge for yourself how crazy that is, but I feel that it's close to as simple as you can get for the power it gives you. As for Monarch, it's a very different beast. For one, it stores all its rules in a protocol buffer format, so it's more structured. But then you have to write Python code that generates the protocol buffers and pushes them to storage. It looks similar but not the same as the ad-hoc query language. I wouldn't go as far as calling it sane. It is also a service and it's optimized for Google's network architecture with datacenter local and global nodes and the language itself is aware of this distinction and some computations are done locally, others globally and so on. For your local monitoring needs (or even global ones, if you're willing to put in the effort), Prometheus is a solid choice.
- mbertschler 9y agoAnd what would be a better alternative available outside of Google?
- user5994461 9y agoThe decade old, open source, self hosted, debug it yourself standard for monitoring is collectd+graphite+grafana. The equivalent easy to setup and use, with ALL the features working out of the box, SaaS standard is https://www.datadoghq.com/ https://www.datadoghq.com/ or potentially Google Stack driver if you are on Google Cloud.
- toomuchtodo 9y agoPlugging https://www.librato.com/ https://www.librato.com/ after reading an HN thread yesterday that DataDog pricing is insane [1]. [1] https://news.ycombinator.com/item?id=15315028 https://news.ycombinator.com/item?id=15315028 Disclaimer: No relation to either org.
- user5994461 9y agoThe thread you are linking to is a complete joke. The guy didn't see that the pricing was per host, even though it's written in big letters. His whole series of rant is ridiculous.
- kyrra 9y agoStackdriver works with AWS. https://cloud.google.com/monitoring/quickstart-aws https://cloud.google.com/monitoring/quickstart-aws
- user5994461 9y agoStack Driver is a Google Cloud service, you need a Google account with billing to use it. It can gather metrics from hundreds of vendors, including AWS and Azure.
- lima 9y agoSome details: https://landing.google.com/sre/book/chapters/practical-alerting.html https://landing.google.com/sre/book/chapters/practical-alert...
- packetslave 9y ago"Borgmon is the worst form of monitoring, except for all those other forms that have been tried from time to time." -- Winston Churchill if he worked at Google.
- bbrazil 9y ago> alternatives have been developed Such as Prometheus :) Even some teams in Google use it.
- alin-sinpalean 9y agoHaving worked both on borgmon and with borgmon, I can think of a few reasons why some (many?) don't like it all that much: 1. As already mentioned, the macro system (and the fact that its use is basically required to set up basic monitoring) has quite a steep learning curve. Prometheus doesn't have that and I personally would prefer it did. 2. It's not a service, so you have to set up your own instance, configure it, maintain it. In many engineers' mind this is just another hurdle in front of them launching their service. 3. As a software engineer (particularly new to Google) you might not expect to have to do ops work and carry a pager yourself. Out of all three, only (1) is a valid reason to hate on borgmon. That and the language itself, which is almost a 1:1 match with Prometheus, are very different from your regular programming language. But given the choice between flat, simple metrics (which is what most monitoring systems give you) and the ability to have arbitrary dimensions and be able to work with them to build useful alerts and dashboards and troubleshoot quickly, I (again personal opinion) will always go with the latter.
- caust1c 9y agoIt wasn't universally liked at Cloudflare. The federation component in particular is a PITA.
- deleted 9y ago[deleted]
- lima 9y agoI don't get the hype around Prometheus. What makes its pull-based mechanism superior to push-based ones like statsd? And using exporters sounds clunky - instead of directly querying a metric and sending it to your metrics collector, you have an intermediate component which exposes them for collection.
- tzumby 9y agoI might be wrong but one reasoning I saw mentioned was that pulling would give you a better picture into failures. If one of the services you are monintoring goes down it won't be able to push anything.
- lima 9y agoBut where's the difference between not being able to pull vs. not receiving metrics for a period of time?
- sz4kerto 9y ago> But where's the difference between not being able to pull vs. not receiving metrics for a period of time? Let's say I have a service that's only used once per day, at a random point in the day. I need to know if the service is available all the time, even if it's not used. With pull, I'll get back an 'OK' e.g. every minute. You could say that we should implement heartbeats, but again, then this service needs to be configurable to pick up changes in the monitoring infrastructure, then I need to have a separate check for the presence of heartbeats and the values of the samples..
- zaroth 9y agoMonitoring if the process is running, if the input queue is free, if it will take a new job, etc. seems to me the responsibility for all that should be with whatever supervisor has the ability to reset/restart the service. Then it can report stats on uptime and number of restarts required. What if the service spins up on demand when it's needed, does its job, and then exits. You want to track how often it starts, how long it runs, how long the job was in the queue, how much CPU/RAM/IO was consumed, etc. With push that's all part of the cleanup code in the process. With pull it seems like you could completely miss that the process was even ever run?
- bogomipz 9y ago>"Monitoring Cloudflare's planet-scale edge network with Prometheus" Is "planet scale" better than "web scale"?
- dsl 9y agoIt's just Cloudflare trying to make themselves sound bigger. Still nowhere near Google or Amazon.
- user5994461 9y agoThey are similar in terms of users, requests and bandwidth.
- bogomipz 9y agoDo you have a citation for that claim?
- user5994461 9y agoThey serve 10% of the internet traffic and have over 1 billion users. They receive more than 5 million requests per second in average.
- bogomipz 9y agoThat's not really a citation. Where are these figures published? What does that mean "10% the internet traffic?" By bandwidth? By request volume?
- user5994461 9y agoYou could start by reading the article you are commenting on. The traffic is by requests. You may not be aware but yes, cloudflare is a significant internet company.
- irl_zebra 9y agoWorking with Prometheus for a while. Only sticking point has been the 15 day storage of metrics. Anyone know why this is the limit on storage, or any strategies for long term storage of metrics?
- lobster_johnson 9y agoAdd "-storage.local.retention" (e.g. "-storage.local.retention=1440h" for 60 days) when running Prometheus.
- alaties 9y agoAs lobster_johnson stated, Prometheus storage retention is a command line launch option. Prometheus is not intentionally designed as a long term cold storage option for metrics. You _can_ store metrics for as long as your storage allows, but Prometheus is not going to replicate or manage that data to prevent long term degradation. Depending on your long term needs, the preferred pattern is to roll data off Prometheus into a metrics store that better handles data over the scope of months. Rolling data off of Prometheus is done with an exporter and documented [here](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage https://prometheus.io/docs/operating/integrations/#remote-en...). In most use operational use cases I've come across, we've only needed about a month of data, so keeping it in Prometheus was kosher. YMMV.
- zaroth 9y agoWhat if you want to keep time-series data for accounting purposes forever? Right now I use graphite but would love something that also handles replication/redundancy with a good query language & enough performance to also use it to fetch the underlying data used to render front-end graphs for users.
- alaties 9y agoThe answer to these kinds of questions is very, very dependant on requirements and available resources. And there are rarely easy, out of the box answers. The big questions to ask are: - what data is of interest - what are the requested ingest patterns (how is data getting into this system? How frequently? What rate of ingest is expected?) - what are the requested query patterns (who is doing queries? What do those queries look like specifically? Are people querying over unbounded time ranges or are people doing more focused queries? Do queries regularly involve aggregations or not?) - what is the requested SLA (i.e. how much partial down time is okay? How much full downtime is okay? What kind of query response time do you need to target?) - what resources are/will be available (money, man power, compute, storage, tech on hand) It's possible that a time series data store might not be the correct system choice once these questions are answered. It's not unusual to see data split or copied into multiple systems to answer all requirements.