10 ms·
Riemann – A network monitoring system
- pyritschard 11y agoI'm very happy to see riemann featured here. I've been using it in production since early 2012 and contributed to it (intensively, at one point) as well. It's been a breeze, rather worry free and its very good collectd support has enabled us to cover very interesting use cases at Exoscale.
- anth1y 11y agoI'm just getting started with riemann and I'm also learning clojure at the same time. How did you get buy in from your company to use riemann
- pyritschard 11y agoI co-founded it :-)
- pkd 11y agoClojure. I think I should stop thinking and really learn that language now. It seems very tempting as a first functional language.
- cpdean 11y agoIf you want to learn foundational FP topics but don't have the time to commit to learning a whole new ecosystem, I really recommend Fogus's Functional Javascript: http://www.amazon.com/Functional-JavaScript-Introducing-Programming-Underscore-js/dp/1449360726/?tag=fogus-20 http://www.amazon.com/Functional-JavaScript-Introducing-Prog... I know that JS isn't the same thing as Clojure but the ideas in this book work with really any language. After reading this book I'm a better Python programmer.
- yogthos 11y agoIt's an excellent beginner FP language. Clojure is fairly minimal and requires learning only a few concepts to become productive. At the same time it continues to see more use in the industry, so it's one of the easier FP languages to actually get a job with as well. I wrote a starter guide that might be helpful https://yogthos.github.io/ClojureDistilled.html https://yogthos.github.io/ClojureDistilled.html There's also a free book that's very good http://www.braveclojure.com/ http://www.braveclojure.com/
- pkd 11y agoThanks for compiling such a great resource. I have had a look at it, and will go through it this weekend. I have also been doing some 4clojure in my spare time. Although Brave and True looks like a great book, its not for me. I would love a book which blazes me through things rather than hand holding me through every concept. I have been looking at Living Clojure and will probably get that.
- mpdehaan2 11y agoI would look at is as ok language for lightweight scripting of Java, but some areas like how bad the tracebacks are or how well many of the clojure-specific libraries are fleshed out (or quality of docs), leaves me with major concerns about the language.
- lmm 11y agoFunctional means different things to different people. By all means learn Clojure, but also learn a functional language with a first-class type system (e.g. Haskell).
- pkd 11y agoAh, Haskell. I find it equal parts fascinating and scary. I definitely won't start at Haskell, although I would surely want to come to it at some point of time. Static typing looks nice.
- lmm 11y agoOCaml might be a friendlier option (though you'll want HKTs eventually, and the syntax is a bit ugly), or maybe Ceylon if you're willing to go a bit less mainstream. (I'm a Scala programmer myself, but I can't really recommend it unless you're already familiar with the JVM and its oddities, there are a lot of warts that have to be there for Java compatibility). But yeah, there's the Lisp tradition and the typed tradition and they're almost entirely separate, but through accidents of history we call them both "functional". So in the same way that you'd learn an OO language and a functional language, I'd say it's worth learning one of each.
- fnordsensei 11y agoDo it. At minimum, you'll have some very useful new perspectives.
- uberneo 11y agohttps://nodequery.com/ https://nodequery.com/ -- this is the lightweight version , not as extensive as Riemann but good for normal monitoring and notifications.
- tobz 11y agoAh, unfortunately, it looks like you need to sign up to even see what features it has. Do you know of an alternative feature list or have some things off the top of your head you could say about it?
- bru 11y agoNo, that's really different. - nodequery source (and ecosystem) is closed - "periodically store various system data" "minute" VS "low-latency" - email notifications only - no query language, but a JSON API → overhead Riemann looks more free (libre as well as liberty of usage, you can just monitor anything), more in-depth (does more with lower latency objectives) and better designed (DSL for rules, protobuff over tcp and udp instead of json over http over tcp).
- hendry 11y agoAnyone use http://prometheus.io/ http://prometheus.io/ ? wdyt
- MayMuncher 11y agoI don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues
- zimpenfish 11y agoThat's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector. (Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)
- bbrazil 11y agoYou need a place with all that information anyway, otherwise how do you alert on something being missing? If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-machines/ http://www.robustperception.io/you-look-good-have-you-lost-m...
- zimpenfish 11y agoIf a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.) Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.
- bbrazil 11y agoYou can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops. Alerting based on state changes is fragile, it's better to compare against what you expect to see.
- FlorianOver 11y agoUsed in production without a hassle for 2 years. For our setup and scenario it was a very fitting and good solution. Easy Setup and easy usage. Recommended.
- panchicore2 11y agocan you give the top 3 use cases and how it solves the problem?
- lafay 11y ago"Network monitoring system" makes me think of routers, switches, Mbps / Gbps, etc. This seems more like a "server monitoring system."
- acd 11y agoVery interesting project in monitoring! Thanks for creating it! Seems like new fresh ideas :).
- porker 11y agoAnyone able to give a quick overview of why they use Riemann vs SaaS offerings, and the distinguishing features (other than cost/control)? I like the look of the config structure, being Clojure.
- dorfsmay 11y agoNote that this isn't a better nagios... It's more like Apache Storm. Think of it as grep on steroid. I've heard of several thousand events per seconds on a single server. The other advantage is that it uses a well known programing language rather than a DSL.
- dozzie 11y agoIt's actually a disadvantage, since it's Clojure, and Riemann requires that its operator actually knows Clojure (knowledge of dozen languages, even when some of them are functional, is not enough).
- Anderkent 11y agoIs requiring someone to learn a programming language worse than requiring someone to learn a custom DSL? Seems a strange assertion.
- lmm 11y agoA full programming language is likely more effort to learn than a custom DSL. If you're in a position where you can reuse the full programming language then it's probably better, but if you're never going to use that programming language for anything other than this one product then the DSL is probably better.
- dozzie 11y agoClojure is a bigger language than small custom DSL.
- adam-_- 11y agoThe presentation video on the homepage is amazingly engaging. It's well worth a worth watch.
- moomin 11y agoReminder that the author of Reimann is available for consulting and is a scary-smart human being based around SF.
- romanhn 11y agoSpecifically, it was written by Kyle Kingsbury aka aphyr of the Jepsen fame.
- VonGuard 11y agoHe consults on Riemann? I was under the impression he wanted nothing to do with helping people use it, and is just a general consultant. Like, he's not trying to support Riemann, he built it because he needed it. Maybe he's changed his mind?
- deleted 11y ago[deleted]
- MayMuncher 11y agoIve been using Riemann at work for the past 6 months or so. There is a learning curve if you don't know clojure, that has been the biggest hurdle for me. But I love what it does and its fantastic along with the dashboard it comes with
- shizcakes 11y agoWe've been using Riemann since 2013 and love it! If you're coming from Nagios (or not), and you'd like something that will schedule Nagios event scripts (and others) and send them to Riemann, I have been using this in production since mid-2013: https://github.com/bmhatfield/riemann-sumd https://github.com/bmhatfield/riemann-sumd It allows you to tap into the huge ecosystem that is Nagios monitors, without requiring any other Nagios component at all. It just translates the output into a Riemann event.
- charlieflowers 11y agoWe also use Riemann in prod and love it. It is pretty much the perfect switchboard/aggregator for stat streams. It's only part of the stack, but it's great for routing some stats to this TSDB and other stats to that TSDB. It's also great for detecting anomalies and sending updates to wherever you want them to go. For us, this happens in 150 lines of clojure plus 150 lines of unit tests. I know that's fairly meaningless without knowing more about our system -- but the point is, it's very expressive so you get a lot done in a few lines of code. And therefore, don't worry too much about it being clojure.
- falcolas 11y agoEvaluated it, and ultimately rejected it for a couple of reasons: - You must pick up Clojure to understand and configure Riemann (we're not a Clojure shop, so this is a non-trivial requirement) - Config file isn't a config file, it's an executed bit of Clojure code - Riemann is not a replacement for an alerting mechanism, it's another signal for alerting mechanisms (though since it's Clojure and the configuration file is a Clojure script, you can absolutely hack it into becoming an alerting system) - Riemann is not a replacement for a trend graphing mechanism. - There are other solutions which can be piped together to get the 80% of the functionality we wanted from Riemann (Graphite + Skyline) in much less invested time Skyline link: https://github.com/etsy/skyline https://github.com/etsy/skyline
- lmm 11y agoIME configs often end up being turing-complete; if so, better to have them in a real programming language where you at least have tools available to manage the complexity.
- falcolas 11y agoThe problem with configs (and this is with my operations hat on), is that they are rarely as well secured (or reviewed) as regular code, so code based configuration files pose a significant privilege escalation threat on production servers. With my programmer's hat on, they're also harder to populate programmatically, so I have a hard time justifying their use.
- lmm 11y ago> The problem with configs (and this is with my operations hat on), is that they are rarely as well secured (or reviewed) as regular code, so code based configuration files pose a significant privilege escalation threat on production servers. Any complex config file runs that kind of risk though, whether it's in a well-known programming language or an ad-hoc DSL. My preferred approach is to include most of the config in the regular code (subject to the normal review/release process), with the only thing on the server being a one-line "which config to use" setting (e.g. dev/stag/prod). Of course that has its own problems. > With my programmer's hat on, they're also harder to populate programmatically, so I have a hard time justifying their use. Not at all true in the case of Clojure - it's just S-expressions, very easy to write, parse or modify programatically. I agree that a config structure should have good programmatic access, but to my mind that's an argument for using a language with a good metamodel rather than anything else.
- foolano 11y agoRiemann is great. We use it at work. I like it so much that I did an experiment to implement it in C++ https://github.com/juruen/cavalieri https://github.com/juruen/cavalieri My implementation sucks, but I had a lot of fun working on it and I got to learn how Riemann works better.
- jordigh 11y agoHm, why the name? I don't get it. Riemann wasn't knowing for being a good monitor or sentry.
- devonkim 11y agoPerhaps the idea is related to Riemann sums being an approximation of areas under curves where the curve is supposed to represent reality? Either that or that the Riemann hypothesis is an important problem that lots of people have tried to solve but have really failed.
- wuliwong 11y agoWhat is the connection between the name Riemann and the product?
- peterwwillis 11y agoA Java version of Ganglia?
- zgohr 11y agoMonitoring request - dynamic configuration. Using Nagios requires configuration file changes and service reload Would prefer dynamic, on the fly configuration. Even better, configuration that can be adjusted externally, possibly from a web API endpoint. Better still, configuration that IS external, where the monitoring service queries an external service every X minutes to determine what to monitor. Performant - In the realm of 6 oid monitoring of 50,000+ devices in 5 minute intervals
- deleted 11y ago[deleted]
- cliffmoon 11y agoI'm working on a startup that does exactly this. It's AWS only and in private beta right now: https://opsee.com/ https://opsee.com/
- deleted 11y ago[deleted]
- donaldguy 11y agoSince the theme of this thread seems to be non-clojure alternatives, I'll point in the direction of InfluxData (formerly InfluxDB)'s new-ish Kapacitor project. While I'm unconvinced their custom JavaScripty DSL (TICKscript) is actually preferable to Clojure or even can be read without careful, quite LISP-like indentation, it is pretty similar in basic functionality to Riemann and is definitely not-Clojure* see: https://influxdata.com/blog/announcing-kapacitor-an-open-source-streaming-and-batch-time-series-processor/ https://influxdata.com/blog/announcing-kapacitor-an-open-sou... https://docs.influxdata.com/kapacitor/v0.2/tick/ https://docs.influxdata.com/kapacitor/v0.2/tick/ https://github.com/influxdata/kapacitor https://github.com/influxdata/kapacitor *at worst, it's a mangled subset of clojure with extraneous dots and the parentheses in the wrong places :-)
- ultramancool 11y agoHow does a system like this differ from say, the ElasticSearch+Logstash system?
- bbrazil 11y agoLogs and metrics service different needs, I wrote recently on this: https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/
- elementai 11y agoEvaluated it and a couple of others. Chose Bosun, never looked back, it's probably the only system with flexible and concise DSL for evaluating alerts (somewhat similar to R in spirit). Riemann rocks, just not as monitoring system. Bosun link: http://bosun.org/ http://bosun.org/
- mVChr 11y agoWe evaluated both Bosun and Riemann for our use case and chose Riemann. Use case is over 10,000 servers with over 2 million metrics incoming every XX seconds. Riemann simply performs better, probably because it uses streams instead of a poller. The other part of our use case is to use it primarily for alerting, and it's fantastically responsive and robust for that, including simple integration with other systems (email, Slack, nagios, PagerDuty, etc). Bosun seems like a good tool for other uses however.
- la6470 11y agoAll new monitoring systems should be named YAMF-X (Yet Another Monitoring Framework) Jokes aside.. while the stream processing on events seem powerful there was something similar in graphite but probably not as advanced or easy to use. However the push approach brings its own limitations specially on existing setups.
- jamtur01 11y agoLove Riemann. So much that I'm writing a book about monitoring with Riemann as the core routing engine: http://artofmonitoring.com/ http://artofmonitoring.com/ There's a sample chapter available, which covers the initial Riemann implementation and a Clojure "getting started" guide which should help anyone - even if you're not interested in the rest of the book! :) http://artofmonitoring.com/TheArtOfMonitoring_sample.pdf http://artofmonitoring.com/TheArtOfMonitoring_sample.pdf
- avodonosov 11y agoWhat is the purpose of such a system? Can anyone describe at least one solid use case?
- bluedonuts 11y agoWe use Riemann to track all requests on Apis amongst other things. Have become rather fond of it. It happily handles 40k events p/s on rather modest hardware. The dashboard is not the prettiest but it is functional. Overall it has proved invaluable for spotting anomalies in our metrics.
- herbst 11y agoWhy Java and Ruby?