19 ms·

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

385 points by crecker 6y ago

abhishekjha 6y ago
This is exactly what I have been planning to do for monitoring my two Raspberry pis. I am still debating on the metrics collector though. My workplace uses monitord for AINT and telegraf for wavefront. I have no idea how well does collectord works.
overcast 6y ago
Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.
- viraptor 6y ago
  Does any of those matter for small-scale monitoring (say <= 10 hosts)? I've got influx sitting pretty much idle in that kind of environments and any data preprocessing / collation in grafana works just fine.
  - overcast 6y ago
    Sure, if you're really not doing anything with it, then any database would work fine for you.
  - dahfizz 6y ago
    Arguably system requirements matter more for small scale deployments. I shouldn't need a server with multiple SSD volumes and 8GB+ ram just to monitor a couple raspberry pi's. EDIT: okay, I get it. You don't need 8GB ram. I was just going by the hardware requirements in their docs: https://docs.influxdata.com/influxdb/v1.8/introduction/install/ https://docs.influxdata.com/influxdb/v1.8/introduction/insta...
    - cbzbc 6y ago
      You don't need 8GB ram to run a small scale deployment of influxdb.
      - dspillett 6y ago
        I'm sure this is true, but the documentation does explicitly say "Each machine should have a minimum of 8GB RAM" with absolutely no caveats about scale, so I think people can be forgiven for taking a quick look and moving on because they don't want to provision that.
        mhall119 6y ago
        That's a valid point, I've brought it up internally to improve the docs to explain what type of load this requirement is expecting. If you'd like you can create an issue for this in Github (https://github.com/influxdata/docs.influxdata.com/issues/new https://github.com/influxdata/docs.influxdata.com/issues/new) to be involved in the discussion and change.
        mhall119 6y ago
        While we're working on this page, also check out https://docs.influxdata.com/influxdb/v1.8/guides/hardware_sizing/#influxdb-oss-guidelines https://docs.influxdata.com/influxdb/v1.8/guides/hardware_si... which gives a better breakdown of resources requirements based on expected load
    - viraptor 6y ago
      ... and it doesn't need that. I'm running telegraf+influx+grafana along with other services on a small rpi without issues.
    - alyandon 6y ago
      Running TICK stack here collecting metrics from around a dozen different hosts (more if you count network devices like switches) to a central server with a default retention policy on spinning rust. InfluxDB process is currently sitting at 600MB RSS.
- candiddevmike 6y ago
  Influx is really shitting the bed with how they're handling the InfluxDB 2.0 release. The docs are a mess, and the migration tool seems like the result of a weekend hackathon. They're leaving a lot of customers with long term metrics in a tough spot. If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and bucking the ecosystem by trying to control the entire stack.
  - rbetts 6y ago
    (I'm the VPE at Influxdata). I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side. However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS. We've spoken about this publicly in other forums. You can google "influxdays London talks" to hear Paul Dix (CTO/Founder) talk more about our OSS plans.
    - edoceo 6y ago
      Classic "two masters" problem.
      - santafen 6y ago
        This was from the CTO last week: "This work won't be landing anywhere until sometime next year and it'll be landing in our Cloud 2 offering first." So the OSS is definitely a second-class citizen. And now that they've dropped all DevRel activity, don't expect much attention for OSS Developers and users.
        mhall119 6y ago
        Quite the contrary, we're finishing up packaging of the OSS version now to make it as easy as possible for you to get it and we're assembled a new team focused on getting OSS 2.0 released. We continue to support all users, including OSS users, in our public Slack and Discourse, as well as Github. We have not "dropped all DevRel" activity.
- mbell 6y ago
  We had major issues with scaling InfluxDB. We use clickhouse (graphite table engine) now and it is more than order of magnitude more resource efficient.
- goliatone 6y ago
  Im on the same boat, I tried very hard. Went through a few version upgrades with data incompatibility and other issues. Used it for personal projects as well as for different projects at work. I truly regret ever recommending it to different teams at work. I ended up moving away from it to TimescaleDB and I’m pretty happy so far.
- julioo 6y ago
  What about storage? We are running influxdb and we are looking for alternative. But a point where Influx is good is storage.
  - tgtweak 6y ago
    You mean storage efficiency? Seems that unless the aggregation is happening before being stored, it would be unfair to compare influxdb with other databases that are tasked with storing per-record granularity.
  - k-rus 6y ago
    TimescaleDB provides such community features as compression, which allows to save space a lot, and continuous aggregates, which gives performance and save space if used together with retentions.
  - mekster 6y ago
    Tried VictoriaMetrics? The author compares with other storage engines comprehensively from time to time. https://medium.com/@valyala https://medium.com/@valyala
- RedShift1 6y ago
  InfluxDB is pretty good if you don't need to do any advanced querying like grouping by month or formulas. Its strong points are low diskspace footprint and very fast queries even over long periods of time.
  - mhall119 6y ago
    Have you looked at Flux? You can do some really incredible things with it, both inside of queries and also in tasks/alerts. Check out https://www.influxdata.com/blog/anomaly-detection-with-median-absolute-deviation/ https://www.influxdata.com/blog/anomaly-detection-with-media... for an example
    - RedShift1 6y ago
      Yes I have looked at it and it's unusable for my application because it is orders of magnitude slower than InfluxQL. Fairly logical if you think about how Flux works, it's very hard to optimize a query engine with a language where every step is supposed to be discrete.
      - mhall119 6y ago
        There are actually multiple optimizations to Flux already underway. The language runtime can "push down" some of these operations to the storage layer where they can be performed more efficiently.
- svd4anything 6y ago
  Is that really true? Do you have any reference for that, I’m curious how different it is. FWIW I use InfluxDB and haven’t really thought it was a huge resource hog.
  - valyala 6y ago
    InfluxDB may require big amounts of RAM when working with high cardinality data. It works OK with up to a million of time series. Then it starts eating RAM as crazy [1]. How many time series does your InfluxDB setup contain? [1] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl... .
- majkinetor 6y ago
  I use InfluxDb on number of big gov production services, and never had a single issue with it. I use it for all metrics (applicative and infrastructure). Its easy to deploy and its x-platform. Not sure what are all the comments here about - maybe for really huge loads which I don't have experience at, I usually use separate influxdb per service.
viraptor 6y ago
Nothing against choosing this set of apps really, but I'm curious why collectd and not telegraf which does the same kind of metrics probes and is a part of the TICK stack.
- bbrks 6y ago
  As somebody new to both, are you able to elaborate on what telegraf/TICK provides that the collection of programs in the post does not? I see lots of posts complaining about the stack in the post, but all of the alternatives posed don't really explain why.
  - viraptor 6y ago
    Telegraf and collectd do roughly the same thing. They run some plugins to get data and push the results to a given metrics sink. I asked because TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system.
    - jimktrains2 6y ago
      > TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system. Amusing that I've never heard of any of this but have heard and used collectd. It's obviously no where near as common as a lamp stack or anywhere near common at all, so asking why it over something else is answered by "someone made it up so it's better".
      - viraptor 6y ago
        Collectd is only one part here. Did you need a solution for inline processing/aggregation and alerting? If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far. I don't get the comparison to LAMP popularity. Insects are more common than cars too. They're different things ¯\_(ツ)_/¯
kristianpaul 6y ago
I guess collectl + graphite is not used that much these days
Kednicma 6y ago
I'm curious why they chose CollectD and not Prometheus or another pull-based ("scraping") solution. Pulling is more compositional than pushing and requires fewer machines to be touched when monitoring configuration is changed.
- viraptor 6y ago
  Probably due to the size of deployment. I'd go to Prometheus in a larger/dynamic environment for flexibility, but I'm happy using telegraf on a home network where I configure things by hand when I'm logged into each pet machine.
- mhall119 6y ago
  The author responded in another thread, it was just due to being familiar with CollectD already
wiradikusuma 6y ago
There's also https://github.com/timberio/vector https://github.com/timberio/vector
- victor106 6y ago
  +1 for vector. We moved from Logstash to Vector and we couldn't be happier. Logstash is awesome but its a memory hog. With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana
  - sevagh 6y ago
    Have you looked into Grafana Loki for logs? If I had to redo one part of our stack, that's what I'd choose.
  - lma21 6y ago
    Does Vector work with TimescaleDB? it looks quite interesting
pachico 6y ago
I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse (yes, not a timeseries dB). From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. It might just be me.
- mhall119 6y ago
  > From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself. Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)
  - pachico 6y ago
    Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.
    - mhall119 6y ago
      The OSS version is going to fill a vital role that SaaS just can't do, and we view it as a critical component in what InfluxDB provides. I don't think we're ever going to get away from needing on-prem and on-device deployments when dealing with time-series data, especially for emerging IoT/Edge use cases. No, the company recognizes that the success of the SaaS is inevitably tied to the success of the OSS product. The only reason we were able to give the SaaS more focus recently is because we already had a working OSS product in the 1.x line that was meeting the needs of existing customers that we were and are continuing to invest in.
second--shift 6y ago
I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The monitor server is just a DO droplet and our collectors require zero overhead. The specs list and setup costs in time and complexity are steep with a Grafana stack - is there any value besides just the visual? I know they have the ability to do all manner of custom plugins, dashboards, etc, but if you just care about the good stuff (uptime+performance), what does Grafana give you that rsync'ing sar data can't? PS: we have a graphical parser of the data written using python and matplotlib. very lightweight, and we also get pretty graphs to print and give to upstairs.
- Nextgrid 6y ago
  The biggest advantage is the near-real-time aspect of this. During an outage, having live metrics is essential. Does your custom system allow you to see live metrics as they happen, or do you need to re-run your aggregation scripts every 5 minutes?
- dewey 6y ago
  At work and in my personal projects I use a Prometheus + Grafana setup and it's very easy to set up. Not sure what you mean with the complex and setup costs for a Grafana stack? Alerting with the Prometheus AlertManager is also pretty straight forward and I'm looking at dashboards every day to see if everything is running smoothly or tracking down what's not working well if there are any issues. Grafana dashboards are always the second thing I look at after an alert fires somewhere and it has been invaluable.
- viraptor 6y ago
  What sar+rsync doesn't provide: - app-specific metrics - quick and easy way to build a number of graphs searching for correlation (how to slice the data to get results that explain issues) - log/metrics correlation - unified way to build alerts - ad-hoc changes - while you're in the middle of an incident and want to get information that's just slightly different from existing, or filter out some values, or overlay a trend - how long would it take in your custom solution vs grafana? And finally - grafana exists. Why would I write a custom graph generator from a custom data store if I can setup a collector + influx + grafana in a fraction of that time and get back more?
ddevault 6y ago
I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-services/ https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...
- robotmay 6y ago
  I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using it with Grafana and me attempting to misuse it. By contrast I think the concepts of something like TimescaleDB are easier to understand when it comes to scaling and optimising that service. In my previous job I had a very clear use-case for not using Prometheus and did for a while use InfluxDB (it involved devices sending data from behind firewalls across many sites). I found it pretty expensive to scale and it fell over when it ran out of storage, which feels like something that should have been handled automatically considering it was a PaaS offering.
  - ddevault 6y ago
    One point of note for SourceHut's Prometheus use is that we generally don't make dashboards. I don't really like Grafana. I will sometimes use gnuplot with styx to plot graphs on an as-needed basis: https://github.com/go-pluto/styx https://github.com/go-pluto/styx This is how I made the plots in that blog post.
    - robotmay 6y ago
      I have a similar relationship to Grafana as I do for Prometheus; love it for my home and I've got some very useful graphs for my home network, but it's almost unusable for work due to its speed degradation the moment you start adding more graphs. Again it's probably due to my lack of knowledge around some of the Prometheus functions for reducing the amount of data returned, but it would be nice if it could handle some of that automatically rather than just grinding to a halt.
roland35 6y ago
InfluxDB and Grafana worked great for us when I created a live monitoring system for a fleet of prototype test robots. It was simple to set up new data streams. We started with Graphite but switched to InfluxDB for it's flexibility (Grafana works with both!) I would add to the guide that you need to be careful about formatting the lines into InfluxDB because where you out the space and commas determines what is indexed or not! Also data types should be specific (ie make sure you are setting integer vs float correctly).
x87678r 6y ago
I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.
- halfmatthalfcat 6y ago
  The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.
  - x87678r 6y ago
    Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.
    - julioo 6y ago
      In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
      - julioo 6y ago
        Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.
  - dewey 6y ago
    I think that's what https://thanos.io/ https://thanos.io/ is for.
    - 6y ago
majkinetor 6y ago
You can quickly do this on Windows: cinst influxdb1 /Service cinst grafana start $Env:ChocolateyInstall\lib\grafana\tools\grafana-*\bin\grafana-server.exe git clone https://github.com/majkinetor/psinflux import-module ./psinflux 1..10 | % { $x = 10*$_ + (Get-Random 10); Send-Data "test1 value=$x"; sleep 1 }
PanosJee 6y ago
Too many parts Netdata will make it a breeze
- linsomniac 6y ago
  Netdata is pretty slick.
carlosdp 6y ago
Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
- Hates_ 6y ago
  I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.
  - dharmab 6y ago
    Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.
    - edoceo 6y ago
      My company's Prometheus setup was super easy, one $10/mo box. About 1 week of fiddling all the exporters and configs but now it just runs and has for months. You can be small with Prometheus and grow into needing an FTE for it - w/o having the migration hurdle of moving out-source to in-source
    - lima 6y ago
      Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).
      - gen220 6y ago
        I think the parent's complaint is a function of how your engineering org "uses" prometheus. If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus). IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".
burtonator 6y ago
I've done this in my company with about 100 servers using KairosDB + Grafana and CollectD... https://kairosdb.github.io/ https://kairosdb.github.io/ If I were to do it again I'd probably use Grafana and Prometheus
deleted 6y ago
[deleted]
avthar 6y ago
Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-time-series-database-postgresql-free-tsdb/ https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today TimescaleDB outperforms InfluxDB across almost all dimensions (credit goes to our database team!), especially for high-cardinality workloads: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/ https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-... TimescaleDB also works with Grafana, Prometheus, Telegraf, Kafka, Apache Spark, Tableau, Django, Rails, anything that speaks SQL...
- jimaek 6y ago
  While influx is pretty bad overall it's super simple to deploy and configure unlike timescale. It's the main reason we decided to use influx in our small team with simple enough timeseries needs
  - avthar 6y ago
    Curious what you found difficult to deploy/ configure? Is this in a self-managed context?
    - samstave 6y ago
      I found this to be a funny, subtle insult. :-) "Why is it difficult, because you're self managed?"
      - avthar 6y ago
        No intention to insult - just trying to get more context :)
        6y ago
zytek 6y ago
VictoriaMetrics eats other TSDBs for breakfast. PromQL support (with extensions) and clustered / HA mode. Great storage efficiency. Plays well for monitoring multiple k8s clusters, works great with Grafana, pretty easily deployed on k8s. No affiliation, just a happy user.
- mekster 6y ago
  I just don't get why VictoriaMetrics doesn't get more visibility. Maybe they need a PR person.
  - valyala 6y ago
    Absolutely! We are working on this.
Papric0re 6y ago
[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention of PCAPs) and save some security logs (I'm not intending on alerting based on logs, just for retention). But being able to have a network overview, insight into current connections (including history) is a very useful thing. Can anybody recommend something, that's maybe a bit lighter than an entire Elastic-Stack?
- floren 6y ago
  I think Gravwell (https://gravwell.io https://gravwell.io) might be what you're looking for--but I work for Gravwell so I may be biased! If I can be forgiven a short sales pitch, we've built a format-agnostic storage & querying system that easily handles raw packets, Netflow records (v5, v9, and IPFIX), collectd data, Windows event logs, and more. You can see some screenshots at https://www.gravwell.io/technology https://www.gravwell.io/technology We have a free tier which allows 2GB of data ingest per day (paid licenses are unlimited) which should be more than enough for capturing logs and flows. The resources needed to run Gravwell basically scale with how much data you put into it, but it's a lot quicker to install and set up than something like Elastic, in our opinion (https://www.gravwell.io/blog/gravwell-installed-in-2-minutes https://www.gravwell.io/blog/gravwell-installed-in-2-minutes) Edit: it's currently a bit roll-your-own, but we're really close to releasing Gravwell 4.0 which enables pre-packaged "kits" containing dashboards, queries, etc. for a variety of data types (Netflow, CoreDNS logs, and so on)
  - Papric0re 6y ago
    I don't mind you giving a small sales pitch...and maybe your product is indeed what I'm searching for. But your pricing model is instantly putting me off. Same as with Splunk, you end up with not being able to predict your cost and paying way too much. Tell me when you fix that and I might be interested ;) Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)
tgtweak 6y ago
Looking for a similar post with APM using opentracing !
rattray 6y ago
I've only ever used third party monitoring tools, but hope to set up a startup again soon and want to do OSS if I can. Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?
- k-rus 6y ago
  You can use Prometheus on top of TimescaleDB. Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way. Sorry for the promo :) check for details in https://github.com/timescale/timescale-prometheus https://github.com/timescale/timescale-prometheus
  - rattray 6y ago
    Thanks!
- cevian 6y ago
  (Timescale-Prometheus team lead here) I'd suggest using Timescale-Prometheus to connect Prometheus with TimescaleDB. Repo: https://github.com/timescale/timescale-prometheus https://github.com/timescale/timescale-prometheus Design Doc: https://tsdb.co/prom-design-doc https://tsdb.co/prom-design-doc
- valyala 6y ago
  Prometheus is a monitoring system (it collects metrics and evaluates alerts on them), while TimescaleDB is a database. While it is possible to use TimescaleDB as a remote storage for Prometheus, I'd recommend taking a look at other remote storage integrations as well [1]. Some of them natively support PromQL (like VictoriaMetrics, m3db, Cortex), others may be easier to setup and operate. For instance, VictoriaMetrics works out of the box without complex configuration. Disclaimer: I work for VictoriaMetrics :). [1] https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage https://prometheus.io/docs/operating/integrations/#remote-en...
aprdm 6y ago
As mentioned by others I have found a Prometheus + Grafana setup to be pretty straight forward to install and maintain. It also aligns with the cloud native foundation which means that this stack easily integrates with a bunch of OSS stack used by cloud native companies.
hnarn 6y ago
Some people are probably going to throw some shade on me for saying this since it's so out of fashion but in my mind, when it comes to some types of basic monitoring (SNMP monitoring of switches/linux servers, disk space usage, backups running and handling them when they don't) then Nagios does get the job done. It's definitely olives and not candy[1] but it's stable, modular, relatively easy to configure (when you get it) and it just keeps chugging away. If anyone has any Nagios questions, I'd be happy to answer them. I'm a Nagios masochist. (Also, I can recommend Nagflux[2] to export performance data, metrics, to InfluxDB because noone should have to touch RRD files) [1]: https://lambdaisland.com/blog/2019-08-07-advice-to-younger-self https://lambdaisland.com/blog/2019-08-07-advice-to-younger-s... [2]: https://github.com/Griesbacher/nagflux https://github.com/Griesbacher/nagflux
- brightball 6y ago
  Ever looked at Zabbix?
  - hnarn 6y ago
    I haven't, unfortunately, but it looks promising from just looking into it briefly. Open source monitoring is always an area that needs more competition.
    - Twirrim 6y ago
      Zabbix has been around for quite a long time. Easily 15 years now. I haven't looked at it since around 2013, but at the time it was placing quite some pressure on a mysql db backend. It looks like they've expanded out to support more than MySQL as the back-end these days.
      - k-rus 6y ago
        I noticed that Zabbix supports PostgreSQL and TimescaleDB as back-ends and just checked the list, which contains also Oracle and SQLite (DB2 support is experimental).
      - 6y ago
9thbyte 6y ago
This seems like a lot of work when you could use Zabbix. ¯\_(ツ)_/¯
9thbyte 6y ago
Seems like a lot of work when Zabbix is available. Read the latest version even supports TimescaleDB.
linsomniac 6y ago
I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk). Influxdb has been a real workhorse. We suffered through some of their early issues, but since then it's been extremely solid. It just runs, is space efficient, and very robust. We almost went with Prometheus instead of Influx (as I said, early growing pains), but I had just struggled through managing a central inventory and hated it, so I really wanted a push rather than pull architecture. But from my limited playing with it, Prometheus seemed solid.
- drez 6y ago
  I just wanted to echo this sentiment. Influx had its fair share of issues a few years ago (v0.8 migration, changing storage engines, tag cardinality issues), but the latest v1.x releases have been solid. I have been using the TIK stack (I use Grafana instead of Chronograf) for monitoring several dozen production-facing machines for 2 years now without a single issue, which I would very much count as a win. I just hope they learned their lesson for the v2.0 release...
- rehevkor5 6y ago
  It's so much easier to write incorrect/misleading queries in influxql than in promql. And you can't perform operations between two different series names in influxdb, last I looked. That makes it impossible to do things like ratios or percentages unless you have control over your metrics, and structure them the way influx likes. Also, no support for calculating percentiles from Prometheus histogram buckets.
  - mhall119 6y ago
    You can do these things (and much, much more) with the new Flux language
- valyala 6y ago
  > I really wanted a push rather than pull architecture Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with big number of time series (aka high cardinality) [2]. [1] https://victoriametrics.github.io/#how-to-import-time-series-data https://victoriametrics.github.io/#how-to-import-time-series... [2] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl...
9thbyte 6y ago
This is like creating Zabbix in pieces and missing the best parts. Triggers, Alerts, and Actions.
aequitas 6y ago
For those who still remember Graphite, the team over at Grafana labs have started maintaining Graphite-web and Carbon since 2017 and it is still in active development getting improvements and feature updates. It might not scale as well as any of the other solutions, but for medium size or homelab setups it's still a nice solution if you don't like PromQL or InfluxQL. https://grafana.com/oss/graphite/ https://grafana.com/oss/graphite/
- lma21 6y ago
  for simple and fast monitoring solutions, i always opted for collected + graphite + grafana. In a containerized environment, it's so easy to deploy (0 configuration by default) and monitor a set of 50-100 nodes. Beyond that, disk tuning and downsampling (pre-aggregation) rules become important.
- rehevkor5 6y ago
  If you're still on graphite and need scaling, metrictank is something to consider.
Aaronstotle 6y ago
I recognized the name almost immediately, cool to see a blog post from a member of the monero community.
- crecker 6y ago
  haha thanks!
mickeyben 6y ago
We're currently using InfluxDB but maintenance is something we'd like to stop doing on our monitoring stack. Datadog is too expensive because of the number of hosts we have. So we're thinking of eventually going to a hosted InfluxDB setup. But we also want to revisit other hosted solutions. Does someone have some experience with using Cloudwatch + Grafana? I've used Cloudwatch many years ago and it was clearly subpar to something like Influx. Is is better nowadays?
metalliqaz 6y ago
I use Telegraf, InfluxDB, and Grafana for my home network, in addition to monitoring my weather station data. It works incredibly well and is so simple compared to the "old" suite of tools such as RRD and Cacti.
sahoo 6y ago
How does this compare to grafana, Prometheus and node monitor
jakobdabo 6y ago
I still can't find any alternative to the old, RRD-based Munin. It is so simple. You want to add a new server to monitor? Just install there the node part, enable any required additional plugins (just by creating a couple of soft-links), add one-line configuration to the main server with the new node's IP address, and you are done. Also, the aesthetics of the UX, you see all the graphs in one single page[1], no additional clicks are required - a quick glance with a slow scroll and you can see if there were any unusual things during the last day/week. [1] - publicly available example, found by googling - https://ansible.fr/munin/agate/agate/index.html https://ansible.fr/munin/agate/agate/index.html
- mhall119 6y ago
  That's more steps than using InfluxDB and Telegraf. Check out: https://github.com/influxdata/community-templates/tree/master/linux_system https://github.com/influxdata/community-templates/tree/maste...
ecoqba11 6y ago
We switched from InfluxDB to TimescaleDB for our IoT solutions. InfluxDB is very difficult to work with large datasets and enterprise/region compliance. We ingest around 100MB data per day and growing.
- mhall119 6y ago
  That doesn't actually sound like a large dataset. Can you describe what kind of problems you faced with InfluxDB?
- valyala 6y ago
  100MB data per day looks like a tiny number. Our users ingest 100GB data per day at the ingestion rate of 1M data points per second on a single-node setup [1]. [1] https://victoriametrics.github.io/CaseStudies.html https://victoriametrics.github.io/CaseStudies.html
DevKoala 6y ago
I’d replace InfluxDB with Prometheus. InfluxDB has been a PITA for me.
mavam 6y ago
Does anyone have experience with an InfluxDB-based netdata setup? When we evaluated multiple options, we found that netdata does a lot of the heavy lifting. What are the advantages of a CollectD-based setup?
- zacksh 6y ago
  Well, that's the beauty of Netdata, no need to configure collectors. As for InfluxDB integration, it's fairly easy to export from Netdata using what InfluxDB calls the line protocol format (the "plaintext interface"). https://learn.netdata.cloud/docs/agent/exporting https://learn.netdata.cloud/docs/agent/exporting

385 points by crecker 6y ago

This is exactly what I have been planning to do for monitoring my two Raspberry pis. I am still debating on the metrics collector though. My workplace uses monitord for AINT and telegraf for wavefront. I have no idea how well does collectord works.

overcast 6y ago

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

viraptor 6y ago
Does any of those matter for small-scale monitoring (say <= 10 hosts)? I've got influx sitting pretty much idle in that kind of environments and any data preprocessing / collation in grafana works just fine.
- overcast 6y ago
  Sure, if you're really not doing anything with it, then any database would work fine for you.
- dahfizz 6y ago
  Arguably system requirements matter more for small scale deployments. I shouldn't need a server with multiple SSD volumes and 8GB+ ram just to monitor a couple raspberry pi's. EDIT: okay, I get it. You don't need 8GB ram. I was just going by the hardware requirements in their docs: https://docs.influxdata.com/influxdb/v1.8/introduction/install/ https://docs.influxdata.com/influxdb/v1.8/introduction/insta...
  - cbzbc 6y ago
    You don't need 8GB ram to run a small scale deployment of influxdb.
    - dspillett 6y ago
      I'm sure this is true, but the documentation does explicitly say "Each machine should have a minimum of 8GB RAM" with absolutely no caveats about scale, so I think people can be forgiven for taking a quick look and moving on because they don't want to provision that.
      - mhall119 6y ago
        That's a valid point, I've brought it up internally to improve the docs to explain what type of load this requirement is expecting. If you'd like you can create an issue for this in Github (https://github.com/influxdata/docs.influxdata.com/issues/new https://github.com/influxdata/docs.influxdata.com/issues/new) to be involved in the discussion and change.
        mhall119 6y ago
        While we're working on this page, also check out https://docs.influxdata.com/influxdb/v1.8/guides/hardware_sizing/#influxdb-oss-guidelines https://docs.influxdata.com/influxdb/v1.8/guides/hardware_si... which gives a better breakdown of resources requirements based on expected load
  - viraptor 6y ago
    ... and it doesn't need that. I'm running telegraf+influx+grafana along with other services on a small rpi without issues.
  - alyandon 6y ago
    Running TICK stack here collecting metrics from around a dozen different hosts (more if you count network devices like switches) to a central server with a default retention policy on spinning rust. InfluxDB process is currently sitting at 600MB RSS.
candiddevmike 6y ago
Influx is really shitting the bed with how they're handling the InfluxDB 2.0 release. The docs are a mess, and the migration tool seems like the result of a weekend hackathon. They're leaving a lot of customers with long term metrics in a tough spot. If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and bucking the ecosystem by trying to control the entire stack.
- rbetts 6y ago
  (I'm the VPE at Influxdata). I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side. However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS. We've spoken about this publicly in other forums. You can google "influxdays London talks" to hear Paul Dix (CTO/Founder) talk more about our OSS plans.
  - edoceo 6y ago
    Classic "two masters" problem.
    - santafen 6y ago
      This was from the CTO last week: "This work won't be landing anywhere until sometime next year and it'll be landing in our Cloud 2 offering first." So the OSS is definitely a second-class citizen. And now that they've dropped all DevRel activity, don't expect much attention for OSS Developers and users.
      - mhall119 6y ago
        Quite the contrary, we're finishing up packaging of the OSS version now to make it as easy as possible for you to get it and we're assembled a new team focused on getting OSS 2.0 released. We continue to support all users, including OSS users, in our public Slack and Discourse, as well as Github. We have not "dropped all DevRel" activity.
mbell 6y ago
We had major issues with scaling InfluxDB. We use clickhouse (graphite table engine) now and it is more than order of magnitude more resource efficient.
goliatone 6y ago
Im on the same boat, I tried very hard. Went through a few version upgrades with data incompatibility and other issues. Used it for personal projects as well as for different projects at work. I truly regret ever recommending it to different teams at work. I ended up moving away from it to TimescaleDB and I’m pretty happy so far.
julioo 6y ago
What about storage? We are running influxdb and we are looking for alternative. But a point where Influx is good is storage.
- tgtweak 6y ago
  You mean storage efficiency? Seems that unless the aggregation is happening before being stored, it would be unfair to compare influxdb with other databases that are tasked with storing per-record granularity.
- k-rus 6y ago
  TimescaleDB provides such community features as compression, which allows to save space a lot, and continuous aggregates, which gives performance and save space if used together with retentions.
- mekster 6y ago
  Tried VictoriaMetrics? The author compares with other storage engines comprehensively from time to time. https://medium.com/@valyala https://medium.com/@valyala
RedShift1 6y ago
InfluxDB is pretty good if you don't need to do any advanced querying like grouping by month or formulas. Its strong points are low diskspace footprint and very fast queries even over long periods of time.
- mhall119 6y ago
  Have you looked at Flux? You can do some really incredible things with it, both inside of queries and also in tasks/alerts. Check out https://www.influxdata.com/blog/anomaly-detection-with-median-absolute-deviation/ https://www.influxdata.com/blog/anomaly-detection-with-media... for an example
  - RedShift1 6y ago
    Yes I have looked at it and it's unusable for my application because it is orders of magnitude slower than InfluxQL. Fairly logical if you think about how Flux works, it's very hard to optimize a query engine with a language where every step is supposed to be discrete.
    - mhall119 6y ago
      There are actually multiple optimizations to Flux already underway. The language runtime can "push down" some of these operations to the storage layer where they can be performed more efficiently.
svd4anything 6y ago
Is that really true? Do you have any reference for that, I’m curious how different it is. FWIW I use InfluxDB and haven’t really thought it was a huge resource hog.
- valyala 6y ago
  InfluxDB may require big amounts of RAM when working with high cardinality data. It works OK with up to a million of time series. Then it starts eating RAM as crazy [1]. How many time series does your InfluxDB setup contain? [1] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl... .
majkinetor 6y ago
I use InfluxDb on number of big gov production services, and never had a single issue with it. I use it for all metrics (applicative and infrastructure). Its easy to deploy and its x-platform. Not sure what are all the comments here about - maybe for really huge loads which I don't have experience at, I usually use separate influxdb per service.

viraptor 6y ago

Nothing against choosing this set of apps really, but I'm curious why collectd and not telegraf which does the same kind of metrics probes and is a part of the TICK stack.

bbrks 6y ago
As somebody new to both, are you able to elaborate on what telegraf/TICK provides that the collection of programs in the post does not? I see lots of posts complaining about the stack in the post, but all of the alternatives posed don't really explain why.
- viraptor 6y ago
  Telegraf and collectd do roughly the same thing. They run some plugins to get data and push the results to a given metrics sink. I asked because TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system.
  - jimktrains2 6y ago
    > TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system. Amusing that I've never heard of any of this but have heard and used collectd. It's obviously no where near as common as a lamp stack or anywhere near common at all, so asking why it over something else is answered by "someone made it up so it's better".
    - viraptor 6y ago
      Collectd is only one part here. Did you need a solution for inline processing/aggregation and alerting? If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far. I don't get the comparison to LAMP popularity. Insects are more common than cars too. They're different things ¯\_(ツ)_/¯

kristianpaul 6y ago

I guess collectl + graphite is not used that much these days

Kednicma 6y ago

I'm curious why they chose CollectD and not Prometheus or another pull-based ("scraping") solution. Pulling is more compositional than pushing and requires fewer machines to be touched when monitoring configuration is changed.

viraptor 6y ago
Probably due to the size of deployment. I'd go to Prometheus in a larger/dynamic environment for flexibility, but I'm happy using telegraf on a home network where I configure things by hand when I'm logged into each pet machine.
mhall119 6y ago
The author responded in another thread, it was just due to being familiar with CollectD already

wiradikusuma 6y ago

There's also https://github.com/timberio/vector https://github.com/timberio/vector

victor106 6y ago
+1 for vector. We moved from Logstash to Vector and we couldn't be happier. Logstash is awesome but its a memory hog. With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana
- sevagh 6y ago
  Have you looked into Grafana Loki for logs? If I had to redo one part of our stack, that's what I'd choose.
- lma21 6y ago
  Does Vector work with TimescaleDB? it looks quite interesting

pachico 6y ago

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome. I'm now having better results and experience storing data in ClickHouse (yes, not a timeseries dB). From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. It might just be me.

mhall119 6y ago
> From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising. Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself. Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)
- pachico 6y ago
  Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.
  - mhall119 6y ago
    The OSS version is going to fill a vital role that SaaS just can't do, and we view it as a critical component in what InfluxDB provides. I don't think we're ever going to get away from needing on-prem and on-device deployments when dealing with time-series data, especially for emerging IoT/Edge use cases. No, the company recognizes that the success of the SaaS is inevitably tied to the success of the OSS product. The only reason we were able to give the SaaS more focus recently is because we already had a working OSS product in the 1.x line that was meeting the needs of existing customers that we were and are continuing to invest in.

second--shift 6y ago

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this? I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The monitor server is just a DO droplet and our collectors require zero overhead. The specs list and setup costs in time and complexity are steep with a Grafana stack - is there any value besides just the visual? I know they have the ability to do all manner of custom plugins, dashboards, etc, but if you just care about the good stuff (uptime+performance), what does Grafana give you that rsync'ing sar data can't? PS: we have a graphical parser of the data written using python and matplotlib. very lightweight, and we also get pretty graphs to print and give to upstairs.

Nextgrid 6y ago
The biggest advantage is the near-real-time aspect of this. During an outage, having live metrics is essential. Does your custom system allow you to see live metrics as they happen, or do you need to re-run your aggregation scripts every 5 minutes?
dewey 6y ago
At work and in my personal projects I use a Prometheus + Grafana setup and it's very easy to set up. Not sure what you mean with the complex and setup costs for a Grafana stack? Alerting with the Prometheus AlertManager is also pretty straight forward and I'm looking at dashboards every day to see if everything is running smoothly or tracking down what's not working well if there are any issues. Grafana dashboards are always the second thing I look at after an alert fires somewhere and it has been invaluable.
viraptor 6y ago
What sar+rsync doesn't provide: - app-specific metrics - quick and easy way to build a number of graphs searching for correlation (how to slice the data to get results that explain issues) - log/metrics correlation - unified way to build alerts - ad-hoc changes - while you're in the middle of an incident and want to get information that's just slightly different from existing, or filter out some values, or overlay a trend - how long would it take in your custom solution vs grafana? And finally - grafana exists. Why would I write a custom graph generator from a custom data store if I can setup a collector + influx + grafana in a fraction of that time and get back more?

ddevault 6y ago

I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-services/ https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

robotmay 6y ago
I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using it with Grafana and me attempting to misuse it. By contrast I think the concepts of something like TimescaleDB are easier to understand when it comes to scaling and optimising that service. In my previous job I had a very clear use-case for not using Prometheus and did for a while use InfluxDB (it involved devices sending data from behind firewalls across many sites). I found it pretty expensive to scale and it fell over when it ran out of storage, which feels like something that should have been handled automatically considering it was a PaaS offering.
- ddevault 6y ago
  One point of note for SourceHut's Prometheus use is that we generally don't make dashboards. I don't really like Grafana. I will sometimes use gnuplot with styx to plot graphs on an as-needed basis: https://github.com/go-pluto/styx https://github.com/go-pluto/styx This is how I made the plots in that blog post.
  - robotmay 6y ago
    I have a similar relationship to Grafana as I do for Prometheus; love it for my home and I've got some very useful graphs for my home network, but it's almost unusable for work due to its speed degradation the moment you start adding more graphs. Again it's probably due to my lack of knowledge around some of the Prometheus functions for reducing the amount of data returned, but it would be nice if it could handle some of that automatically rather than just grinding to a halt.

roland35 6y ago

InfluxDB and Grafana worked great for us when I created a live monitoring system for a fleet of prototype test robots. It was simple to set up new data streams. We started with Graphite but switched to InfluxDB for it's flexibility (Grafana works with both!) I would add to the guide that you need to be careful about formatting the lines into InfluxDB because where you out the space and commas determines what is indexed or not! Also data types should be specific (ie make sure you are setting integer vs float correctly).

x87678r 6y ago

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

halfmatthalfcat 6y ago
The question is how to do long term storage though. Something I've had a bit of trouble rationing about. Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.
- x87678r 6y ago
  Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.
  - julioo 6y ago
    In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.
    - julioo 6y ago
      Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.
- dewey 6y ago
  I think that's what https://thanos.io/ https://thanos.io/ is for.
  - 6y ago

majkinetor 6y ago

You can quickly do this on Windows: cinst influxdb1 /Service cinst grafana start $Env:ChocolateyInstall\lib\grafana\tools\grafana-*\bin\grafana-server.exe git clone https://github.com/majkinetor/psinflux import-module ./psinflux 1..10 | % { $x = 10*$_ + (Get-Random 10); Send-Data "test1 value=$x"; sleep 1 }

PanosJee 6y ago

Too many parts Netdata will make it a breeze

linsomniac 6y ago
Netdata is pretty slick.

carlosdp 6y ago

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

Hates_ 6y ago
I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.
- dharmab 6y ago
  Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.
  - edoceo 6y ago
    My company's Prometheus setup was super easy, one $10/mo box. About 1 week of fiddling all the exporters and configs but now it just runs and has for months. You can be small with Prometheus and grow into needing an FTE for it - w/o having the migration hurdle of moving out-source to in-source
  - lima 6y ago
    Please elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).
    - gen220 6y ago
      I think the parent's complaint is a function of how your engineering org "uses" prometheus. If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus). IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".

burtonator 6y ago

I've done this in my company with about 100 servers using KairosDB + Grafana and CollectD... https://kairosdb.github.io/ https://kairosdb.github.io/ If I were to do it again I'd probably use Grafana and Prometheus

deleted 6y ago

[deleted]

avthar 6y ago

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB. We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-time-series-database-postgresql-free-tsdb/ https://blog.timescale.com/blog/multi-node-petabyte-scale-ti... Today TimescaleDB outperforms InfluxDB across almost all dimensions (credit goes to our database team!), especially for high-cardinality workloads: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/ https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-... TimescaleDB also works with Grafana, Prometheus, Telegraf, Kafka, Apache Spark, Tableau, Django, Rails, anything that speaks SQL...

jimaek 6y ago
While influx is pretty bad overall it's super simple to deploy and configure unlike timescale. It's the main reason we decided to use influx in our small team with simple enough timeseries needs
- avthar 6y ago
  Curious what you found difficult to deploy/ configure? Is this in a self-managed context?
  - samstave 6y ago
    I found this to be a funny, subtle insult. :-) "Why is it difficult, because you're self managed?"
    - avthar 6y ago
      No intention to insult - just trying to get more context :)
      - 6y ago

zytek 6y ago

VictoriaMetrics eats other TSDBs for breakfast. PromQL support (with extensions) and clustered / HA mode. Great storage efficiency. Plays well for monitoring multiple k8s clusters, works great with Grafana, pretty easily deployed on k8s. No affiliation, just a happy user.

mekster 6y ago
I just don't get why VictoriaMetrics doesn't get more visibility. Maybe they need a PR person.
- valyala 6y ago
  Absolutely! We are working on this.

Papric0re 6y ago

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention of PCAPs) and save some security logs (I'm not intending on alerting based on logs, just for retention). But being able to have a network overview, insight into current connections (including history) is a very useful thing. Can anybody recommend something, that's maybe a bit lighter than an entire Elastic-Stack?

floren 6y ago
I think Gravwell (https://gravwell.io https://gravwell.io) might be what you're looking for--but I work for Gravwell so I may be biased! If I can be forgiven a short sales pitch, we've built a format-agnostic storage & querying system that easily handles raw packets, Netflow records (v5, v9, and IPFIX), collectd data, Windows event logs, and more. You can see some screenshots at https://www.gravwell.io/technology https://www.gravwell.io/technology We have a free tier which allows 2GB of data ingest per day (paid licenses are unlimited) which should be more than enough for capturing logs and flows. The resources needed to run Gravwell basically scale with how much data you put into it, but it's a lot quicker to install and set up than something like Elastic, in our opinion (https://www.gravwell.io/blog/gravwell-installed-in-2-minutes https://www.gravwell.io/blog/gravwell-installed-in-2-minutes) Edit: it's currently a bit roll-your-own, but we're really close to releasing Gravwell 4.0 which enables pre-packaged "kits" containing dashboards, queries, etc. for a variety of data types (Netflow, CoreDNS logs, and so on)
- Papric0re 6y ago
  I don't mind you giving a small sales pitch...and maybe your product is indeed what I'm searching for. But your pricing model is instantly putting me off. Same as with Splunk, you end up with not being able to predict your cost and paying way too much. Tell me when you fix that and I might be interested ;) Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)

tgtweak 6y ago

Looking for a similar post with APM using opentracing !

rattray 6y ago

I've only ever used third party monitoring tools, but hope to set up a startup again soon and want to do OSS if I can. Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?

k-rus 6y ago
You can use Prometheus on top of TimescaleDB. Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way. Sorry for the promo :) check for details in https://github.com/timescale/timescale-prometheus https://github.com/timescale/timescale-prometheus
- rattray 6y ago
  Thanks!
cevian 6y ago
(Timescale-Prometheus team lead here) I'd suggest using Timescale-Prometheus to connect Prometheus with TimescaleDB. Repo: https://github.com/timescale/timescale-prometheus https://github.com/timescale/timescale-prometheus Design Doc: https://tsdb.co/prom-design-doc https://tsdb.co/prom-design-doc
valyala 6y ago
Prometheus is a monitoring system (it collects metrics and evaluates alerts on them), while TimescaleDB is a database. While it is possible to use TimescaleDB as a remote storage for Prometheus, I'd recommend taking a look at other remote storage integrations as well [1]. Some of them natively support PromQL (like VictoriaMetrics, m3db, Cortex), others may be easier to setup and operate. For instance, VictoriaMetrics works out of the box without complex configuration. Disclaimer: I work for VictoriaMetrics :). [1] https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage https://prometheus.io/docs/operating/integrations/#remote-en...

aprdm 6y ago

As mentioned by others I have found a Prometheus + Grafana setup to be pretty straight forward to install and maintain. It also aligns with the cloud native foundation which means that this stack easily integrates with a bunch of OSS stack used by cloud native companies.

hnarn 6y ago

Some people are probably going to throw some shade on me for saying this since it's so out of fashion but in my mind, when it comes to some types of basic monitoring (SNMP monitoring of switches/linux servers, disk space usage, backups running and handling them when they don't) then Nagios does get the job done. It's definitely olives and not candy[1] but it's stable, modular, relatively easy to configure (when you get it) and it just keeps chugging away. If anyone has any Nagios questions, I'd be happy to answer them. I'm a Nagios masochist. (Also, I can recommend Nagflux[2] to export performance data, metrics, to InfluxDB because noone should have to touch RRD files) [1]: https://lambdaisland.com/blog/2019-08-07-advice-to-younger-self https://lambdaisland.com/blog/2019-08-07-advice-to-younger-s... [2]: https://github.com/Griesbacher/nagflux https://github.com/Griesbacher/nagflux

brightball 6y ago
Ever looked at Zabbix?
- hnarn 6y ago
  I haven't, unfortunately, but it looks promising from just looking into it briefly. Open source monitoring is always an area that needs more competition.
  - Twirrim 6y ago
    Zabbix has been around for quite a long time. Easily 15 years now. I haven't looked at it since around 2013, but at the time it was placing quite some pressure on a mysql db backend. It looks like they've expanded out to support more than MySQL as the back-end these days.
    - k-rus 6y ago
      I noticed that Zabbix supports PostgreSQL and TimescaleDB as back-ends and just checked the list, which contains also Oracle and SQLite (DB2 support is experimental).
    - 6y ago

9thbyte 6y ago

This seems like a lot of work when you could use Zabbix. ¯\_(ツ)_/¯

9thbyte 6y ago

Seems like a lot of work when Zabbix is available. Read the latest version even supports TimescaleDB.

linsomniac 6y ago

I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk). Influxdb has been a real workhorse. We suffered through some of their early issues, but since then it's been extremely solid. It just runs, is space efficient, and very robust. We almost went with Prometheus instead of Influx (as I said, early growing pains), but I had just struggled through managing a central inventory and hated it, so I really wanted a push rather than pull architecture. But from my limited playing with it, Prometheus seemed solid.

drez 6y ago
I just wanted to echo this sentiment. Influx had its fair share of issues a few years ago (v0.8 migration, changing storage engines, tag cardinality issues), but the latest v1.x releases have been solid. I have been using the TIK stack (I use Grafana instead of Chronograf) for monitoring several dozen production-facing machines for 2 years now without a single issue, which I would very much count as a win. I just hope they learned their lesson for the v2.0 release...
rehevkor5 6y ago
It's so much easier to write incorrect/misleading queries in influxql than in promql. And you can't perform operations between two different series names in influxdb, last I looked. That makes it impossible to do things like ratios or percentages unless you have control over your metrics, and structure them the way influx likes. Also, no support for calculating percentiles from Prometheus histogram buckets.
- mhall119 6y ago
  You can do these things (and much, much more) with the new Flux language
valyala 6y ago
> I really wanted a push rather than pull architecture Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with big number of time series (aka high cardinality) [2]. [1] https://victoriametrics.github.io/#how-to-import-time-series-data https://victoriametrics.github.io/#how-to-import-time-series... [2] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl...

9thbyte 6y ago

This is like creating Zabbix in pieces and missing the best parts. Triggers, Alerts, and Actions.

aequitas 6y ago

For those who still remember Graphite, the team over at Grafana labs have started maintaining Graphite-web and Carbon since 2017 and it is still in active development getting improvements and feature updates. It might not scale as well as any of the other solutions, but for medium size or homelab setups it's still a nice solution if you don't like PromQL or InfluxQL. https://grafana.com/oss/graphite/ https://grafana.com/oss/graphite/

lma21 6y ago
for simple and fast monitoring solutions, i always opted for collected + graphite + grafana. In a containerized environment, it's so easy to deploy (0 configuration by default) and monitor a set of 50-100 nodes. Beyond that, disk tuning and downsampling (pre-aggregation) rules become important.
rehevkor5 6y ago
If you're still on graphite and need scaling, metrictank is something to consider.

Aaronstotle 6y ago

I recognized the name almost immediately, cool to see a blog post from a member of the monero community.

crecker 6y ago
haha thanks!

mickeyben 6y ago

We're currently using InfluxDB but maintenance is something we'd like to stop doing on our monitoring stack. Datadog is too expensive because of the number of hosts we have. So we're thinking of eventually going to a hosted InfluxDB setup. But we also want to revisit other hosted solutions. Does someone have some experience with using Cloudwatch + Grafana? I've used Cloudwatch many years ago and it was clearly subpar to something like Influx. Is is better nowadays?

metalliqaz 6y ago

I use Telegraf, InfluxDB, and Grafana for my home network, in addition to monitoring my weather station data. It works incredibly well and is so simple compared to the "old" suite of tools such as RRD and Cacti.

sahoo 6y ago

How does this compare to grafana, Prometheus and node monitor

jakobdabo 6y ago

I still can't find any alternative to the old, RRD-based Munin. It is so simple. You want to add a new server to monitor? Just install there the node part, enable any required additional plugins (just by creating a couple of soft-links), add one-line configuration to the main server with the new node's IP address, and you are done. Also, the aesthetics of the UX, you see all the graphs in one single page[1], no additional clicks are required - a quick glance with a slow scroll and you can see if there were any unusual things during the last day/week. [1] - publicly available example, found by googling - https://ansible.fr/munin/agate/agate/index.html https://ansible.fr/munin/agate/agate/index.html

mhall119 6y ago
That's more steps than using InfluxDB and Telegraf. Check out: https://github.com/influxdata/community-templates/tree/master/linux_system https://github.com/influxdata/community-templates/tree/maste...

ecoqba11 6y ago

We switched from InfluxDB to TimescaleDB for our IoT solutions. InfluxDB is very difficult to work with large datasets and enterprise/region compliance. We ingest around 100MB data per day and growing.

mhall119 6y ago
That doesn't actually sound like a large dataset. Can you describe what kind of problems you faced with InfluxDB?
valyala 6y ago
100MB data per day looks like a tiny number. Our users ingest 100GB data per day at the ingestion rate of 1M data points per second on a single-node setup [1]. [1] https://victoriametrics.github.io/CaseStudies.html https://victoriametrics.github.io/CaseStudies.html

DevKoala 6y ago

I’d replace InfluxDB with Prometheus. InfluxDB has been a PITA for me.

mavam 6y ago

Does anyone have experience with an InfluxDB-based netdata setup? When we evaluated multiple options, we found that netdata does a lot of the heavy lifting. What are the advantages of a CollectD-based setup?

zacksh 6y ago
Well, that's the beauty of Netdata, no need to configure collectors. As for InfluxDB integration, it's fairly easy to export from Netdata using what InfluxDB calls the line protocol format (the "plaintext interface"). https://learn.netdata.cloud/docs/agent/exporting https://learn.netdata.cloud/docs/agent/exporting