I note that a lot of people on this thread are saying that there are better TSDBs out there. There probably are (I personally don't like Prometheus), but they a

I note that a lot of people on this thread are saying that there are better TSDBs out there. There probably are (I personally don't like Prometheus), but they are missing the point which is: having metrics is better than not. Now, for something constructive: what to metric and how? Because time series DBs tend to "bin" metrics, that is put metrics into 1s/1m/5m "holes" you will end up being at the mercy of quantisation/averaging. To get round this, use counters for most thing. This has the advantage that its much simpler implement accurately as a programmer. every time an error happens, increment an error, every time you get a 200, increment. but why? because if you do the math locally and work out the hit/second and emit that number, if you get a spike, and you aren't aware of the time binning, your 45k 1 minute spike will be averaged out as the retention lowers the temporal resolution. This means that your 45k spike could be halved each time you go down a temporal step. Using incremental counters overcomes this, even if temporal resolution is thinned. (you can use derivative (well non negative derivative) to turn it into hit/s later on) Microservices and metrics Microservices tend to be simpler to add monitoring, as they _should_ be simple. Yes, by all means record memory, CPU, network and that, but you will most likely not look at that unless you have a difficult bug. make sure each of your instances of the service has a unique ID in the metric address. This will save you many hours of fuckery. Have counters for all upstream and down stream requests(and error states). Then have gauges for response times of those requests. (beware that gauges will be averaged by time bin.) Do the same for all of the important actions your service does as well. These metrics will give you a good base to make alerts from: service x has had 1500 request errors from service y conversely it can allow you to summarise all your metrics into a green/orange/red tile (ie if no errors in the last minute, be green)