3 ms·
> Metrics are fetched over HTTP by Prometheus, and they usually don't come from files. You'd either have Prometheus scrape each job individually, or scrape your
by jabl 11y ago
> Metrics are fetched over HTTP by Prometheus, and they usually don't come from files. You'd either have Prometheus scrape each job individually, or scrape your controller daemon that'd attach a labels indicating the job of each metric.
I was thinking more of the data model (https://prometheus.io/docs/concepts/data_model/ https://prometheus.io/docs/concepts/data_model/). Specifically, from https://prometheus.io/docs/practices/naming/ https://prometheus.io/docs/practices/naming/ , "Remember that every unique key-value label pair represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.". So in our case the job ID would be such an unbounded value..?
- bbrazil 11y agoThese would be the exact details I'd need to advise you. It depends on how many jobs you have, how much churn there is, and what exactly you want to monitor. If jobs are short-lived then tracking individual jobs would be unwise, something like the ELK stack intended for event logging would be better. If jobs are long-lived and there's not many of them then you should be okay. Otherwise you'd just be looking at tracking system rather than per-job stats. To give a very rough idea, if you can keep it below say 10M metrics across the history a single Prometheus server has that should be okay with the current implementation.