4 ms·

What's the general thinking around putting private identifiers (eg user name) in kube labels? That data would be going to a monitoring stack, eg Prometheus or h

What's the general thinking around putting private identifiers (eg user name) in kube labels? That data would be going to a monitoring stack, eg Prometheus or hosted Cortex?

michaelmdresser 5y ago
I'm a little confused by this use case. Do you spin up individual pods for each user/group? Is this for other resources like secrets?
- ben_olsen 5y ago
  We spin up a new namespace for each company/user on our platform. Looking to add labels as identifiers for general analysis and alerting.
AjayTripathy 5y ago
Each use case is different, but I think expiring the data after 60d in your monitoring stack makes sense both from a scaling perspective of your monitoring stack and from a privacy perspective. I wouldn't necessarily put _user_ data in labels, but team/product names and contact info of the coworkers responsible for the service seem fine to me.
brancz 5y ago
Unless you explicitly allow them (via a flag) in your kube-state-metrics deployment, Kubernetes labels and annotations are not translated into Prometheus labels.
- jrockway 5y ago
  kube-state-metrics doesn't specifically matter. Prometheus has built-in Kubernetes service discovery, and you can drop labels/annotations or not. Most people never see this because they take an opaque blob from a vendor (helm chart, prometheus-operator), etc. -- but it's there: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config https://prometheus.io/docs/prometheus/latest/configuration/c... kube-state-metrics does its own service discovery and outputs prometheus metrics that it cares about, but any app could do that.
  - brancz 5y ago
    You’re absolutely right, though none of these transfer labels by default (neither does the example Prometheus config in the prometheus repo). Prometheus, including Prometheus operator require you to allow-list them. (I should have mentioned that I maintain the Prometheus Kubernetes SD, and am the original creator of kube-state-metrics, prometheus-operator and kube-prometheus)

What's the general thinking around putting private identifiers (eg user name) in kube labels? That data would be going to a monitoring stack, eg Prometheus or hosted Cortex?

michaelmdresser 5y ago

I'm a little confused by this use case. Do you spin up individual pods for each user/group? Is this for other resources like secrets?

ben_olsen 5y ago
We spin up a new namespace for each company/user on our platform. Looking to add labels as identifiers for general analysis and alerting.

AjayTripathy 5y ago

Each use case is different, but I think expiring the data after 60d in your monitoring stack makes sense both from a scaling perspective of your monitoring stack and from a privacy perspective. I wouldn't necessarily put _user_ data in labels, but team/product names and contact info of the coworkers responsible for the service seem fine to me.

brancz 5y ago

Unless you explicitly allow them (via a flag) in your kube-state-metrics deployment, Kubernetes labels and annotations are not translated into Prometheus labels.

jrockway 5y ago
kube-state-metrics doesn't specifically matter. Prometheus has built-in Kubernetes service discovery, and you can drop labels/annotations or not. Most people never see this because they take an opaque blob from a vendor (helm chart, prometheus-operator), etc. -- but it's there: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config https://prometheus.io/docs/prometheus/latest/configuration/c... kube-state-metrics does its own service discovery and outputs prometheus metrics that it cares about, but any app could do that.
- brancz 5y ago
  You’re absolutely right, though none of these transfer labels by default (neither does the example Prometheus config in the prometheus repo). Prometheus, including Prometheus operator require you to allow-list them. (I should have mentioned that I maintain the Prometheus Kubernetes SD, and am the original creator of kube-state-metrics, prometheus-operator and kube-prometheus)