3 ms·

How do people tend to get metrics off of embedded devices? I’m working on autometrics (https://github.com/autometrics-dev/autometrics-rs https://github.com/aut

How do people tend to get metrics off of embedded devices? I’m working on autometrics (https://github.com/autometrics-dev/autometrics-rs https://github.com/autometrics-dev/autometrics-rs) and people asked whether it could be used in embedded contexts but I wasn’t sure how you’d hook up the device to something like Prometheus.

MasterIdiot 4y ago
Prometheus push gateway maybe? I guess it depends on limitations like battery life, bandwidth and network access (maybe it's behind a NAT or something)
teaearlgraycold 4y ago
The Pixel phones have an embedded environment hooked up to the normal ARM/Linux SoC. So you pipe from embedded to Linux and then Linux relays up to the cloud.
mycoliza 4y ago
in my use case (a hobby project; https://github.com/hawkw/eclss https://github.com/hawkw/eclss), the device and Prometheus are on the same LAN and i have Prometheus set up to discover scrape targets using multicast DNS. this is…probably not what you’d do if you were shipping a consumer project, i think, but it’s a nice setup for a home network.
- emschwartz 4y ago
  Ah, that is indeed a cool setup for a device on your home network :)
ilyt 4y ago
For embedded systems probably just... allow users to read the metrics easily and let them implement what they need using the method of pushing they use. use-cases will probably be different enough that they'd need to customize it anyway. Most would probably end up as pushing metrics via... something (MQTT?) to gateway that will push it to their metrics system. Pulling/polling metrics honestly is pretty bad idea most of the time. It makes it easy to implement ("just* provide a web page with metrics) and allows poller to control intervals (which is only a problem if your configuration management isn't up to snuff), but complicates a lot, now you need to setup whole endpoint discovery instead of just saying "hey, apps, here is URL to push your metrics to, have fun". We for long (10+ year) have system based on collectd and it's line protocol, and while it have some drawbacks (no tags, just rigid structure with host->plugin->plugin_instance->type_type instance, works well most of the time but 100% of the time), push nature is very light on resources (it's just an UDP packet full of metrics, no need to setup whole TCP connection, doesn't even need bidirectional firewall)
- wbl 4y ago
  Polling has several benefits over pushing: a misbehaving endpoint can't knock out the whole system by having too much data, but at worse will just not get polled. Similarly load shedding is pretty simple. In my experience the discovery and configuration is needed anyway to do something with the metrics.
nagisa 4y ago
Small embedded devices tend have a serial port of some sort and some ad-hoc protocol or mechanism to trigger metric output. A prime example of this that comes to mind is the P1 port on some of the European electricity meters. This user-accessible port outputs some metrics every so often. Specification for the curious is here: https://www.netbeheernederland.nl/_upload/Files/Slimme_meter_15_a727fce1f1.pdf https://www.netbeheernederland.nl/_upload/Files/Slimme_meter... ARM chips intended for embedded uses also have a single wire output peripheral for tracing and similar purposes that could be used for this.
quake 4y ago
Memfault is another option.

MasterIdiot 4y ago

Prometheus push gateway maybe? I guess it depends on limitations like battery life, bandwidth and network access (maybe it's behind a NAT or something)

teaearlgraycold 4y ago

The Pixel phones have an embedded environment hooked up to the normal ARM/Linux SoC. So you pipe from embedded to Linux and then Linux relays up to the cloud.

mycoliza 4y ago

in my use case (a hobby project; https://github.com/hawkw/eclss https://github.com/hawkw/eclss), the device and Prometheus are on the same LAN and i have Prometheus set up to discover scrape targets using multicast DNS. this is…probably not what you’d do if you were shipping a consumer project, i think, but it’s a nice setup for a home network.

emschwartz 4y ago
Ah, that is indeed a cool setup for a device on your home network :)

ilyt 4y ago

For embedded systems probably just... allow users to read the metrics easily and let them implement what they need using the method of pushing they use. use-cases will probably be different enough that they'd need to customize it anyway. Most would probably end up as pushing metrics via... something (MQTT?) to gateway that will push it to their metrics system. Pulling/polling metrics honestly is pretty bad idea most of the time. It makes it easy to implement ("just* provide a web page with metrics) and allows poller to control intervals (which is only a problem if your configuration management isn't up to snuff), but complicates a lot, now you need to setup whole endpoint discovery instead of just saying "hey, apps, here is URL to push your metrics to, have fun". We for long (10+ year) have system based on collectd and it's line protocol, and while it have some drawbacks (no tags, just rigid structure with host->plugin->plugin_instance->type_type instance, works well most of the time but 100% of the time), push nature is very light on resources (it's just an UDP packet full of metrics, no need to setup whole TCP connection, doesn't even need bidirectional firewall)

wbl 4y ago
Polling has several benefits over pushing: a misbehaving endpoint can't knock out the whole system by having too much data, but at worse will just not get polled. Similarly load shedding is pretty simple. In my experience the discovery and configuration is needed anyway to do something with the metrics.

nagisa 4y ago

Small embedded devices tend have a serial port of some sort and some ad-hoc protocol or mechanism to trigger metric output. A prime example of this that comes to mind is the P1 port on some of the European electricity meters. This user-accessible port outputs some metrics every so often. Specification for the curious is here: https://www.netbeheernederland.nl/_upload/Files/Slimme_meter_15_a727fce1f1.pdf https://www.netbeheernederland.nl/_upload/Files/Slimme_meter... ARM chips intended for embedded uses also have a single wire output peripheral for tracing and similar purposes that could be used for this.

quake 4y ago

Memfault is another option.