5 ms·

Why not a unixish simple tool to handle metrics? assuming $prog outputs log info to stdout: $prog | collect | action and 'collect | action' could all be done so

Why not a unixish simple tool to handle metrics? assuming $prog outputs log info to stdout: $prog | collect | action and 'collect | action' could all be done somewhat simply with awk and 1000x more portable.

skykooler 8y ago
because some people want prebuilt solutions rather than messing with command-line scripts for everything.
- nwmcsween 8y ago
  So people want non unixish tools on unix, seems like a road we keep going down over and over and eventually goes back to the unix way. Also piping isn't 'scripting'
kryptk 8y ago
And store it where? And query it how? What will render my NoC dashboards? With graphite protocol the metric stream is already plain-text with just 3 fields per line (path value time), its pumped over a socket to a collector.. but that's where the hard stuff starts.
- nwmcsween 8y ago
  First I'm not trashing the project, I'm just wondering why simple unix like solutions aren't used. Store it where ever you want, this isn't a magical datastore that makes things faster, use clickhouse-client, whatever it doesn't matter. There is a widening disconnect between the unix way and how new projects are created.
sciurus 8y ago
From what I can tell you're talking about processing logs to device metrics, but this project is specifically about storing the raw logs themselves. Two different (albeit potentially related) problems. What you are proposing for deriving metrics from logs exists in plenty of forms. Check out https://github.com/google/mtail https://github.com/google/mtail , for example. Of course, you still have to store them somewhere, visualize them somehow, etc. That won't simply be part of your shell pipeline.
SuperQue 8y ago
This is not metrics, this is logs. Logs contain a lot of detail that are not appropriate for metrics. Logs output tends to be expensive, it costs a lot of time and IO to emit, compress, send, store logs. Logs also tend to contain PII things. Usernames, IP addresses, trace IDs, etc. We don't want that kind of cardinality in metrics. Metrics are another thing. They tend to be internal counters inside a single process. With multi-threaded languages like Go, Java, C++, we can track metric data in memory very efficiently. For metrics, see https://openmetrics.io/ https://openmetrics.io/
- nwmcsween 8y ago
  If you read my earlier posts logs vs metrics doesn't matter in this context and hence why I said to stream to clickhouse-client...

skykooler 8y ago

because some people want prebuilt solutions rather than messing with command-line scripts for everything.

nwmcsween 8y ago
So people want non unixish tools on unix, seems like a road we keep going down over and over and eventually goes back to the unix way. Also piping isn't 'scripting'

kryptk 8y ago

And store it where? And query it how? What will render my NoC dashboards? With graphite protocol the metric stream is already plain-text with just 3 fields per line (path value time), its pumped over a socket to a collector.. but that's where the hard stuff starts.

nwmcsween 8y ago
First I'm not trashing the project, I'm just wondering why simple unix like solutions aren't used. Store it where ever you want, this isn't a magical datastore that makes things faster, use clickhouse-client, whatever it doesn't matter. There is a widening disconnect between the unix way and how new projects are created.

sciurus 8y ago

From what I can tell you're talking about processing logs to device metrics, but this project is specifically about storing the raw logs themselves. Two different (albeit potentially related) problems. What you are proposing for deriving metrics from logs exists in plenty of forms. Check out https://github.com/google/mtail https://github.com/google/mtail , for example. Of course, you still have to store them somewhere, visualize them somehow, etc. That won't simply be part of your shell pipeline.

SuperQue 8y ago

This is not metrics, this is logs. Logs contain a lot of detail that are not appropriate for metrics. Logs output tends to be expensive, it costs a lot of time and IO to emit, compress, send, store logs. Logs also tend to contain PII things. Usernames, IP addresses, trace IDs, etc. We don't want that kind of cardinality in metrics. Metrics are another thing. They tend to be internal counters inside a single process. With multi-threaded languages like Go, Java, C++, we can track metric data in memory very efficiently. For metrics, see https://openmetrics.io/ https://openmetrics.io/

nwmcsween 8y ago
If you read my earlier posts logs vs metrics doesn't matter in this context and hence why I said to stream to clickhouse-client...