10 ms·

Do we need to store all that telemetry?

102 points by mklein123 2y ago

gghffguhvc 2y ago
“A lot of telemetry doesn’t need to be stored for very long” is the attitude I take. Keeps costs down but gives good visibility.
deleted 2y ago
[deleted]
blobcode 2y ago
> we can also add local storage of telemetry data in an efficient circular buffer. Typically, local storage is cheap and underutilized, allowing for “free” storage of a finite amount of historical data, that wraps automatically. Local storage provides the ability to “time travel” when a particular event is hit. I think that this is a good idea when storage is concern for high-volume logs / production. Persisting the buffer when high error rates / unusual system behavior is observed would be a cool idea.
- taneq 2y ago
  This is a good approach and is pretty common in the embedded world. You use a ring buffer to store a relatively short but detailed log, and then if you encounter an error (or whatever other relevant trigger criteria you use) you snapshot the contents of that ring buffer. Then later you can retrieve the snapshots to figure out what happened.
jedberg 2y ago
NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operational issues. Keeping all of your application logs and telemetry forever is expensive, and I can't recall a single time when having more than a day's with of history was ever useful in tracking down an operational issue.
- Etheryte 2y ago
  I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.
  - jedberg 2y ago
    You should have a ton of telemetry on business metrics. You would absolutely notice an issue before your users if you have those. For example at Netflix we monitored stream starts per second -- how often you hit play and it worked. That metric was the most important, and the one that triggered most investigations. If your CPU and memory aren't affecting the business metrics, then it's not super relevant.
  - dimitar 2y ago
    Very few things are worth keeping after two weeks, I like short retention policies
  - tjoff 2y ago
    Very little harm... If the telemetry is from your users I'd like you to value them more than that. Also consider the potential risks of handling personal data and leaks.
    - Etheryte 2y ago
      This only holds if you assume telemetry means personal data, but that is a very big if. Meta, Google and other giants generally deal in telemetry that includes personal data, however for most run of the mill software that's not the case. Outside of advertising, I would argue that for most applications you're already pretty close to being clear of personal data as long as you exclude the user's email and other identifiers from the logs. Sure, there are examples where this is not the case, but it isn't even remotely as big of a problem as you claim it to be.
      - 2y ago
mountainriver 2y ago
Great post, the observability folks have gone off the rails in the last 5 years. I’ve seen it do more harm than good in terms of dev speed and ironically often make things less observable for the common path.
thisislife2 2y ago
"Data is the new oil" - if you don't collect your customer data, and treat it as an asset, you are guilty of mismanagement . /s
- hooverd 2y ago
  Does that mean Google is the new Saudi Arabia?
  - ozymandias1337 2y ago
    Rock the Kasbah
  - akira2501 2y ago
    More like Iraq or Daniel Plainsview. Drilling diagonally to tap neighboring fields.
    - dragonwriter 2y ago
      That sounds like a popular misunderstanding of the mutual accusations before the 1990 invasion between Iraq and Kuwait of overpumping from the oil field that crosses their border (which did not involve “slant drilling”).
- ryandrake 2y ago
  Data is more like uranium than oil.[1] Valuable for its limited purpose, but dangerous to just collect and hold on to forever. 1: https://www.forbes.com/sites/forbestechcouncil/2022/10/03/the-data-security-conundrum-is-it-oil-or-uranium/?sh=ff8e9bde3c92 https://www.forbes.com/sites/forbestechcouncil/2022/10/03/th...
murat124 2y ago
YES you do. BUT with varying retention periods for each a) environment b) region c) function d) criticality e) metric namespace/name f) team etc. Nobody needs to retain metrics like CPU, Memory for weeks but I may want to see their numbers during an incident, or not long after it is over.
jauntywundrkind 2y ago
We've turned off logging & tracing on a bunch of our high volume routes. Ideally I'd prefer we still sample them, at like 0.1% or what not, to give us some indicator, some chance of seeing anomalies. It just seems easier to gather & use this information than it is to go develop a suite of metrics that can register all issues. OpenTelemetry recently ish gained Open Agent Management Protocol (OpAMP), which allows some runtime control over things generating telemetry. The ability to stay fairly low but then scale up as needed sounds tempting, but gee it also sends shivers down my spine thinking of having such a elastic demands on one's telemetry infrastructure, as engineers turn telemetry up as problems are occuring. https://opentelemetry.io/docs/specs/opamp/ https://opentelemetry.io/docs/specs/opamp/ The idea of having a local circular buffer sounds excellent to me. Being able to run local queries & aggregate would be sweet. Are there any open otel issues discussing these ideas?
wrs 2y ago
We continue to recreate features of single-computer OSes in distributed systems. This seems like the dtrace/bpftrace of microservices world.
jcgrillo 2y ago
Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000...
- simonw 2y ago
  I thought compressed JSON was pretty efficient. How much would you expect to save over that with a custom binary format?
  - tredre3 2y ago
    The network difference between compressed JSON or a compressed format is likely negligible. But jcgrillo was talking about storage (at least his link was). And when parsing for analysis or for storing millions of points daily, there's no doubt that a binary format is simply a lot more CPU and disk efficient.
    - winrid 2y ago
      Usually the JSON gets transformed into a binary format (example: BSON).
  - jcgrillo 2y ago
    The thing about telemetry data is it's extremely repetitive. Take for example a CLF[1] log line: 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 As written this is 99 bytes (792 bits), but how much information is actually in it? We have an IP address which is taking up 9 bytes but only needs at most 4 (fewer in cases like this where two of the bytes are zero if we employ varint encoding). Across log lines the ident and user will likely be very repetitive, so storing each unique occurrence more than once is really wasteful. The timestamp takes up 28 bytes but only needs 13 bytes--far fewer if that field is delta encoded between log lines. The HTTP method is taking up 5+ bytes, it's only worth 1 byte. The URLs are also super redundant--no need to store a copy in each line. The HTTP version is 1 byte but it's taking up 8. The status code is taking up 3 bytes but it's only worth 1--there are only 63 "real" HTTP status codes. The content length is taking up 4 bytes when it needs only 2. So I guess this log line only really has ~33 bytes of information in it (assuming a 32 bit pointer for each string--ident, user, URL). Much less if amortized across many lines. So maybe by naively parsing this log line and throwing a bunch of them in columnar, packed protobuf fields (where we get varint encoding for free), and delta-encoding the timestamps, and maintaining a dictionary for all the strings, we might achieve something like a ~5x compression ratio. Playing around with gzip -9 on some test data[2] (not exactly CLF, but maybe similar entropy) I'm getting like ~1.9x compression. Obviously if I parse this log line into a JSON blob, that blob will compress with a much higher ratio due to the repetitive nature of JSON, but it'll still be larger than the equivalent compressed CLF. I'm working on a demo for my "protobof + fst[3]" idea, so I'm not sure if my "maybe ~5x" claim is totally off the mark or not. But I'm confident we can do way better than JSON. [1] https://en.wikipedia.org/wiki/Common_Log_Format https://en.wikipedia.org/wiki/Common_Log_Format [2] https://www.sec.gov/about/data/edgar-log-file-data-sets https://www.sec.gov/about/data/edgar-log-file-data-sets [3] https://crates.io/crates/fst https://crates.io/crates/fst EDIT: I guess maybe another way to state my conjecture is "telemetry compression is not general purpose text compression". These data have a schema, and by ignoring that fact and treating them always as schemaless data (employing general purpose text compression methods) we're leaving something on the table.
binary132 2y ago
In general I think many programmers have internalized the idea that it’s best to waste as many computing resources as we can possibly afford as long as it’s not the bottleneck. Then, in the future, if and when it becomes the bottleneck, we’ll have plenty of headroom to optimize and look like heroes for saving the millions of dollars we never had to spend in the first place. It’s really insane (at best) or genuinely a type of grift at worst.
klabb3 2y ago
Isn’t the issue more that off-the-shelf solutions optimize for features and not cost? For instance, if I sell you an observability product, I want to show off all the cool realtime debugging features and such. And since there’s a cost to having all these features available (retention, indexing, sampling), we end up paying for features we don’t need. In a world of usage-based XaaS, there’s very little incentive to be cost-effective. Arguably even a perverse incentive to waste resources. I bet you a full dollar that both in-house and open source solutions, on average, are way more stingy with resources. As they should be.
- ChrisCooney 2y ago
  Hey! Just wanna wave a flag for Coralogix (I work there, disclaimer). We've built a ton of cost optimization because we know that the industry is just ridiculous right now. Your assessment is absolutely correct, and there are more than a few multi-billion dollar companies whose bottom line is predicated on their customers wasting money and being inefficient. We're not one of them! https://coralogix.com/ https://coralogix.com/
m3047 2y ago
Agree with the article enough that I did something about it which I call "Poor Fred's SIEM". The heart of it is a DNS proxy for Redis (https://github.com/m3047/rkvdns https://github.com/m3047/rkvdns). However it's not targeted at environments where everything is in a "bubble" such that there are no ingress / egress costs. (Lookin' at you, Cloud.) Furthermore "control plane" is an important concept, and it's well understood in the industrial control world as the Purdue Model. From a systems standpoint do you need to have all resources stored centrally in order to do centralized reporting? No, of course not. Admittedly it's handy if bandwidth and storage are free. The alternative is distributed storage, with or without summarization at the edge (and aggregating from distributed storage for reporting). Having it distributed does raise access issues: access needs to be controlled, and management of access needs to be managed. Philosophically the Cloud solutions sell centralized management, but federation is a perfectly viable option. The choice is largely dictated by organizational structure not technology. There is also a difference between diagnostic and evaluative indicators. Trying to evaluate from diagnostics causes fatigue because humans aren't built that way; evaluatives can and should be built from diagnostics. Diagnostics can't be built from evaluatives. The logging/telemetry stack that I propose is: 1) Ephemeral logging at the limits of whatever observability you can build. E.g.: systemd journal with a small backing store, similar to a ring buffer. 2) Your compliance framework may require shipping some classes of events off of the local host, but I don't think any of them require shipping it to the cloud. 3) Build evaluatives locally in Redis. 4) Use DNS to query those evaluatives from elsewhere for ad hoc as well as historical purposes. This could be a centralized location or it could be true federation where each site accesses all other site's evaluatives. I wouldn't put Redis on the internet, but I don't worry too much about DNS; and there are well-understood ways of securing DNS from tampering, unauthorized access, and even observation. By the way, DNS will handle hundreds or thousands of queries per second you just have to build for it.
- m3047 2y ago
  So I went off and set up an actual "live fire" demo because it's that easy: curl http://athena.m3047.net/grafitti.html dig @athena.m3047.net grafitti\;*.keys.redis.athena.m3047 txt
h1fra 2y ago
I understand the point but I also advocate for the opposite, it's not cool for the planet for sure but having all the data points for at least a couple of months is very useful on any large system and +15months for metrics so you can compare with the year before. I can't count the number of times users (or myself) discovered bug after many weeks because something gradually failed over time. Also it saves a lot of time to be able to pin point the exact day a behavior as changed so you can check the deploy of that day and quickly find the bug. Sometimes a trend is not obvious after a deploy but is clearly visible on the graph after a long period of time. And for business intelligence, it's always when you badly need a metric that you realize you never tracked it.
- jcgrillo 2y ago
  Yeah I've definitely been saved a bunch of times by long retention, and the BI questions that might arise are impossible to predict. So some sort of retention is definitely necessary, IME. But let's take the case of metrics as an example--do we need full sample granularity for "old" data? Do we need full tag cardinality? Sample granularity reduction could be done with a transform to rollups at a coarser time granularity. That's a 60x reduction going from Hz to 1/min. You might lose a bunch of frequency information this way, but maybe that's ok? Numbers are really nice in ways that text is not.
zug_zug 2y ago
I think most places don't collect enough telemetry in the right formats. It's also possible they collect too much in the wrong formats. But the ability to vet a hypothesis (I bet our users are confused about feature X, which we can test by looking at how many times they go to page X, then Y, then X again in 30 second window) in an hour versus 2 sprints is vastly underappreciated/underutilized. I feel like this article paints with too broad a brush.
yetanotherdood 2y ago
> For 30 years how telemetry is produced has not changed: we define all of the data points that we need ahead of time and ship them out of the origin process, typically at large expense. If we apply the control plane / data plane split to observability telemetry production we can fundamentally change the status quo for the first time in three decades Has Matt read any prior art in this field? https://research.google/pubs/monarch-googles-planet-scale-in-memory-time-series-database/ https://research.google/pubs/monarch-googles-planet-scale-in...

102 points by mklein123 2y ago

“A lot of telemetry doesn’t need to be stored for very long” is the attitude I take. Keeps costs down but gives good visibility.

deleted 2y ago

[deleted]

blobcode 2y ago

> we can also add local storage of telemetry data in an efficient circular buffer. Typically, local storage is cheap and underutilized, allowing for “free” storage of a finite amount of historical data, that wraps automatically. Local storage provides the ability to “time travel” when a particular event is hit. I think that this is a good idea when storage is concern for high-volume logs / production. Persisting the buffer when high error rates / unusual system behavior is observed would be a cool idea.

taneq 2y ago
This is a good approach and is pretty common in the embedded world. You use a ring buffer to store a relatively short but detailed log, and then if you encounter an error (or whatever other relevant trigger criteria you use) you snapshot the contents of that ring buffer. Then later you can retrieve the snapshots to figure out what happened.

jedberg 2y ago

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operational issues. Keeping all of your application logs and telemetry forever is expensive, and I can't recall a single time when having more than a day's with of history was ever useful in tracking down an operational issue.

Etheryte 2y ago
I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.
- jedberg 2y ago
  You should have a ton of telemetry on business metrics. You would absolutely notice an issue before your users if you have those. For example at Netflix we monitored stream starts per second -- how often you hit play and it worked. That metric was the most important, and the one that triggered most investigations. If your CPU and memory aren't affecting the business metrics, then it's not super relevant.
- dimitar 2y ago
  Very few things are worth keeping after two weeks, I like short retention policies
- tjoff 2y ago
  Very little harm... If the telemetry is from your users I'd like you to value them more than that. Also consider the potential risks of handling personal data and leaks.
  - Etheryte 2y ago
    This only holds if you assume telemetry means personal data, but that is a very big if. Meta, Google and other giants generally deal in telemetry that includes personal data, however for most run of the mill software that's not the case. Outside of advertising, I would argue that for most applications you're already pretty close to being clear of personal data as long as you exclude the user's email and other identifiers from the logs. Sure, there are examples where this is not the case, but it isn't even remotely as big of a problem as you claim it to be.
    - 2y ago

mountainriver 2y ago

Great post, the observability folks have gone off the rails in the last 5 years. I’ve seen it do more harm than good in terms of dev speed and ironically often make things less observable for the common path.

thisislife2 2y ago

"Data is the new oil" - if you don't collect your customer data, and treat it as an asset, you are guilty of mismanagement . /s

hooverd 2y ago
Does that mean Google is the new Saudi Arabia?
- ozymandias1337 2y ago
  Rock the Kasbah
- akira2501 2y ago
  More like Iraq or Daniel Plainsview. Drilling diagonally to tap neighboring fields.
  - dragonwriter 2y ago
    That sounds like a popular misunderstanding of the mutual accusations before the 1990 invasion between Iraq and Kuwait of overpumping from the oil field that crosses their border (which did not involve “slant drilling”).
ryandrake 2y ago
Data is more like uranium than oil.[1] Valuable for its limited purpose, but dangerous to just collect and hold on to forever. 1: https://www.forbes.com/sites/forbestechcouncil/2022/10/03/the-data-security-conundrum-is-it-oil-or-uranium/?sh=ff8e9bde3c92 https://www.forbes.com/sites/forbestechcouncil/2022/10/03/th...

murat124 2y ago

YES you do. BUT with varying retention periods for each a) environment b) region c) function d) criticality e) metric namespace/name f) team etc. Nobody needs to retain metrics like CPU, Memory for weeks but I may want to see their numbers during an incident, or not long after it is over.

jauntywundrkind 2y ago

We've turned off logging & tracing on a bunch of our high volume routes. Ideally I'd prefer we still sample them, at like 0.1% or what not, to give us some indicator, some chance of seeing anomalies. It just seems easier to gather & use this information than it is to go develop a suite of metrics that can register all issues. OpenTelemetry recently ish gained Open Agent Management Protocol (OpAMP), which allows some runtime control over things generating telemetry. The ability to stay fairly low but then scale up as needed sounds tempting, but gee it also sends shivers down my spine thinking of having such a elastic demands on one's telemetry infrastructure, as engineers turn telemetry up as problems are occuring. https://opentelemetry.io/docs/specs/opamp/ https://opentelemetry.io/docs/specs/opamp/ The idea of having a local circular buffer sounds excellent to me. Being able to run local queries & aggregate would be sweet. Are there any open otel issues discussing these ideas?

wrs 2y ago

We continue to recreate features of single-computer OSes in distributed systems. This seems like the dtrace/bpftrace of microservices world.

jcgrillo 2y ago

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https://lists.w3.org/Archives/Public/www-logging/1996May/0000.html https://lists.w3.org/Archives/Public/www-logging/1996May/000...

simonw 2y ago
I thought compressed JSON was pretty efficient. How much would you expect to save over that with a custom binary format?
- tredre3 2y ago
  The network difference between compressed JSON or a compressed format is likely negligible. But jcgrillo was talking about storage (at least his link was). And when parsing for analysis or for storing millions of points daily, there's no doubt that a binary format is simply a lot more CPU and disk efficient.
  - winrid 2y ago
    Usually the JSON gets transformed into a binary format (example: BSON).
- jcgrillo 2y ago
  The thing about telemetry data is it's extremely repetitive. Take for example a CLF[1] log line: 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 As written this is 99 bytes (792 bits), but how much information is actually in it? We have an IP address which is taking up 9 bytes but only needs at most 4 (fewer in cases like this where two of the bytes are zero if we employ varint encoding). Across log lines the ident and user will likely be very repetitive, so storing each unique occurrence more than once is really wasteful. The timestamp takes up 28 bytes but only needs 13 bytes--far fewer if that field is delta encoded between log lines. The HTTP method is taking up 5+ bytes, it's only worth 1 byte. The URLs are also super redundant--no need to store a copy in each line. The HTTP version is 1 byte but it's taking up 8. The status code is taking up 3 bytes but it's only worth 1--there are only 63 "real" HTTP status codes. The content length is taking up 4 bytes when it needs only 2. So I guess this log line only really has ~33 bytes of information in it (assuming a 32 bit pointer for each string--ident, user, URL). Much less if amortized across many lines. So maybe by naively parsing this log line and throwing a bunch of them in columnar, packed protobuf fields (where we get varint encoding for free), and delta-encoding the timestamps, and maintaining a dictionary for all the strings, we might achieve something like a ~5x compression ratio. Playing around with gzip -9 on some test data[2] (not exactly CLF, but maybe similar entropy) I'm getting like ~1.9x compression. Obviously if I parse this log line into a JSON blob, that blob will compress with a much higher ratio due to the repetitive nature of JSON, but it'll still be larger than the equivalent compressed CLF. I'm working on a demo for my "protobof + fst[3]" idea, so I'm not sure if my "maybe ~5x" claim is totally off the mark or not. But I'm confident we can do way better than JSON. [1] https://en.wikipedia.org/wiki/Common_Log_Format https://en.wikipedia.org/wiki/Common_Log_Format [2] https://www.sec.gov/about/data/edgar-log-file-data-sets https://www.sec.gov/about/data/edgar-log-file-data-sets [3] https://crates.io/crates/fst https://crates.io/crates/fst EDIT: I guess maybe another way to state my conjecture is "telemetry compression is not general purpose text compression". These data have a schema, and by ignoring that fact and treating them always as schemaless data (employing general purpose text compression methods) we're leaving something on the table.

binary132 2y ago

In general I think many programmers have internalized the idea that it’s best to waste as many computing resources as we can possibly afford as long as it’s not the bottleneck. Then, in the future, if and when it becomes the bottleneck, we’ll have plenty of headroom to optimize and look like heroes for saving the millions of dollars we never had to spend in the first place. It’s really insane (at best) or genuinely a type of grift at worst.

klabb3 2y ago

Isn’t the issue more that off-the-shelf solutions optimize for features and not cost? For instance, if I sell you an observability product, I want to show off all the cool realtime debugging features and such. And since there’s a cost to having all these features available (retention, indexing, sampling), we end up paying for features we don’t need. In a world of usage-based XaaS, there’s very little incentive to be cost-effective. Arguably even a perverse incentive to waste resources. I bet you a full dollar that both in-house and open source solutions, on average, are way more stingy with resources. As they should be.

ChrisCooney 2y ago
Hey! Just wanna wave a flag for Coralogix (I work there, disclaimer). We've built a ton of cost optimization because we know that the industry is just ridiculous right now. Your assessment is absolutely correct, and there are more than a few multi-billion dollar companies whose bottom line is predicated on their customers wasting money and being inefficient. We're not one of them! https://coralogix.com/ https://coralogix.com/

m3047 2y ago

Agree with the article enough that I did something about it which I call "Poor Fred's SIEM". The heart of it is a DNS proxy for Redis (https://github.com/m3047/rkvdns https://github.com/m3047/rkvdns). However it's not targeted at environments where everything is in a "bubble" such that there are no ingress / egress costs. (Lookin' at you, Cloud.) Furthermore "control plane" is an important concept, and it's well understood in the industrial control world as the Purdue Model. From a systems standpoint do you need to have all resources stored centrally in order to do centralized reporting? No, of course not. Admittedly it's handy if bandwidth and storage are free. The alternative is distributed storage, with or without summarization at the edge (and aggregating from distributed storage for reporting). Having it distributed does raise access issues: access needs to be controlled, and management of access needs to be managed. Philosophically the Cloud solutions sell centralized management, but federation is a perfectly viable option. The choice is largely dictated by organizational structure not technology. There is also a difference between diagnostic and evaluative indicators. Trying to evaluate from diagnostics causes fatigue because humans aren't built that way; evaluatives can and should be built from diagnostics. Diagnostics can't be built from evaluatives. The logging/telemetry stack that I propose is: 1) Ephemeral logging at the limits of whatever observability you can build. E.g.: systemd journal with a small backing store, similar to a ring buffer. 2) Your compliance framework may require shipping some classes of events off of the local host, but I don't think any of them require shipping it to the cloud. 3) Build evaluatives locally in Redis. 4) Use DNS to query those evaluatives from elsewhere for ad hoc as well as historical purposes. This could be a centralized location or it could be true federation where each site accesses all other site's evaluatives. I wouldn't put Redis on the internet, but I don't worry too much about DNS; and there are well-understood ways of securing DNS from tampering, unauthorized access, and even observation. By the way, DNS will handle hundreds or thousands of queries per second you just have to build for it.

m3047 2y ago
So I went off and set up an actual "live fire" demo because it's that easy: curl http://athena.m3047.net/grafitti.html dig @athena.m3047.net grafitti\;*.keys.redis.athena.m3047 txt

h1fra 2y ago

I understand the point but I also advocate for the opposite, it's not cool for the planet for sure but having all the data points for at least a couple of months is very useful on any large system and +15months for metrics so you can compare with the year before. I can't count the number of times users (or myself) discovered bug after many weeks because something gradually failed over time. Also it saves a lot of time to be able to pin point the exact day a behavior as changed so you can check the deploy of that day and quickly find the bug. Sometimes a trend is not obvious after a deploy but is clearly visible on the graph after a long period of time. And for business intelligence, it's always when you badly need a metric that you realize you never tracked it.

jcgrillo 2y ago
Yeah I've definitely been saved a bunch of times by long retention, and the BI questions that might arise are impossible to predict. So some sort of retention is definitely necessary, IME. But let's take the case of metrics as an example--do we need full sample granularity for "old" data? Do we need full tag cardinality? Sample granularity reduction could be done with a transform to rollups at a coarser time granularity. That's a 60x reduction going from Hz to 1/min. You might lose a bunch of frequency information this way, but maybe that's ok? Numbers are really nice in ways that text is not.

zug_zug 2y ago

I think most places don't collect enough telemetry in the right formats. It's also possible they collect too much in the wrong formats. But the ability to vet a hypothesis (I bet our users are confused about feature X, which we can test by looking at how many times they go to page X, then Y, then X again in 30 second window) in an hour versus 2 sprints is vastly underappreciated/underutilized. I feel like this article paints with too broad a brush.

yetanotherdood 2y ago

> For 30 years how telemetry is produced has not changed: we define all of the data points that we need ahead of time and ship them out of the origin process, typically at large expense. If we apply the control plane / data plane split to observability telemetry production we can fundamentally change the status quo for the first time in three decades Has Matt read any prior art in this field? https://research.google/pubs/monarch-googles-planet-scale-in-memory-time-series-database/ https://research.google/pubs/monarch-googles-planet-scale-in...