5 ms·

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/1344068132978221056 https://twitter.com/richardstartin/status/134406813297822105...

Animats 6y ago
"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?
- deleted 6y ago
  [deleted]
- brown9-2 6y ago
  think of the type of monitoring/observability data that Datadog's agent collects
- viraptor 6y ago
  People tend to jump from telemetry straight to snooping on users even though it's just one possible case. A possible megabyte of data: A rolling in-memory debug-level log + stack traces with context + timing information, that you snapshot when the app catches and exception. Or a buffered set of timing stats from a minute of runtime. The author mentioned implementing ddsketch for probabilistic quantile distribution. That's something you run on your own app/infrastructure rather than for snooping on users.
- sebastos 6y ago
  This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.
  - sofixa 6y ago
    > Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Did you even read the post? It's obvious OP works at Datadog ( they mention it and mention the work is for their employer), so telemetry as in OpenTelemetry ( the new standard around metrics, traces and logs).
    - sebastos 6y ago
      Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry library needs to be lightweight to avoid perturbing the application; inefficient diagnostic tools are self-defeating. Unlike other formats, nested Protobuf messages cannot be written contiguously into a stream without significant buffering. The post doesn’t argue to never use Protobuf, but that the trade-off made by the wire-format itself, as opposed to any existing implementation, is unlikely to work for lightweight message senders. My field is related to robotics, where we actually send, you know, telemetry. Like, over a radio. What I'm complaining about is that this is arguably the "main" usage of this word, but people in the web space assume that like protobuf, the concept of telemetry needs no introduction: it means sending back statistics, metrics and logs about your web services. When he says "a good telemetry library" needs to be lightweight, there's zero preamble here to get me oriented. You saying "did you even read the post" is exactly missing the point. Yes, I read through the post, and through context clues, discerned that this post isn't relevant to my interests. I just notice that people writing about cloud service technology never feel compelled to warn you that's what they're going to be talking about. They make grand proclamations, and you have to play detective to find out whether their claims apply to you. "Microservices are the future!" "Okay, does he mean like... for all of software? Or for SaaS companies?" Imagine if the world was a different place, and the average programmer worked at a robotics company instead of a web tech company. And then you saw an article titled "gRPC is dead - DDS is the future". And then it jumped in and started talking about all these throughput metrics, and it took you 6 paragraphs to figure out that the reason they're done with gRPC is that they're shipping around 4K images at 60fps to dozens of microservices that all run on the same computer. And you'd go oh, well that's... irrelevant. That's how it feels, all the time.
kevincox 6y ago
If you are expecting multi-MB messages protobuf isn't the best option as the author highlighted. Often you can combat this by breaking a large message into a stream of smaller messages but for larger amounts of data you will want something else.
cbsmith 6y ago
So, essentially, they're shoving a square peg into a round hole, and are surprised there's a bit of an air gap.

Animats 6y ago

"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

deleted 6y ago
[deleted]
brown9-2 6y ago
think of the type of monitoring/observability data that Datadog's agent collects
viraptor 6y ago
People tend to jump from telemetry straight to snooping on users even though it's just one possible case. A possible megabyte of data: A rolling in-memory debug-level log + stack traces with context + timing information, that you snapshot when the app catches and exception. Or a buffered set of timing stats from a minute of runtime. The author mentioned implementing ddsketch for probabilistic quantile distribution. That's something you run on your own app/infrastructure rather than for snooping on users.
sebastos 6y ago
This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.
- sofixa 6y ago
  > Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Did you even read the post? It's obvious OP works at Datadog ( they mention it and mention the work is for their employer), so telemetry as in OpenTelemetry ( the new standard around metrics, traces and logs).
  - sebastos 6y ago
    Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry library needs to be lightweight to avoid perturbing the application; inefficient diagnostic tools are self-defeating. Unlike other formats, nested Protobuf messages cannot be written contiguously into a stream without significant buffering. The post doesn’t argue to never use Protobuf, but that the trade-off made by the wire-format itself, as opposed to any existing implementation, is unlikely to work for lightweight message senders. My field is related to robotics, where we actually send, you know, telemetry. Like, over a radio. What I'm complaining about is that this is arguably the "main" usage of this word, but people in the web space assume that like protobuf, the concept of telemetry needs no introduction: it means sending back statistics, metrics and logs about your web services. When he says "a good telemetry library" needs to be lightweight, there's zero preamble here to get me oriented. You saying "did you even read the post" is exactly missing the point. Yes, I read through the post, and through context clues, discerned that this post isn't relevant to my interests. I just notice that people writing about cloud service technology never feel compelled to warn you that's what they're going to be talking about. They make grand proclamations, and you have to play detective to find out whether their claims apply to you. "Microservices are the future!" "Okay, does he mean like... for all of software? Or for SaaS companies?" Imagine if the world was a different place, and the average programmer worked at a robotics company instead of a web tech company. And then you saw an article titled "gRPC is dead - DDS is the future". And then it jumped in and started talking about all these throughput metrics, and it took you 6 paragraphs to figure out that the reason they're done with gRPC is that they're shipping around 4K images at 60fps to dozens of microservices that all run on the same computer. And you'd go oh, well that's... irrelevant. That's how it feels, all the time.

kevincox 6y ago

If you are expecting multi-MB messages protobuf isn't the best option as the author highlighted. Often you can combat this by breaking a large message into a stream of smaller messages but for larger amounts of data you will want something else.

cbsmith 6y ago

So, essentially, they're shoving a square peg into a round hole, and are surprised there's a bit of an air gap.