3 ms·
It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale i
by cpascal 6y ago
It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale is sufficient, a “micro” optimization like this is actually a “macro” optimization. Also the author of this article works at DataDog and I suspect the number of messages they process each second falls under the sufficient category.
For example, suppose you are processing 1M messages per second and you can shave 1 byte off the message size, that shaves off 1MB/sec of data that needs to be processed. If you’re paying for network bandwidth or storing the messages, that saves you something like 2.6TB of data each month.
2.6TB/month is not likely to be a huge deal when it comes to cost savings, but if you keep scaling the messages/sec or the bytes/msg you can start to get some significant savings.
Now I used message size as an example, and the article focuses on processing time not message size, but the point still stands. When you can make a micro optimization for something that is done a very large number of times, there are not-insignificant gains to be had.
- nijave 6y agoI think this was for client/application instrumentation where they want to try to have little-to-no overhead for including the datadog agent that ships telemetry back to Datadog's service
- Thaxll 6y agoIf you manage scale in the milion messages/sec you really don't care about some TB/month. Most compagnies are still using REST/json in places where a binary protocol "would be better".