3 ms·

Imagine a metric firing on something that happens a 1000 times a millisecond. Imagine a metric firing in a way that grabs a mutex at an inopportune time that cr

Imagine a metric firing on something that happens a 1000 times a millisecond. Imagine a metric firing in a way that grabs a mutex at an inopportune time that creates a deadlock. Imagine a metric firing that suddenly spams your metrics server with data that nobody except you cares about in this one temporal moment. There are so many reasons this is a bad idea. If you must do it, do so locally but there are so many easier ways to determine "if something is used."

yazaddaruvala 4y ago
If adding a few counters to your code base cause any of these issues, your current team is already dysfunctional and will get more productive my improving the system. eg “Metrics server goes down” metrics should be sent pre-aggregated, and in batches, even possibly polled for. Having 10 metrics that fire 10^N times a second shouldn’t impact the metrics server, ever! Metrics servers are impacted by cardinality, so yes don’t create 10^N unique metrics. eg “something that fires 1000times a millisecond” if the rest of the team doesn’t know where their hot-loops are and has them commented or caught in a code review, it’s very likely someone will add some logic to it that will hurt. The metric is just one such change, and might as well find out sooner than later. If it will cripple your service, that is why CI/CD encourages canary deployments / other automated performance testing. eg “metric causes a deadlock” - umm, this is just too contrived. I’ve never used or heard of any metric library that was built this way. This type of metrics client would cripple even a very experienced developer on that team. Summary, if adding a metric can cripple your service or team, reconsider priorities. It will speed you up even in the medium term.
- ninepoints 4y ago
  I guess every game I've ever shipped was shipped on a dysfunctional team!
  - yazaddaruvala 4y ago
    I don't mean to diminish any of your achievements. Shipping amazing products is orthogonal to creating a robust codebase/system. I also don't mean to be negative towards you or your experience. Instead, I'll pose the question to you: What would you call a team/system that describes itself as "at risk of being crippled by a single line change like adding a metric"? At the end of the day, the product is the only thing that matters and you can be and should be proud of the products you've launched. That said, are you proud of the way you and your team built them? Have you or anyone on your team proclaimed it was a joy to build/improve that product? idk... after all I'm just some guy on the internet. Best of wishes to you! I truly hope you did not, do not, and never do work on a dysfunctional team/system.
    - ninepoints 4y ago
      Recording a metric is not a cheap operation in many contexts. I work with code we typically instrument at a microsecond to millisecond granularity. Sampling profilers typically record instruction samples at most every 200 us or so, so adding instrumentation at a finer granularity than this has a dramatic effect on performance. If a game operates at 60 fps, adding a metric at an inopportune time will render the game unplayable. No, I haven't been on teams that dysfunctional. These restrictions are born out of necessity. My point was that you seem to have a somewhat narrow view on software as a whole, and you are extrapolating from your personal experience general advice that simply doesn't apply.
12345hn6789 4y ago
This is not what the post was talking about though, if you have a service that truly is being fired 1000 things a millisecond, I can almost guarantee you have other metrics already up, and thus won't need anything else.
- ninepoints 4y ago
  I work with realtime graphics. Thousands of things are indeed happening every millisecond and I sure wouldn't want a new hire blindly instrumenting those things just to "see if they happen." There are appropriate tools to measure performance at this granularity.

yazaddaruvala 4y ago

If adding a few counters to your code base cause any of these issues, your current team is already dysfunctional and will get more productive my improving the system. eg “Metrics server goes down” metrics should be sent pre-aggregated, and in batches, even possibly polled for. Having 10 metrics that fire 10^N times a second shouldn’t impact the metrics server, ever! Metrics servers are impacted by cardinality, so yes don’t create 10^N unique metrics. eg “something that fires 1000times a millisecond” if the rest of the team doesn’t know where their hot-loops are and has them commented or caught in a code review, it’s very likely someone will add some logic to it that will hurt. The metric is just one such change, and might as well find out sooner than later. If it will cripple your service, that is why CI/CD encourages canary deployments / other automated performance testing. eg “metric causes a deadlock” - umm, this is just too contrived. I’ve never used or heard of any metric library that was built this way. This type of metrics client would cripple even a very experienced developer on that team. Summary, if adding a metric can cripple your service or team, reconsider priorities. It will speed you up even in the medium term.

ninepoints 4y ago
I guess every game I've ever shipped was shipped on a dysfunctional team!
- yazaddaruvala 4y ago
  I don't mean to diminish any of your achievements. Shipping amazing products is orthogonal to creating a robust codebase/system. I also don't mean to be negative towards you or your experience. Instead, I'll pose the question to you: What would you call a team/system that describes itself as "at risk of being crippled by a single line change like adding a metric"? At the end of the day, the product is the only thing that matters and you can be and should be proud of the products you've launched. That said, are you proud of the way you and your team built them? Have you or anyone on your team proclaimed it was a joy to build/improve that product? idk... after all I'm just some guy on the internet. Best of wishes to you! I truly hope you did not, do not, and never do work on a dysfunctional team/system.
  - ninepoints 4y ago
    Recording a metric is not a cheap operation in many contexts. I work with code we typically instrument at a microsecond to millisecond granularity. Sampling profilers typically record instruction samples at most every 200 us or so, so adding instrumentation at a finer granularity than this has a dramatic effect on performance. If a game operates at 60 fps, adding a metric at an inopportune time will render the game unplayable. No, I haven't been on teams that dysfunctional. These restrictions are born out of necessity. My point was that you seem to have a somewhat narrow view on software as a whole, and you are extrapolating from your personal experience general advice that simply doesn't apply.

12345hn6789 4y ago

This is not what the post was talking about though, if you have a service that truly is being fired 1000 things a millisecond, I can almost guarantee you have other metrics already up, and thus won't need anything else.

ninepoints 4y ago
I work with realtime graphics. Thousands of things are indeed happening every millisecond and I sure wouldn't want a new hire blindly instrumenting those things just to "see if they happen." There are appropriate tools to measure performance at this granularity.