4 ms·

Oh hey, this is me. The background is that I've thought about applying SPC to software behavior for a long time[0]. While I was at Shopify I began to think abo

Oh hey, this is me. The background is that I've thought about applying SPC to software behavior for a long time[0]. While I was at Shopify I began to think about using it for detecting regressions in the YJIT benchmark suite[1]. My first thought was to code a Ruby library (Shopify is a Ruby on Rails outfit), but I decided that it would be more accessible to more folks if it worked inside a database instead. I got caught in a 20% layoff at Shopify and that somewhat took the wind out of my sails, which is why it's been dormant for a year. There are some limitations to be aware of in a software context. First, sample sizes have to be constant. In a world of unreliable systems and networks this is a pretty tough constraint. It could be achieved by taking fixed subsamples at random from a varying larger sample (eg, normally we get 100, pick 50 at random). Switching to variable sample size would be possible, but it will be a lot of work and I haven't had the motivation to tackle it yet. It's also only lightly treated in the main textbook I've worked from, because the usual focus is manufacturing operations where fixed sample sizes are common. Second, it doesn't deal with non-parametric distributions. Classical SPC is rooted in the Normal distribution, and continues to work quite well with any distribution that has some kind of centrally-located mass that tapers off to the edges. But a lot of software behavior follows power laws, especially Pareto distributions. There is SPC literature to deal with non-normal distributions using non-parametric statistics. But I haven't bought and digested the relevant books. It's worth noting that SPC is as much a system of problem solving as it is a bunch of statistical tools. Most major metrics vendors now offer some kind of anomaly detection, but then what? What are your out-of-control action plans? What is your concept of an acceptable amount of staying close to the mean? What is your target, and why is it your target? Can you even achieve such a target with the system as-is, or are you dreaming? SPC attends to these questions as well. I can definitely recommend kqr's introduction[2], which is linked from the repo. [0] https://theoryof.predictable.software/articles/what-is-predictable-software/ https://theoryof.predictable.software/articles/what-is-predi... [1] https://speed.yjit.org https://speed.yjit.org [2] https://two-wrongs.com/statistical-process-control-a-practitioners-guide.html https://two-wrongs.com/statistical-process-control-a-practit...

jacques_chester 3y ago
I might add that if you're wondering about how the SPC tools work, read the comments in the SQL. I deliberately set out to write comments that would be helpful both to experienced SPC folks and to curious beginners. Unfortunately the GitHub Linguist grammar for SQL doesn't recognize PostgreSQL comments, so highlighting will be wonky in places.
OJFord 3y ago
Before I noticed your readme reading list I found: https://r-bar.net/xmr-control-chart-tutorial-examples https://r-bar.net/xmr-control-chart-tutorial-examples which I think I preferred (having now read both that and [2]) for being a bit quicker to the point since I was already imagining things I might measure so didn't need the motivating examples. (SPC completely new to me - a bit difficult to search for information about since it seems popular with management consulting, six sigma certs, and the like.) I'd also be interested in applying this to software for better metrics, just thinking about how I might be able to get somewhere quickly in an innovation/hackathon days (aside from your project) came across https://grafana.com/grafana/plugins/kensobi-spc-panel/ https://grafana.com/grafana/plugins/kensobi-spc-panel/ - looks like it doesn't actually do any calculation, you have to feed it your measurements & sub-group sample size etc. but could be handy in conjunction with yours.
- jacques_chester 3y ago
  The XmR is in some ways the best beginner's plot because it's so immediately tied to the data. The other Shewhart charts work with samples, which is enough indirection to cause some mild confusion on first encounter (for me at least). That Grafana plugin looks awesome. If I can shave off some time from endless leetcode grinding I might play with it.
  - OJFord 3y ago
    Ha, since I found sample based examples before that article I was actually confused in the other direction when I read it. (Even more so when it links the table of constants and says that's the d2 value for n=2... I assume n=2 is the sub-group size, but in XmR as described it seems to correspond to the moving average/'range' of two (n=1 sized) samples?) Do you have a recommendation that discusses the various charts so I can better understand from XmR to n=8 'x bar'/'x bar R' etc. (I think I'm using the terminology correctly enough - where as I understand it we have n=8 xs in each time sample, the mean of which (i.e. x bar) is what we plot and then look at the moving average/'range' of)?

jacques_chester 3y ago

I might add that if you're wondering about how the SPC tools work, read the comments in the SQL. I deliberately set out to write comments that would be helpful both to experienced SPC folks and to curious beginners. Unfortunately the GitHub Linguist grammar for SQL doesn't recognize PostgreSQL comments, so highlighting will be wonky in places.

OJFord 3y ago

Before I noticed your readme reading list I found: https://r-bar.net/xmr-control-chart-tutorial-examples https://r-bar.net/xmr-control-chart-tutorial-examples which I think I preferred (having now read both that and [2]) for being a bit quicker to the point since I was already imagining things I might measure so didn't need the motivating examples. (SPC completely new to me - a bit difficult to search for information about since it seems popular with management consulting, six sigma certs, and the like.) I'd also be interested in applying this to software for better metrics, just thinking about how I might be able to get somewhere quickly in an innovation/hackathon days (aside from your project) came across https://grafana.com/grafana/plugins/kensobi-spc-panel/ https://grafana.com/grafana/plugins/kensobi-spc-panel/ - looks like it doesn't actually do any calculation, you have to feed it your measurements & sub-group sample size etc. but could be handy in conjunction with yours.

jacques_chester 3y ago
The XmR is in some ways the best beginner's plot because it's so immediately tied to the data. The other Shewhart charts work with samples, which is enough indirection to cause some mild confusion on first encounter (for me at least). That Grafana plugin looks awesome. If I can shave off some time from endless leetcode grinding I might play with it.
- OJFord 3y ago
  Ha, since I found sample based examples before that article I was actually confused in the other direction when I read it. (Even more so when it links the table of constants and says that's the d2 value for n=2... I assume n=2 is the sub-group size, but in XmR as described it seems to correspond to the moving average/'range' of two (n=1 sized) samples?) Do you have a recommendation that discusses the various charts so I can better understand from XmR to n=8 'x bar'/'x bar R' etc. (I think I'm using the terminology correctly enough - where as I understand it we have n=8 xs in each time sample, the mean of which (i.e. x bar) is what we plot and then look at the moving average/'range' of)?