On one hand, I think it would be useful to have a couple of benchmarks for super common tech stacks like Python, React, etc. that allowed comparing variants of

On one hand, I think it would be useful to have a couple of benchmarks for super common tech stacks like Python, React, etc. that allowed comparing variants of rules to find the best ones. On the other hand, if something can be turned into a benchmark then it can probably be learned in the weights of a model. A lot of the benefits of rules are completely unique to your situation and the definition of “better” is likely to differ between people. I think the videos above are the best way I currently have to display this. There are potential solutions though to measure improvement, and we actually made it possible for you to obtain the necessary data. If you add a “data” block to your custom assistant you can capture all of the accept/reject, thumbs up/down, etc. data and use it to do an analysis. We definitely will be working more in that direction