4 ms·

My stack lately has been Next.js with Prettier and very strict lint, type, and complexity gates, plus opinionated tests that aim for real signal over mere cover

My stack lately has been Next.js with Prettier and very strict lint, type, and complexity gates, plus opinionated tests that aim for real signal over mere coverage theatre. I want to find the time to fine-tune GPT-4o using before/after examples of code that fails the gates and then passes them. The hope being it generates gate-compliant code on the first try much more often, which is cheaper and more reliable than relying on a base model with brute-force retries. I think this might also align with research showing that grounding models in execution feedback yields order-of-magnitude gains in sample efficiency and improved code quality, not just speed [0][1]. References [0] https://arxiv.org/abs/2307.04349 https://arxiv.org/abs/2307.04349 [1] https://arxiv.org/abs/2410.02089 https://arxiv.org/abs/2410.02089

pc86 1y ago
What does "real signal" mean in this context? I think (hope?) most people agree test coverage for the sake of coverage isn't particularly helpful but it's not clear to me how an "opinionated test" would differ from the tests you'd find in your standard "we want 100% coverage" projects.
- mkw5053 1y ago
  I am trying to capture the idea that opinionated tests assert invariants and contracts at integration boundaries, while coverage-driven suites often spend effort on shallow checks. The goal is fewer but stronger tests that actually prevent regressions. Here is one attempt at defining a TESTING.md that I pass to Claude Code [0]. Honestly, it is different from what I now use in other projects. For example, I have found that mutation tests are rarely worth the added complexity. I have not yet found a good way to enforce a deterministic gate that only permits “good” tests, so I settle on defining expectations in a markdown file that I pass to whichever coding agent I am using. [0] https://github.com/Airbolt-AI/airbolt/blob/main/TESTING.md https://github.com/Airbolt-AI/airbolt/blob/main/TESTING.md

pc86 1y ago

What does "real signal" mean in this context? I think (hope?) most people agree test coverage for the sake of coverage isn't particularly helpful but it's not clear to me how an "opinionated test" would differ from the tests you'd find in your standard "we want 100% coverage" projects.

mkw5053 1y ago
I am trying to capture the idea that opinionated tests assert invariants and contracts at integration boundaries, while coverage-driven suites often spend effort on shallow checks. The goal is fewer but stronger tests that actually prevent regressions. Here is one attempt at defining a TESTING.md that I pass to Claude Code [0]. Honestly, it is different from what I now use in other projects. For example, I have found that mutation tests are rarely worth the added complexity. I have not yet found a good way to enforce a deterministic gate that only permits “good” tests, so I settle on defining expectations in a markdown file that I pass to whichever coding agent I am using. [0] https://github.com/Airbolt-AI/airbolt/blob/main/TESTING.md https://github.com/Airbolt-AI/airbolt/blob/main/TESTING.md