Removed LoC should count double. Edit: But then again, the point is not about individual examples really – the point is that whichever metric you choose, with

Removed LoC should count double. Edit: But then again, the point is not about individual examples really – the point is that whichever metric you choose, with time you'll see diminishing returns followed by negative ones. In case of doubling removals, you can easily game it by dumping json files for tests, then removing them ie. in favor of generator etc. What's interesting is universality of this phenomenon (strong goodhart's law?) – overfitting in llms, using metrics discussed here and why it makes sense to vote on opposite ruling party etc.