What we did (and continue to do) in the TechEmpower Framework Benchmarks [1] is publish a set of requirements for our test types, provide a small set of example

What we did (and continue to do) in the TechEmpower Framework Benchmarks [1] is publish a set of requirements for our test types, provide a small set of example implementations on mainstream frameworks, and then solicit input and contributions from the open source community. Fast forward to today, and we'll be at 5,000 pull requests processed soon, and receive many from framework maintainers or advocates. Other test implementations are maintained by broader fans of languages. Obviously there are an unbounded set of ways people can test the boundaries or contribute implementations that aren't in the spirit of the project. We've had to deal with several contributions that required push back or removal. Since we run our benchmarks project in our spare time, we rely heavily on the community to help us keep an eye on implementations, and also help determine where to set boundaries when they require clarification. For example, is pipelining in the Postgres driver acceptable given we disallowed batch querying? We ultimately decided it was acceptable because it's leaving the ergonomics of application development unchanged. I feel that overall, the project has been effective in large part because the community has done a good job of self-policing. [1] https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/