Has anyone attempted to create a "search engine benchmark" of, say, oddly specific phrases that almost certainly need to return a certain set of pages as the to

Has anyone attempted to create a "search engine benchmark" of, say, oddly specific phrases that almost certainly need to return a certain set of pages as the top result? This could quantify the loss in quality we keep talking about. If the full list were public then the results could of course be gamed, but if only a sample were public and the full set were private then it would probably provide a useful metric (and would be able to measure attempted gaming by the score on the public and private datasets).