The quote matches my own understanding of the system from when I worked on Google Search circa 2005. Human ratings were used to facilitate an automated test tha

The quote matches my own understanding of the system from when I worked on Google Search circa 2005. Human ratings were used to facilitate an automated test that could tell you if your new algorithm actually made search results better or not. That let search quality engineers iterate on ideas quickly. But the human ratings were not directly used to produce search results, since that wouldn't scale (only a relatively small number of queries actually had rated results). Obviously 2005 was a long time ago though and everything could have changed since then.