Surely not "useless" but it doesn't reflect the way speech recognition is used today. Unless you're routinely listening in on two humans having a phone call con

Surely not "useless" but it doesn't reflect the way speech recognition is used today. Unless you're routinely listening in on two humans having a phone call conversation that they don't expect a computer to be hearing, which is what the test set actually contains. If you looked at modern performance on test sets from the 1980s (like Resource Management or TIDIGITS) you might be under the impression that we'd achieved human-level accuracy levels years ago, but we clearly haven't. And similarly, what users expect from speech recognition today is in many ways much more demanding than it was in 2000: vocabularies are huge (think about all the words you could say to Google), latency needs be very low, and no one thinks it's acceptable to require users to perform enrollment any more. So yes, just like other benchmarks, we should retire them after a few years. The fact that a modern computer could get 100,000 FPS on a video game from 2000 wouldn't be considered a "milestone."