Agree that it’s not representative of real world queries, but I think it’s more of a “measure progress against a simple but consistent baseline” and SOTA is sti

Agree that it’s not representative of real world queries, but I think it’s more of a “measure progress against a simple but consistent baseline” and SOTA is still struggling so it’s clearly not ready for real queries. Most of the papers on this topic mention a lack of large volumes of high quality training data… Spider is probably the best public one right now.