3 ms·
Spider isn't anything like the queries that analysts and data scientists write against DBs. I don't think it even has many joins. Do you not have access to a mo
by iandanforth 3y ago
Spider isn't anything like the queries that analysts and data scientists write against DBs. I don't think it even has many joins. Do you not have access to a more realistic training set?
- ainesh93 3y agoAlthough Spider is better known in the text-to-SQL world, you're right that BiRD may provide a better testing ground. Comparing against the current leaderboard on that standard is on the docket!
- tmostak 3y agoI agree that Spider queries are not necessarily representative of the SQL you might see in the wild from real users, but looking at some analysis I did of the dataset around 43% of the queries had joins, and a number had 3, 4, or 5-way joins.
- philipodonnell 3y agoAgree that it’s not representative of real world queries, but I think it’s more of a “measure progress against a simple but consistent baseline” and SOTA is still struggling so it’s clearly not ready for real queries. Most of the papers on this topic mention a lack of large volumes of high quality training data… Spider is probably the best public one right now.