3 ms·

Okay, wait this is actually doing a really good job. I still have to ask follow up questions to get reasonable results but when I tested earlier this year it w

Okay, wait this is actually doing a really good job. I still have to ask follow up questions to get reasonable results but when I tested earlier this year it was outright failing on most of my test queries.

rushingcreek 2y ago
That’s great to hear! What are you asking it?
- CSMastermind 2y ago
  These are the types of questions I want to ask it: > What degrees are held by each of the current Fortune 100 CEOs? > What job did each of the current NFL GMs hold before their current position? > Which genre would each of the current Billboard Hot 100 songs be considered part of? > How many recipients of the Presidential Medal of Freedom were born outside of the US? > Which US car company has the most models in their 2025 line-up across all of their brands? It can't handle those directly right now. You need to break the problem down step by step and sort of walk it through gathering the data with follow up questions. But much better than it used to be.
  - druskacik 2y ago
    There is a method that could help immensely when answering questions like these. E.g. some of these question may be answered quite quickly using WikiData [0] (answer to question about the recipients of Medal of Freedom, query written with the help of Claude), instead of just scraping and compiling information from potentially hundreds of websites. I believe this idea is quite under-explored compared to just blindly putting everything to the model's context. [0] https://query.wikidata.org/#SELECT%20%28COUNT%28DISTINCT%20%3Fperson%29%20AS%20%3Fcount%29%20%3FcountryLabel%0AWHERE%20%7B%0A%20%20%3Fperson%20wdt%3AP166%20wd%3AQ17144%20.%20%20%23%20Person%20received%20Presidential%20Medal%20of%20Freedom%0A%20%20%3Fperson%20wdt%3AP19%20%3FbirthPlace%20.%20%20%23%20Person%27s%20place%20of%20birth%0A%20%20%3FbirthPlace%20wdt%3AP17%20%3Fcountry%20.%20%23%20Country%20of%20birth%20place%0A%20%20%0A%20%20FILTER%28%3Fcountry%20%21%3D%20wd%3AQ30%29%20%20%20%20%23%20Filter%20out%20United%20States%20%28Q30%29%0A%20%20%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0AGROUP%20BY%20%3FcountryLabel%0AORDER%20BY%20DESC%28%3Fcount%29 https://query.wikidata.org/#SELECT%20%28COUNT%28DISTINCT%20%...
    - kreyenborgi 2y ago
      Yeah, I've used gpt to create wikidata queries for me, it worked great :-)
  - rushingcreek 2y ago
    Thanks! Have you tried enabling “multi-query” mode in the search box?
  - pseudosavant 2y ago
    Part of the problem will not be solved by LLMs but maybe hiding aspects of one running. LLMs basically "think" "out loud" as it processes and produces tokens. The amount of thought required to answer any of those questions is pretty high, especially because they are all sizeable lists. It is going to take a lot of thinking out loud, and detailed training data covering all those items, to do that well.

rushingcreek 2y ago

That’s great to hear! What are you asking it?

CSMastermind 2y ago
These are the types of questions I want to ask it: > What degrees are held by each of the current Fortune 100 CEOs? > What job did each of the current NFL GMs hold before their current position? > Which genre would each of the current Billboard Hot 100 songs be considered part of? > How many recipients of the Presidential Medal of Freedom were born outside of the US? > Which US car company has the most models in their 2025 line-up across all of their brands? It can't handle those directly right now. You need to break the problem down step by step and sort of walk it through gathering the data with follow up questions. But much better than it used to be.
- druskacik 2y ago
  There is a method that could help immensely when answering questions like these. E.g. some of these question may be answered quite quickly using WikiData [0] (answer to question about the recipients of Medal of Freedom, query written with the help of Claude), instead of just scraping and compiling information from potentially hundreds of websites. I believe this idea is quite under-explored compared to just blindly putting everything to the model's context. [0] https://query.wikidata.org/#SELECT%20%28COUNT%28DISTINCT%20%3Fperson%29%20AS%20%3Fcount%29%20%3FcountryLabel%0AWHERE%20%7B%0A%20%20%3Fperson%20wdt%3AP166%20wd%3AQ17144%20.%20%20%23%20Person%20received%20Presidential%20Medal%20of%20Freedom%0A%20%20%3Fperson%20wdt%3AP19%20%3FbirthPlace%20.%20%20%23%20Person%27s%20place%20of%20birth%0A%20%20%3FbirthPlace%20wdt%3AP17%20%3Fcountry%20.%20%23%20Country%20of%20birth%20place%0A%20%20%0A%20%20FILTER%28%3Fcountry%20%21%3D%20wd%3AQ30%29%20%20%20%20%23%20Filter%20out%20United%20States%20%28Q30%29%0A%20%20%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0AGROUP%20BY%20%3FcountryLabel%0AORDER%20BY%20DESC%28%3Fcount%29 https://query.wikidata.org/#SELECT%20%28COUNT%28DISTINCT%20%...
  - kreyenborgi 2y ago
    Yeah, I've used gpt to create wikidata queries for me, it worked great :-)
- rushingcreek 2y ago
  Thanks! Have you tried enabling “multi-query” mode in the search box?
- pseudosavant 2y ago
  Part of the problem will not be solved by LLMs but maybe hiding aspects of one running. LLMs basically "think" "out loud" as it processes and produces tokens. The amount of thought required to answer any of those questions is pretty high, especially because they are all sizeable lists. It is going to take a lot of thinking out loud, and detailed training data covering all those items, to do that well.