4 ms·

Along with that, a portion of the content of Wikipedia is already available in structured assertion form, thanks to DBPedia[1] and Wikidata[2]. I don't know the

Along with that, a portion of the content of Wikipedia is already available in structured assertion form, thanks to DBPedia[1] and Wikidata[2]. I don't know the exact percentage, but it's a starting point at the very least. [1]: https://www.dbpedia.org/ https://www.dbpedia.org/ [2]: https://www.wikidata.org/wiki/Wikidata:Main_Page https://www.wikidata.org/wiki/Wikidata:Main_Page

throw310822 2y ago
I've tried to use ChatGPT to produce wikidata queries- it sounds like a great combination. Unfortunately it's pretty hard to make it produce valid queries, and to find the wikidata documentation to teach it.
- mindcrime 2y ago
  My interest is a little more like flipping that around the other way. Take output from the LLM, get it into structured assertion form, and then use those assertions as part of a query / inference process which pulls in other "known good" (or "ground truth" if you will) assertions from sources like DBPedia or Wikidata. The idea being to either verify the LLM output, or possibly to extend it using inferred conclusions. I think the way I think about it is somewhat akin to what AWS are doing here, where they talk about using automated reasoning to reduce hallucinations from LLM's: https://aws.amazon.com/blogs/machine-learning/reducing-hallucinations-in-large-language-models-with-custom-intervention-using-amazon-bedrock-agents/ https://aws.amazon.com/blogs/machine-learning/reducing-hallu...
  - kziemski 2y ago
    Was looking at prolog and having it shadow the llm activity in order to flag whenever the conversation trips.
- binarymax 2y ago
  If you can express the DSL SPARQL query as a transformation to JSON Schema, you can have GPT give you the JSON predictably in strict mode, and then reverse the transformation.
  - petesergeant 2y ago
    Have you done this and had it working reliably for you in practice?
    - binarymax 2y ago
      Not for SPARQL, no. But I have done it for other query DSLs like SQL and Opensearch.
      - petesergeant 2y ago
        Fair enough. Any particular tricks that helped you get valid queries out of it the first time?
        binarymax 2y ago
        Think of it like this: don't ask it to construct the query, ask it for the parts of the query, in a predefined format. Then populate one of many pre-defined templates with the values. For example, you might have ten types of queries. Each query will have parameters and a classification/description. Get the classification, and get the parameters, and stick them into your respective template. This gives you way more safety and predictability. I've gone through the whole gamut to do this since first using the GPT's API in early 2023. As first I was asking for HTML with few-shot since getting reliable JSON was impossible. Then I switched to function calling with JSON mode when that was released later in 2023, but still with fallbacks because sometimes it would spit out markdown with JSON inside, and other fun mistakes. Finally summer 2024 strict mode became available which "guarantees" a well-formed JSON response corresponding to a given schema. I've run hundreds of thousands of queries through that and have not had a formatting issue (yet).
        petesergeant 2y ago
        Very smart approach, thanks
  - 2y ago
- petesergeant 2y ago
  Very interesting that you’ve done this! I’ve read a few times about people talking about knowledge bases and LLMs and there seems a lot of hand waving around “and at this point the LLM produces the SPARQL query” which feels very much “now draw the rest of the owl” to me

throw310822 2y ago

I've tried to use ChatGPT to produce wikidata queries- it sounds like a great combination. Unfortunately it's pretty hard to make it produce valid queries, and to find the wikidata documentation to teach it.

mindcrime 2y ago
My interest is a little more like flipping that around the other way. Take output from the LLM, get it into structured assertion form, and then use those assertions as part of a query / inference process which pulls in other "known good" (or "ground truth" if you will) assertions from sources like DBPedia or Wikidata. The idea being to either verify the LLM output, or possibly to extend it using inferred conclusions. I think the way I think about it is somewhat akin to what AWS are doing here, where they talk about using automated reasoning to reduce hallucinations from LLM's: https://aws.amazon.com/blogs/machine-learning/reducing-hallucinations-in-large-language-models-with-custom-intervention-using-amazon-bedrock-agents/ https://aws.amazon.com/blogs/machine-learning/reducing-hallu...
- kziemski 2y ago
  Was looking at prolog and having it shadow the llm activity in order to flag whenever the conversation trips.
binarymax 2y ago
If you can express the DSL SPARQL query as a transformation to JSON Schema, you can have GPT give you the JSON predictably in strict mode, and then reverse the transformation.
- petesergeant 2y ago
  Have you done this and had it working reliably for you in practice?
  - binarymax 2y ago
    Not for SPARQL, no. But I have done it for other query DSLs like SQL and Opensearch.
    - petesergeant 2y ago
      Fair enough. Any particular tricks that helped you get valid queries out of it the first time?
      - binarymax 2y ago
        Think of it like this: don't ask it to construct the query, ask it for the parts of the query, in a predefined format. Then populate one of many pre-defined templates with the values. For example, you might have ten types of queries. Each query will have parameters and a classification/description. Get the classification, and get the parameters, and stick them into your respective template. This gives you way more safety and predictability. I've gone through the whole gamut to do this since first using the GPT's API in early 2023. As first I was asking for HTML with few-shot since getting reliable JSON was impossible. Then I switched to function calling with JSON mode when that was released later in 2023, but still with fallbacks because sometimes it would spit out markdown with JSON inside, and other fun mistakes. Finally summer 2024 strict mode became available which "guarantees" a well-formed JSON response corresponding to a given schema. I've run hundreds of thousands of queries through that and have not had a formatting issue (yet).
        petesergeant 2y ago
        Very smart approach, thanks
- 2y ago
petesergeant 2y ago
Very interesting that you’ve done this! I’ve read a few times about people talking about knowledge bases and LLMs and there seems a lot of hand waving around “and at this point the LLM produces the SPARQL query” which feels very much “now draw the rest of the owl” to me