5 ms·
Why RDF is the natural knowledge layer for AI systems
- never_inline 1y agoSo so obvious slop and pains my eyes to read.
- Michael_Keller 1y ago[dead]
- flanked-evergl 1y agoRDF is great but it's somewhat inadvertently captured by academia. The tooling is not in a state where you can use it for any commercial or mission critical application. The tooling is mainly maintained by academics, and their concerns run almost exactly counter to normal engineering concerns. An engineer would rather have tooling with limited functionality that is well designed and behaves correctly without bugs. Academics would rather have tooling with lots of niche features, and they can tolerate poor design, incorrect behavior and bugs. They care more for features, even if they are incorrect, as they need to publish something "novel". The end result is that almost all things you find for RDF is academia quality and lots of it is abandoned because it was just part of publication spam being pumped and dumped by academics that need to publish or perish. Anyone who wants to use it commercially really has to start from scratch almost.
- jraph 1y ago> even if they are incorrect Uh. Do you have a source for this? Correctness is a major need in academia.
- tsimionescu 1y agoI think they mean things like a tool that has feature X even if it crashes 50% when it is used is preferable to a tool that doesn't have feature X at all.
- jraph 1y agoOk, makes sense, I hadn't read it like this. For me, "correct" means "provides correct results".
- rglullis 1y agoCorrect != Bug-free. My experience working with software developed by academics is that it is focused on getting the job done for a very small user base of people who are okay with getting their hands dirty. This means lots of workarounds, one-off scripts, zero regards for maintainability or future-proofing...
- lmm 1y ago> Correctness is a major need in academia. How so? Consider the famous result that most published research findings are false.
- jraph 1y agoHow so? Finding correct stuff is the whole point of research, no matter the extent at which it actually succeeds in reaching this. So yes, regardless on the actual results it is a major need in academia. We have nothing better anyway (which doesn't need it can't improve; we critically need it to improve). Now. I'll assume you are referring to "Why Most Published Research Findings Are False". This paper is 20 years old, only addresses medical research despite its title, and seems to have mixed reception [1] > Biostatisticians Jager and Leek criticized the model as being based on justifiable but arbitrary assumptions rather than empirical data, and did an investigation of their own which calculated that the false positive rate in biomedical studies was estimated to be around 14%, not over 50% as Ioannidis asserted.[12] Their paper was published in a 2014 special edition of the journal Biostatistics along with extended, supporting critiques from other statisticians 14% is a huge concern and I think nobody will disagree with this. But we are far from most, if this is true. [1] https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False#Reception https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
- ragebol 1y agoTooling sounds like it can be fixed? If the knowledge bases are useful, why not use them with better tools?
- philjohn 1y agoYes and no. I worked for a company that went hard into "Semantic Web" tech for libraries (as in, the places with books), using an RDF Quad Store for data storage (OpenLink Virtuoso) and structuring all data as triples - which is a better fit for the Heirarchical MARC21 format than a relational database. There are a few libraries (the software kind) out there that follow the W3 spec correctly, Redland being one of them.
- Zardoz84 1y agoI'm in a similar boat. On my case, it's software for public libraries, and it's a must having data accessible as RDF. Event, we have our own public fork of Marc4j .
- simonw 1y agoHow well did that work? Based on your experience at that company would you build a new project on the stack that they chose?
- philjohn 1y agoIt worked very well, as I mentioned, Marc21 (the interchange format for bibliographic data) is heirarchical, not relational, so there was already a better impedance match. Then with URL's being the primary identifiers, it was trivial to take a large dataset like VIAF (Virtual International Authority File - canonical representation of all authors) and query the two together seamlessly. Virtuoso was a pretty good Quad Store, and we got away with storing tens of billions of triples on a 4 node cluster, with very fast query times (although sticking to Sparql 1.1 and not leaning on property paths). As to if I would choose it again ... I don't know. I'm now a decade out of the library space and haven't seen anything in my day-to-day work (backend distributed systems) that would benefit from the RDF data model.
- evrennetwork 1y ago[dead]
- mdhb 1y agoMaybe worth also pointing out that a meaningful refresh of the RDF specification is getting rather close to completion. Hopefully version 1.2 which addresses a lot of shortcomings should officially be a thing this year. In the meantime you can take a look at some of the specification docs here https://w3c.github.io/rdf-concepts/spec/ https://w3c.github.io/rdf-concepts/spec/
- jiggawatts 1y agoThe sibling comment by flanked-evergl "RDF is great but it's somewhat inadvertently captured by academia." is made manifestly obvious when reading this spec. It's overburdened by terminology, an exponential explosion of nested definitions, and abstraction to the point of unintelligibility. It is clear that the authors have implementation(s) of the spec in mind while writing, but very carefully dance around it and refuse to be nailed down with pedestrian specifics. I'm reminded of the Wikipedia mathematics articles that define everything in terms of other definitions, and if you navigate to those definitions you eventually end up going in circles back to the article you started out at, no wiser.
- mdhb 1y agoFor what it’s worth, I’m half way through writing a RDF library at the moment and I get what you are saying in an abstract / first impression sense but it all made sense and wasn’t a huge pain to work with in practice. I just think that the idea of how to represent knowledge graphs is a topic that has hidden complexities in it that you need to make explicit. My impression is that the mismatch comes from you get introduced to the idea of a triple just being a “subject, predicate and and object” and it’s incredibly simple but again, real life knowledge representation is a complicated thing and there’s a bit to unpack beyond that first impression.
- rglullis 1y agoWrote this about one month ago here at https://news.ycombinator.com/item?id=44839132 https://news.ycombinator.com/item?id=44839132 I'm completely out of time or energy for any side project at the moment, but if someone wants to steal my idea: please take an llm model and fine tune so that it can take any question and turn it into a SparQL query for Wikidata. Also, make a web crawler that reads the page and turns into a set of RDF triples or QuickStatements for any new facts that are presented. This would effectively be the "ultimate information organizer" and could potentially turn Wikidata into most people's entry page of the internet.
- luguenth 1y agohttps://spinach.genie.stanford.edu/ https://spinach.genie.stanford.edu/ Here you are :)
- deleted 1y ago[deleted]
- yorwba 1y agoI asked "Which country has the most subway stations?" and got the query SELECT ?country (COUNT(*) AS ?stationCount) WHERE { ?station wdt:P31 wd:Q928830. ?station wdt:P17 ?country. } GROUP BY ?country ORDER BY DESC(?stationCount) LIMIT 1 https://query.wikidata.org/#SELECT%20%3Fcountry%20%28COUNT%28%2a%29%20AS%20%3FstationCount%29%20WHERE%20%7B%0A%20%20%3Fstation%20wdt%3AP31%20wd%3AQ928830.%0A%20%20%3Fstation%20wdt%3AP17%20%3Fcountry.%0A%7D%0AGROUP%20BY%20%3Fcountry%0AORDER%20BY%20DESC%28%3FstationCount%29%0ALIMIT%201%0A https://query.wikidata.org/#SELECT%20%3Fcountry%20%28COUNT%2... which is not unreasonable as a quick first attempt, but doesn't account for the fact that many things on Wikidata aren't tagged directly with a country (P17) and instead you first need to walk up a chain of "located in the administrative territorial entity" (P131) to find it, i.e. I would write SELECT ?country (COUNT(DISTINCT ?station) AS ?stationCount) WHERE { ?station wdt:P31 wd:Q928830. ?station wdt:P131*/wdt:P17 ?country. } GROUP BY ?country ORDER BY DESC(?stationCount) LIMIT 1 https://query.wikidata.org/#SELECT%20%3Fcountry%20%28COUNT%28DISTINCT%20%3Fstation%29%20AS%20%3FstationCount%29%20WHERE%20%7B%0A%20%20%3Fstation%20wdt%3AP31%20wd%3AQ928830.%0A%20%20%3Fstation%20wdt%3AP131%2a%2Fwdt%3AP17%20%3Fcountry.%0A%7D%0AGROUP%20BY%20%3Fcountry%0AORDER%20BY%20DESC%28%3FstationCount%29%0ALIMIT%201%0A https://query.wikidata.org/#SELECT%20%3Fcountry%20%28COUNT%2... In this case it doesn't change the answer (it only finds 3 more subway stations in China), but sometimes it does.
- barrenko 1y agoOr a semantic layer it's called?
- retube 1y agoWhat is "RDF" ? Not defined in the article
- jraph 1y agoResource Description Framework [1] is basically a way to describes resources with (subject, verb, object) predicates, where subject is the resource being described and object is another resource related to the subject in a way verb defines (verb is not necessarily a grammatical verb/action, it's often a property name). There are several formats to represent these predicates (Turtle), database implementations, query languages (SPARQL), and there are ontologies, which are schemas, basically, defining/describing what how to describe resource in some domain. It's highly related to the semantic web vision of the early 2000s. If you don't know about it, it is worth taking a few minutes to study it. It sometimes surfaces and it's nice to understand what's going on, it can give good design ideas, and it's an important piece of computer history. It's also the quiet basis for many things, OpenGraph [3] metadata tags in HTML documents are basically RDF for instance. (TIL about RDFa [4] btw, I had always seen these meta tags as very RDF-like, for a good reason indeed). [1] https://en.wikipedia.org/wiki/Resource_Description_Framework https://en.wikipedia.org/wiki/Resource_Description_Framework [2] https://en.wikipedia.org/wiki/Semantic_Web https://en.wikipedia.org/wiki/Semantic_Web [3] https://ogp.me/ https://ogp.me/ [4] https://en.wikipedia.org/wiki/RDFa https://en.wikipedia.org/wiki/RDFa
- pbronez 1y agoAlso worth checking out schema.org, which provides a baseline ontology that pops up in many places https://schema.org/Thing https://schema.org/Thing
- rapnie 1y agoDoes OpenGraph gain any benefit from its definition as linked data? Or might it just as well have been defined as, say, JSON Schema's and referring to those property names in html?
- jraph 1y ago
- jandrewrogers 1y agoAs the article itself points out, this has been around for 25 years. It isn’t an accident that nobody does things this way, it wasn’t an oversight. I worked on semantic web tech back in the day, the approach has major weaknesses and limitations that are being glossed over here. The same article touting RDF as the missing ingredient has been written for every tech trend since it was invented. We don’t need to re-litigate it for AI.
- rglullis 1y agoI would be very interested in reading what you think it can't work. I am inclined to agree with the post on a sibling thread that mentions that the main problem with RDF is that it is been captured by academia.
- 4ndrewl 1y agoIME it's less than a "capture", more that most outside of academia don't have the requisite learning to be able to think in the abstract outside of trivial examples.
- FrankyHollywood 1y agoThe article states "When that same data is transformed into a knowledge graph" This is a non-trivial exercise. How does one transform knowledge into a knowledge graph using RDF? RFD is extremely flexible and can represent any data and that's exactly it's great weakness. It's such a free format there is no consensus on how to represent knowledge. Many academic panels exist to set standards, but many of these efforts end up in github as unmaintained repositories. The most important thing about RDF is that everyone needs to agree on the same modeling standards and use the same ontologies. This is very hard to achieve, and room for a lot of discussion, which makes it 'academic' :)
- bfuller 1y ago>This is a non-trivial exercise. How does one transform knowledge into a knowledge graph using RDF? by using the mcp memory knowledge graph tool, which just worked out of the box for my application of turning forum posts into code implementations.
- verisimi 1y agoRDF - Resource Description Framework > The Resource Description Framework (RDF) is a method to describe and exchange graph data. It was originally designed as a data model for metadata by the World Wide Web Consortium (W3C). https://www.wikipedia.org/wiki/Resource_Description_Framework https://www.wikipedia.org/wiki/Resource_Description_Framewor...
- zekrioca 1y agoAuthor listed RDF a couple dozen of times but didn’t define it, so: The Resource Description Framework (RDF) is a standard model for data interchange on the web, designed to represent interconnected data using a structure of subject-predicate-object triples. It facilitates the merging of data from different sources and supports the evolution of schemas over time without requiring changes to all data consumers.
- ricksunny 1y agoFive times in that article he says some version of “Accuracy triples”. What does that even mean? Suppose something 97% accurate became 99.5% accurate? How can we talk of accuracy doubling or tripling in that context? The only way I could see that working is if the accuracy of something went from say 1% to 3% or 33% to 99%. Which are not realistic values in the LLM case. (And I’m writing as a fan of knowledge graphs).
- IanCal 1y agoThis seems to miss the other side of why all this failed before. Rdf has the same problems as the sql schemas with information scattered. What fields mean requires documentation. There - they have a name on a person. What name? Given? Legal? Chosen? Preferred for this use case? You only have one id for apple eh? Companies are complex to model, do you mean apple just as someone would talk about it? The legal structure of entities that underpins all major companies, what part of it is referred to? I spent a long time building identifiers for universities and companies (which was taken for ROR later) and it was a nightmare to say what a university even was. What’s the name of Cambridge? It’s not “Cambridge University” or “The university of Cambridge” legally. But it also is the actual name as people use it. The university of Paris went from something like 13 institutes to maybe one to then a bunch more. Are companies locations at their headquarters? Which headquarters? Someone will suggest modelling to solve this but here lies the biggest problem: The correct modelling depends on the questions you want to answer. Our modelling had good tradeoffs for mapping academic citation tracking. It had bad modelling for legal ownership. There isn’t one modelling that solves both well. And this is all for the simplest of questions about an organisation - what is it called and is it one or two things?
- dwaite 1y ago> The correct modelling depends on the questions you want to answer. Coincidentally, my main point in any conversation about UML I've ever had
- AtlasBarfed 1y agoBasically it's name spacing hell right? To adapt the saying, an engineer is talking to another engineer about is system, saying he's having issues with names. So he's thinking of using name spaces. Now he has two problems
- simonw 1y agoThat university example is fantastic. I went looking and as far as I can tell "The Chancellor, Masters, and Scholars of the University of Cambridge" is the official name! https://www.cam.ac.uk/about-the-university/how-the-university-and-colleges-work/the-university-as-a-charity https://www.cam.ac.uk/about-the-university/how-the-universit...
- crabmusket 1y agoI really like RDF in theory, as a lot of its ideas just make sense to me: - Using URIs to clarify ambiguous IDs and terms - EAV or subject/verb/object representation for all knowledge - "Open world" graph where you can munge together facts from different sources I guess using RDF specifically, instead of just inventing your own graph database with namespaced properties, means using existing RDF tooling and languages like SPARQL, OWL, SHACL etc. Having looked into the RDF ecosystem to see if I can put something together for a side project inspired by https://paradicms.github.io https://paradicms.github.io, it really feels like there's a whole shed of tools out there, but the shed is a bit dingy, you can't really tell the purpose of the oddly-shaped tools you can see, nobody's organised and laid things out in a clear arrangement and, well, everything seems to be written in Java, which shouldn't be a huge issue but really isn't to my taste.
- Kwpolska 1y agoStop trying to make Semantic Web happen. It’s not going to happen.
- epolanski 1y agoSmall rant: I hate how RDF is the central topic of the blog post, yet it isn't defined even *once*. For the interested: resource description framework.
- tannhaeuser 1y agoHere's the first paragraph of that article: > The Big Picture: Knowledge graphs triple LLM accuracy on enterprise data. But here’s what nobody tells you upfront: every knowledge graph converges on the same patterns, the same solutions. This series reveals why RDF isn’t just one option among many — it’s the natural endpoint of knowledge representation. By Post 6, you’ll see real enterprises learning this lesson at great cost — or great savings. If you really want to continue reading and discuss this kind of drivel, go ahead. RDF the "natural endpoint of knowledge representation" right. As someone having worked on commercial RDF projects at the time, after two decades of pushing RDF by a self-serving W3C and academia until around 2018 or so, let's say I welcome people having come to their senses and are back at working with Datalog and Prolog. Even as a target for neurolinguistics and generation by coding LLMs does SPARQL suck because of its idiosyncratic, design-by-comittee nature compared to the minimalism and elegance of Prolog.
- Emma_Schmidt 1y ago[dead]
- kidehen2 1y agoRDF provides a very natural layer for AI systems. Today, LLM-based AI systems are fundamentally challenged by hallucinations, which makes RDF-based knowledge graphs—constructed using Linked Data Principles—a powerful complement. By using hyperlinks to denote edges and nodes, these graphs enable context enrichment through ontology lookups combined with reasoning and inference. For a detailed post on this synergy, see: https://www.linkedin.com/pulse/large-language-models-llms-powerful-generic-rdf-clients-idehen-xwhfe https://www.linkedin.com/pulse/large-language-models-llms-po... Disclaimer: I am the Founder & CEO of OpenLink Software, creators of Virtuoso.