5 ms·

Diffbot launches AI-powered knowledge graph of 1T facts

47 points by borisjabes 8y ago

d--b 8y ago
It's incredible people still try and do things like this. This is such an over ambitious problem... 1. Webpages are ladden with errors, how do you deal with this? 2. Knowledge does not fit in a graph. It's asymptically a graph, as in: I can define relationships like: This recipe contains carrots. carrots contain sugar => this recipe contains sugar. Cool. But, what about this "sugar-free carrot cake recipe?" Well it still contains carrots, so still contains sugar... Contradiction? => requires human curating... 3. It doesn't even solve a real problem... Look at IBM watson, it probably knows a lot more crap than diffbot, and yet, is a pretty useless piece of software...
- moxious 8y ago
  Wanna thread other known attempts at this? Maybe someone will jump in with extra detail about how these approaches are different, or what extra value we'd expect. - OpenCyc http://www.cyc.com/opencyc/ http://www.cyc.com/opencyc/ - DBPedia https://wiki.dbpedia.org/ https://wiki.dbpedia.org/
  - miket 8y ago
    Founder here. OpenCyc (as well as Freebase) are human attempts to enter and curate a structured knowledge base. Likewise DBPedia is a set of scripts that extract Wikipedia infoboxes (semi-structured data which is also human crowd-sourced). The Diffbot Knowledge Graph is built by applying computer vision and natural language processing techniques to reading all the pages on the web (which can be in any structure and human language) and extracting it into a structured form, without the element of human annotation in the build pipeline.
    - moxious 8y ago
      Can you expand on major points of how this will make the content different, (for example, Wikipedia is curated and non-notable people pages get thrown out, so if you're reading all of the web, presumably you'd know about non-notable people) -- and why it's better?
      - subhobroto 8y ago
        Fantastic question. A major component of a machine generated ontology has to be a notability score, otherwise it would be practically impossible to store all entities (and their relationships). Further, for this to scale, Diffbot has to have a way to align their entity IDs with IDs from other notable graphs like Wikipedia, Wikidata, Freebase, Wordnet or even Yelp, and the like, otherwise the data could be potential of diminished value. How would I know that the "Cardi B" that's in my database with ID 321 and wikidata ID Q29033668 is the same as Diffbot's "Cardi B" with ID 561?
      - miket 8y ago
        Founder here. There are many differences in the result when you have an automated system building a Knowledge Graph vs. a human one. Obvious one is scale, Wikipedia has on order 10M entities and represents the work of thousands of humans whereas the Diffbot KG has 10B entities and is discovering about 120M each day, and is largely limited by the number of machines running the algorithms in the datacenter. The properties and facts indexed about each entity are also a superset because it is not limited to those that would be worthwhile for a human to curate. Lastly, it can be more accurate than facts found in a single source because the automated system utilizes multiple sources of that fact found across the web to estimate a probability of the accuracy of the fact. The result is that you have a Knowledge Graph that is more useful for work and business because they are the entities you interact with day to day, not the "head" entities that optimize for popularity and the constraints of human curation.
- neffy 8y ago
  Personally I´m convinced there´s a "taxonomise everything" gene. Just look at biology. And while it may ultimately be doomed to partial failure, it´s still a beautiful thing to watch.
- o_____________o 8y ago
  Based on a short trial of the API, doesn't seem like they're divining much from pages anyways – it couldn't identify Who and What I was parsing, and just seems to extract terms found in dbpedia, which are often wildly irrelevant as primary properties. If we're talking notable people/things, you'd get more structured meaning out of searching a local Wikidata graph.
- miket 8y ago
  Founder here. The internet is full of contradictory statements, but that doesn't mean we can't attempt to form a useful model of reality. Humans do, just to get through daily life and AI can do the same as well. As you point out, the concept "sugar-free" does not equate to there being no "sugar" in the dish. Perhaps it means to folks something closer to that fact that no added sugar was explicit in the recipe. The word "sugar" itself has many meanings (the well-known problem of word Polysemy). So there are several similar but distinct concepts at involved here and an entity could have the value of true for one of these and false for another.
johnymontana 8y ago
What technologies are this built on? Something like Neo4j graph database?
- dwynings 8y ago
  We tried Neo4j, but it couldn't support the throughput we needed for injecting facts.
  - subhobroto 8y ago
    Upvoted. Have you given ArangoDB a shot?
    - miket 8y ago
      We've tried Neo4J, ArangoDB, as well as many others to store the triples. Neo4J locked up at around 100M entities, and also the loading/injection times weren't sufficient build the KG at a regular interval. However, we are closely following any developments in these projects as they improve. There is more detail in this interview here: https://www.zdnet.com/article/the-web-as-a-database-the-biggest-knowledge-graph-ever/ https://www.zdnet.com/article/the-web-as-a-database-the-bigg...
- subhobroto 8y ago
  +1 to this question. Approaching this problem requires a scalable graph database. AllegroGraph is a well known solution to this problem but out of reach of most enthusiasts like me, so if DiffBot came up with their own solution, I, among others would love to read about it more
subhobroto 8y ago
Mike, fantastic work here! As someone who's really excited about a machine readable web and have been working on it, this is fantastic! Unfortunately, while the Semantic web was to tackle this, the real life proliferation of the Semweb has been, atleast to me personally, extremely disappointing. So this is a fantastic initiative, personally for me to know about. Is there a plan to expose this data via a dev API of some sort for enthusiasts like us? Say a SPARQL or even (Open)Graph API perhaps? My experiences consulting with and working with companies interested in the domain has been that monetizing this data is extremely hard both legally and quality wise. "Nike Tanjun near me" is a query fraught with danger. People typing this query want to find a retailer in their vicinity that sells this Nike product, but where do we source that inventory list from and how do we get our cut? Before people start talking about DSPs and SSPs, this is a very different problem at hand. To know that Nike Tanjun is a shoe sold by Nike, an ontology needs to exist that captures this knowledge so that the user's query can be decoded. How will that ontology be sourced? Further, for it to be usable commercially, Nike has to agree to that encoding. Therin begin the challenges. If we encoded Nike Downshifter to be Tanjun, by mistake, then the user bought them based off our results, disliked them expecting the Downshifters to be like Tanjuns, we have an issue and Nike could persue the matter because we mislead the customer and affected their branding. My primary clients are search companies or companies that want to provide rich search functionality: Google, Bing or even DDG do a phenomenal job in this space and the barrier to entry is pretty high. So knowledge quality, mainteanaanace, versioning and temporal resolution ("The President of the U.S.", "The iPhone" are different entities over time) aside, is diffbot going to monetize this knowledge only as a B2B offering/addon to their clients or are there other "bigger" plans to monetize this tremendous undertaking and keep it rolling in the future?

47 points by borisjabes 8y ago

d--b 8y ago

It's incredible people still try and do things like this. This is such an over ambitious problem... 1. Webpages are ladden with errors, how do you deal with this? 2. Knowledge does not fit in a graph. It's asymptically a graph, as in: I can define relationships like: This recipe contains carrots. carrots contain sugar => this recipe contains sugar. Cool. But, what about this "sugar-free carrot cake recipe?" Well it still contains carrots, so still contains sugar... Contradiction? => requires human curating... 3. It doesn't even solve a real problem... Look at IBM watson, it probably knows a lot more crap than diffbot, and yet, is a pretty useless piece of software...

moxious 8y ago
Wanna thread other known attempts at this? Maybe someone will jump in with extra detail about how these approaches are different, or what extra value we'd expect. - OpenCyc http://www.cyc.com/opencyc/ http://www.cyc.com/opencyc/ - DBPedia https://wiki.dbpedia.org/ https://wiki.dbpedia.org/
- miket 8y ago
  Founder here. OpenCyc (as well as Freebase) are human attempts to enter and curate a structured knowledge base. Likewise DBPedia is a set of scripts that extract Wikipedia infoboxes (semi-structured data which is also human crowd-sourced). The Diffbot Knowledge Graph is built by applying computer vision and natural language processing techniques to reading all the pages on the web (which can be in any structure and human language) and extracting it into a structured form, without the element of human annotation in the build pipeline.
  - moxious 8y ago
    Can you expand on major points of how this will make the content different, (for example, Wikipedia is curated and non-notable people pages get thrown out, so if you're reading all of the web, presumably you'd know about non-notable people) -- and why it's better?
    - subhobroto 8y ago
      Fantastic question. A major component of a machine generated ontology has to be a notability score, otherwise it would be practically impossible to store all entities (and their relationships). Further, for this to scale, Diffbot has to have a way to align their entity IDs with IDs from other notable graphs like Wikipedia, Wikidata, Freebase, Wordnet or even Yelp, and the like, otherwise the data could be potential of diminished value. How would I know that the "Cardi B" that's in my database with ID 321 and wikidata ID Q29033668 is the same as Diffbot's "Cardi B" with ID 561?
    - miket 8y ago
      Founder here. There are many differences in the result when you have an automated system building a Knowledge Graph vs. a human one. Obvious one is scale, Wikipedia has on order 10M entities and represents the work of thousands of humans whereas the Diffbot KG has 10B entities and is discovering about 120M each day, and is largely limited by the number of machines running the algorithms in the datacenter. The properties and facts indexed about each entity are also a superset because it is not limited to those that would be worthwhile for a human to curate. Lastly, it can be more accurate than facts found in a single source because the automated system utilizes multiple sources of that fact found across the web to estimate a probability of the accuracy of the fact. The result is that you have a Knowledge Graph that is more useful for work and business because they are the entities you interact with day to day, not the "head" entities that optimize for popularity and the constraints of human curation.
neffy 8y ago
Personally I´m convinced there´s a "taxonomise everything" gene. Just look at biology. And while it may ultimately be doomed to partial failure, it´s still a beautiful thing to watch.
o_____________o 8y ago
Based on a short trial of the API, doesn't seem like they're divining much from pages anyways – it couldn't identify Who and What I was parsing, and just seems to extract terms found in dbpedia, which are often wildly irrelevant as primary properties. If we're talking notable people/things, you'd get more structured meaning out of searching a local Wikidata graph.
miket 8y ago
Founder here. The internet is full of contradictory statements, but that doesn't mean we can't attempt to form a useful model of reality. Humans do, just to get through daily life and AI can do the same as well. As you point out, the concept "sugar-free" does not equate to there being no "sugar" in the dish. Perhaps it means to folks something closer to that fact that no added sugar was explicit in the recipe. The word "sugar" itself has many meanings (the well-known problem of word Polysemy). So there are several similar but distinct concepts at involved here and an entity could have the value of true for one of these and false for another.

johnymontana 8y ago

What technologies are this built on? Something like Neo4j graph database?

dwynings 8y ago
We tried Neo4j, but it couldn't support the throughput we needed for injecting facts.
- subhobroto 8y ago
  Upvoted. Have you given ArangoDB a shot?
  - miket 8y ago
    We've tried Neo4J, ArangoDB, as well as many others to store the triples. Neo4J locked up at around 100M entities, and also the loading/injection times weren't sufficient build the KG at a regular interval. However, we are closely following any developments in these projects as they improve. There is more detail in this interview here: https://www.zdnet.com/article/the-web-as-a-database-the-biggest-knowledge-graph-ever/ https://www.zdnet.com/article/the-web-as-a-database-the-bigg...
subhobroto 8y ago
+1 to this question. Approaching this problem requires a scalable graph database. AllegroGraph is a well known solution to this problem but out of reach of most enthusiasts like me, so if DiffBot came up with their own solution, I, among others would love to read about it more

subhobroto 8y ago

Mike, fantastic work here! As someone who's really excited about a machine readable web and have been working on it, this is fantastic! Unfortunately, while the Semantic web was to tackle this, the real life proliferation of the Semweb has been, atleast to me personally, extremely disappointing. So this is a fantastic initiative, personally for me to know about. Is there a plan to expose this data via a dev API of some sort for enthusiasts like us? Say a SPARQL or even (Open)Graph API perhaps? My experiences consulting with and working with companies interested in the domain has been that monetizing this data is extremely hard both legally and quality wise. "Nike Tanjun near me" is a query fraught with danger. People typing this query want to find a retailer in their vicinity that sells this Nike product, but where do we source that inventory list from and how do we get our cut? Before people start talking about DSPs and SSPs, this is a very different problem at hand. To know that Nike Tanjun is a shoe sold by Nike, an ontology needs to exist that captures this knowledge so that the user's query can be decoded. How will that ontology be sourced? Further, for it to be usable commercially, Nike has to agree to that encoding. Therin begin the challenges. If we encoded Nike Downshifter to be Tanjun, by mistake, then the user bought them based off our results, disliked them expecting the Downshifters to be like Tanjuns, we have an issue and Nike could persue the matter because we mislead the customer and affected their branding. My primary clients are search companies or companies that want to provide rich search functionality: Google, Bing or even DDG do a phenomenal job in this space and the barrier to entry is pretty high. So knowledge quality, mainteanaanace, versioning and temporal resolution ("The President of the U.S.", "The iPhone" are different entities over time) aside, is diffbot going to monetize this knowledge only as a B2B offering/addon to their clients or are there other "bigger" plans to monetize this tremendous undertaking and keep it rolling in the future?