6 ms·

It's incredible people still try and do things like this. This is such an over ambitious problem... 1. Webpages are ladden with errors, how do you deal with t

by d--b 8y ago

It's incredible people still try and do things like this. This is such an over ambitious problem... 1. Webpages are ladden with errors, how do you deal with this? 2. Knowledge does not fit in a graph. It's asymptically a graph, as in: I can define relationships like: This recipe contains carrots. carrots contain sugar => this recipe contains sugar. Cool. But, what about this "sugar-free carrot cake recipe?" Well it still contains carrots, so still contains sugar... Contradiction? => requires human curating... 3. It doesn't even solve a real problem... Look at IBM watson, it probably knows a lot more crap than diffbot, and yet, is a pretty useless piece of software...

moxious 8y ago
Wanna thread other known attempts at this? Maybe someone will jump in with extra detail about how these approaches are different, or what extra value we'd expect. - OpenCyc http://www.cyc.com/opencyc/ http://www.cyc.com/opencyc/ - DBPedia https://wiki.dbpedia.org/ https://wiki.dbpedia.org/
- miket 8y ago
  Founder here. OpenCyc (as well as Freebase) are human attempts to enter and curate a structured knowledge base. Likewise DBPedia is a set of scripts that extract Wikipedia infoboxes (semi-structured data which is also human crowd-sourced). The Diffbot Knowledge Graph is built by applying computer vision and natural language processing techniques to reading all the pages on the web (which can be in any structure and human language) and extracting it into a structured form, without the element of human annotation in the build pipeline.
  - moxious 8y ago
    Can you expand on major points of how this will make the content different, (for example, Wikipedia is curated and non-notable people pages get thrown out, so if you're reading all of the web, presumably you'd know about non-notable people) -- and why it's better?
    - subhobroto 8y ago
      Fantastic question. A major component of a machine generated ontology has to be a notability score, otherwise it would be practically impossible to store all entities (and their relationships). Further, for this to scale, Diffbot has to have a way to align their entity IDs with IDs from other notable graphs like Wikipedia, Wikidata, Freebase, Wordnet or even Yelp, and the like, otherwise the data could be potential of diminished value. How would I know that the "Cardi B" that's in my database with ID 321 and wikidata ID Q29033668 is the same as Diffbot's "Cardi B" with ID 561?
    - miket 8y ago
      Founder here. There are many differences in the result when you have an automated system building a Knowledge Graph vs. a human one. Obvious one is scale, Wikipedia has on order 10M entities and represents the work of thousands of humans whereas the Diffbot KG has 10B entities and is discovering about 120M each day, and is largely limited by the number of machines running the algorithms in the datacenter. The properties and facts indexed about each entity are also a superset because it is not limited to those that would be worthwhile for a human to curate. Lastly, it can be more accurate than facts found in a single source because the automated system utilizes multiple sources of that fact found across the web to estimate a probability of the accuracy of the fact. The result is that you have a Knowledge Graph that is more useful for work and business because they are the entities you interact with day to day, not the "head" entities that optimize for popularity and the constraints of human curation.
neffy 8y ago
Personally I´m convinced there´s a "taxonomise everything" gene. Just look at biology. And while it may ultimately be doomed to partial failure, it´s still a beautiful thing to watch.
o_____________o 8y ago
Based on a short trial of the API, doesn't seem like they're divining much from pages anyways – it couldn't identify Who and What I was parsing, and just seems to extract terms found in dbpedia, which are often wildly irrelevant as primary properties. If we're talking notable people/things, you'd get more structured meaning out of searching a local Wikidata graph.
miket 8y ago
Founder here. The internet is full of contradictory statements, but that doesn't mean we can't attempt to form a useful model of reality. Humans do, just to get through daily life and AI can do the same as well. As you point out, the concept "sugar-free" does not equate to there being no "sugar" in the dish. Perhaps it means to folks something closer to that fact that no added sugar was explicit in the recipe. The word "sugar" itself has many meanings (the well-known problem of word Polysemy). So there are several similar but distinct concepts at involved here and an entity could have the value of true for one of these and false for another.

by d--b 8y ago

moxious 8y ago

Wanna thread other known attempts at this? Maybe someone will jump in with extra detail about how these approaches are different, or what extra value we'd expect. - OpenCyc http://www.cyc.com/opencyc/ http://www.cyc.com/opencyc/ - DBPedia https://wiki.dbpedia.org/ https://wiki.dbpedia.org/

miket 8y ago
Founder here. OpenCyc (as well as Freebase) are human attempts to enter and curate a structured knowledge base. Likewise DBPedia is a set of scripts that extract Wikipedia infoboxes (semi-structured data which is also human crowd-sourced). The Diffbot Knowledge Graph is built by applying computer vision and natural language processing techniques to reading all the pages on the web (which can be in any structure and human language) and extracting it into a structured form, without the element of human annotation in the build pipeline.
- moxious 8y ago
  Can you expand on major points of how this will make the content different, (for example, Wikipedia is curated and non-notable people pages get thrown out, so if you're reading all of the web, presumably you'd know about non-notable people) -- and why it's better?
  - subhobroto 8y ago
    Fantastic question. A major component of a machine generated ontology has to be a notability score, otherwise it would be practically impossible to store all entities (and their relationships). Further, for this to scale, Diffbot has to have a way to align their entity IDs with IDs from other notable graphs like Wikipedia, Wikidata, Freebase, Wordnet or even Yelp, and the like, otherwise the data could be potential of diminished value. How would I know that the "Cardi B" that's in my database with ID 321 and wikidata ID Q29033668 is the same as Diffbot's "Cardi B" with ID 561?
  - miket 8y ago
    Founder here. There are many differences in the result when you have an automated system building a Knowledge Graph vs. a human one. Obvious one is scale, Wikipedia has on order 10M entities and represents the work of thousands of humans whereas the Diffbot KG has 10B entities and is discovering about 120M each day, and is largely limited by the number of machines running the algorithms in the datacenter. The properties and facts indexed about each entity are also a superset because it is not limited to those that would be worthwhile for a human to curate. Lastly, it can be more accurate than facts found in a single source because the automated system utilizes multiple sources of that fact found across the web to estimate a probability of the accuracy of the fact. The result is that you have a Knowledge Graph that is more useful for work and business because they are the entities you interact with day to day, not the "head" entities that optimize for popularity and the constraints of human curation.

neffy 8y ago

Personally I´m convinced there´s a "taxonomise everything" gene. Just look at biology. And while it may ultimately be doomed to partial failure, it´s still a beautiful thing to watch.

o_____________o 8y ago

Based on a short trial of the API, doesn't seem like they're divining much from pages anyways – it couldn't identify Who and What I was parsing, and just seems to extract terms found in dbpedia, which are often wildly irrelevant as primary properties. If we're talking notable people/things, you'd get more structured meaning out of searching a local Wikidata graph.

miket 8y ago

Founder here. The internet is full of contradictory statements, but that doesn't mean we can't attempt to form a useful model of reality. Humans do, just to get through daily life and AI can do the same as well. As you point out, the concept "sugar-free" does not equate to there being no "sugar" in the dish. Perhaps it means to folks something closer to that fact that no added sugar was explicit in the recipe. The word "sugar" itself has many meanings (the well-known problem of word Polysemy). So there are several similar but distinct concepts at involved here and an entity could have the value of true for one of these and false for another.