5 ms·

OT: I was using Wikipedia the other day and it occurred to me how primitive it is to have all the inner links to other Wikipedia articles defined manually, sure

by nsns 13y ago

OT: I was using Wikipedia the other day and it occurred to me how primitive it is to have all the inner links to other Wikipedia articles defined manually, surely these should have been automated by now (i.e., marking a word or two would link you to the relevant article).

eksith 13y ago
I would venture there's a way to make "concepts" and "entities" become linkable automatically based on existing articles, but that would mean a bit of engineering. I.E. A name, product or academic field. But then there's going to be a high number of links to articles that haven't been created yet or deleted/merged etc... in cases of lack of notability.
- mindcrime 13y ago
  I don't know if it has that exact feature or not, but Semantic Mediawiki has a lot of extensions to base Mediawiki that involve working with data at a semantic level. http://semantic-mediawiki.org/wiki/Help:Introduction_to_Semantic_MediaWiki http://semantic-mediawiki.org/wiki/Help:Introduction_to_Sema... If not, doing that should be possible using an NLP library that does NER. Along with heuristics, one could use the list of currently existing articles as a seed. Edit: Of course, if all you're trying to do is link to existing pages, then you don't use the set of existing pages as a seed, you just use them as the list. But if you're trying to extract "entities" that don't have WP pages yet, then you'd still want to fall back to other NER techniques, which include various heuristics and what-not. Whether or not there would be an value in that is an open question, I suppose.
  - eksith 13y ago
    Good resource! And I think your idea of linking to the disambiguation page makes sense, but there may be a way to infer the correct article from the list of links based on the context of the text in the linking article.
    - mindcrime 13y ago
      Yeah, there's that as well. Also, if you're interested in that sort of thing, two other projects you might find interest are: http://stanbol.apache.org http://stanbol.apache.org and http://uima.apache.org http://uima.apache.org Both involve extracting semantic meaning from unstructured data. It's pretty cool stuff.
      - bergie 13y ago
        Here is a quick demo of Stanbol-provided Wikipedia annotations and disambiguation in a WYSIWYG editor: https://www.youtube.com/watch?v=zAMUpd6rb9k&feature=youtube_gdata_player https://www.youtube.com/watch?v=zAMUpd6rb9k&feature=yout...
nostromo 13y ago
The disambiguation could be challenging. (Does "The Sun" link to our closest star or the tabloid in the UK.) It would be a fun project to try and determine the correct link based on the context.
- mindcrime 13y ago
  Just make the link to the disambiguation page, if there is one? Otherwise, make it a special link that doesn't go anywhere directly, but uses some javascript/CSS to raise a dialog when clicked, that gives you the different choices?
  - stephengillie 13y ago
    Auto-generate disambiguation pages? When a search term matches a tag, and none of the tagged pages have a clear "majority probability" of being correct, it would display a list of all pages with the tag, in order of popularity.
  - helium 13y ago
    I don't think it would be a good idea if wikipedia required that you to run Javascript to navigate the site, and it would make for pretty bad SEO.
    - eru 13y ago
      Wikipedia doesn't need to care to much about SEO.
  - derleth 13y ago
    > Just make the link to the disambiguation page, if there is one? Otherwise, make it a special link that doesn't go anywhere directly, but uses some javascript/CSS to raise a dialog when clicked, that gives you the different choices? Both of these things would be amazingly annoying to the majority of Wikipedia users.
    - mindcrime 13y ago
mjn 13y ago
There's a lot of research dating back to the early days of hypertext on automatic link insertion, but afaict it hasn't really caught on in any system, whether hypertext or wiki-based, with the exception of those somewhat spammy content farms that auto-cross-link their articles. Wikipedia has indeed spawned a new wave of such research, e.g. (random example pulled out of Google Scholar): http://www.cs.waikato.ac.nz/~ihw/papers/08-DNM-IHW-LearningToLinkWithWikipedia.pdf http://www.cs.waikato.ac.nz/~ihw/papers/08-DNM-IHW-LearningT... I'd be curious how good the results are. I've found a bunch of articles, but no live demo. If someone set up a small-scale version where you compare the auto-linked version of a few hundred Wikipedia articles with the existing manually-linked version, I think that could convince people it was worth adopting (if the results looked good).
- mseebach 13y ago
  > There's a lot of research dating back to the early days of hypertext on automatic link insertion, but afaict it hasn't really caught on in any system Except, of course, any mention of the name of a listed company in a financial/business publication.
  - illuminate 13y ago
    Plenty of webforums use text ad-links interspersed with content for unlogged-in guest users.
- nsns 13y ago
  There's a certain confusion, or duality, in the way Wikipedia links work. If you access a director's page for example, and a sentence states he has made "20 movies", clicking on "movies" can either take you to his separate filmography page or to the general article "Movie". I believe only the first option should be manually defined. The Free Dictionary uses something like that - try double clicking words within the definition: http://www.thefreedictionary.com/link http://www.thefreedictionary.com/link
mongol 13y ago
I don't think it is primitive. Wikipedia is edited by humans, I don't think an automatic link algorithm could do a better job at this time.
bergie 13y ago
Here is one solution they could use: http://bergie.iki.fi/blog/automated-linking/ http://bergie.iki.fi/blog/automated-linking/

by nsns 13y ago

eksith 13y ago

I would venture there's a way to make "concepts" and "entities" become linkable automatically based on existing articles, but that would mean a bit of engineering. I.E. A name, product or academic field. But then there's going to be a high number of links to articles that haven't been created yet or deleted/merged etc... in cases of lack of notability.

mindcrime 13y ago
I don't know if it has that exact feature or not, but Semantic Mediawiki has a lot of extensions to base Mediawiki that involve working with data at a semantic level. http://semantic-mediawiki.org/wiki/Help:Introduction_to_Semantic_MediaWiki http://semantic-mediawiki.org/wiki/Help:Introduction_to_Sema... If not, doing that should be possible using an NLP library that does NER. Along with heuristics, one could use the list of currently existing articles as a seed. Edit: Of course, if all you're trying to do is link to existing pages, then you don't use the set of existing pages as a seed, you just use them as the list. But if you're trying to extract "entities" that don't have WP pages yet, then you'd still want to fall back to other NER techniques, which include various heuristics and what-not. Whether or not there would be an value in that is an open question, I suppose.
- eksith 13y ago
  Good resource! And I think your idea of linking to the disambiguation page makes sense, but there may be a way to infer the correct article from the list of links based on the context of the text in the linking article.
  - mindcrime 13y ago
    Yeah, there's that as well. Also, if you're interested in that sort of thing, two other projects you might find interest are: http://stanbol.apache.org http://stanbol.apache.org and http://uima.apache.org http://uima.apache.org Both involve extracting semantic meaning from unstructured data. It's pretty cool stuff.
    - bergie 13y ago
      Here is a quick demo of Stanbol-provided Wikipedia annotations and disambiguation in a WYSIWYG editor: https://www.youtube.com/watch?v=zAMUpd6rb9k&feature=youtube_gdata_player https://www.youtube.com/watch?v=zAMUpd6rb9k&feature=yout...

nostromo 13y ago

The disambiguation could be challenging. (Does "The Sun" link to our closest star or the tabloid in the UK.) It would be a fun project to try and determine the correct link based on the context.

mindcrime 13y ago
Just make the link to the disambiguation page, if there is one? Otherwise, make it a special link that doesn't go anywhere directly, but uses some javascript/CSS to raise a dialog when clicked, that gives you the different choices?
- stephengillie 13y ago
  Auto-generate disambiguation pages? When a search term matches a tag, and none of the tagged pages have a clear "majority probability" of being correct, it would display a list of all pages with the tag, in order of popularity.
- helium 13y ago
  I don't think it would be a good idea if wikipedia required that you to run Javascript to navigate the site, and it would make for pretty bad SEO.
  - eru 13y ago
    Wikipedia doesn't need to care to much about SEO.
- derleth 13y ago
  > Just make the link to the disambiguation page, if there is one? Otherwise, make it a special link that doesn't go anywhere directly, but uses some javascript/CSS to raise a dialog when clicked, that gives you the different choices? Both of these things would be amazingly annoying to the majority of Wikipedia users.
  - mindcrime 13y ago

mjn 13y ago

There's a lot of research dating back to the early days of hypertext on automatic link insertion, but afaict it hasn't really caught on in any system, whether hypertext or wiki-based, with the exception of those somewhat spammy content farms that auto-cross-link their articles. Wikipedia has indeed spawned a new wave of such research, e.g. (random example pulled out of Google Scholar): http://www.cs.waikato.ac.nz/~ihw/papers/08-DNM-IHW-LearningToLinkWithWikipedia.pdf http://www.cs.waikato.ac.nz/~ihw/papers/08-DNM-IHW-LearningT... I'd be curious how good the results are. I've found a bunch of articles, but no live demo. If someone set up a small-scale version where you compare the auto-linked version of a few hundred Wikipedia articles with the existing manually-linked version, I think that could convince people it was worth adopting (if the results looked good).

mseebach 13y ago
> There's a lot of research dating back to the early days of hypertext on automatic link insertion, but afaict it hasn't really caught on in any system Except, of course, any mention of the name of a listed company in a financial/business publication.
- illuminate 13y ago
  Plenty of webforums use text ad-links interspersed with content for unlogged-in guest users.
nsns 13y ago
There's a certain confusion, or duality, in the way Wikipedia links work. If you access a director's page for example, and a sentence states he has made "20 movies", clicking on "movies" can either take you to his separate filmography page or to the general article "Movie". I believe only the first option should be manually defined. The Free Dictionary uses something like that - try double clicking words within the definition: http://www.thefreedictionary.com/link http://www.thefreedictionary.com/link

mongol 13y ago

I don't think it is primitive. Wikipedia is edited by humans, I don't think an automatic link algorithm could do a better job at this time.

bergie 13y ago

Here is one solution they could use: http://bergie.iki.fi/blog/automated-linking/ http://bergie.iki.fi/blog/automated-linking/