4 ms·

Show HN: Graph of wikipedia articles semantic similarity (LSI, Python, d3.js)

54 points by lucamartinetti 15y ago

Small experiment of visualization of wikipedia articles as a graph using d3.js.Articles with more traffic are bigger. I computed the semantic similarity using LSI with python (gensim) You have to scroll down/right a bit!http://similarityapi.appspot.com/graph/?title=blade%20runnerThere is also a JSON api: http://similarityapi.appspot.com/api/v1/?limit=100&title=blade%20runnerAll feedback is appreciated:@lucamartinetti luca@luca.io

lucamartinetti 15y ago
Small experiment of visualization of wikipedia articles as a graph using d3.js. Articles with more traffic are bigger. I computed the semantic similarity using LSI with python (gensim) You have to scroll down/right a bit! http://similarityapi.appspot.com/graph/?title=blade%20runner http://similarityapi.appspot.com/graph/?title=blade%20runner There is also a JSON api: http://similarityapi.appspot.com/api/v1/?limit=100&title=blade%20runner http://similarityapi.appspot.com/api/v1/?limit=100&title... All feedback is appreciated: @lucamartinetti luca@luca.io
- viscanti 15y ago
  The JSON api should degrade gracefully if results aren't found. I.E. There should be a JSON message explaining that that item doesn't exist.
  - lucamartinetti 15y ago
    Right! It could use some input checking / normalization too. It expects the title parameter to be lower case now.
- rplnt 15y ago
  Option to select language version could be a good feature (defaulting to en as now).
- 3pt14159 15y ago
  I've had much, much better results with LDA than LSI. Give that a shot if you have a chance, you'll be blown away. Stop word ratios are important, and make the max number of tokens 500,000.
- Radim 15y ago
  how much data did you use for the semantic analysis?
  - lucamartinetti 15y ago
    The whole text of all articles from wikipedia english (then filtered those with more the 1k views last month)
ssn 15y ago
Down?
- lucamartinetti 15y ago
  Not for me. You need a modern browser (chrome or firefox) and scroll a bit
stephengoodwin 15y ago
Does the font size for a node represent it's similarity with the query page?
- lucamartinetti 15y ago
  It represents the traffic of the article. Ten most related articles are displayed for each expanded node. Articles with more inbound links are darker
lucian1900 15y ago
Blank page in Chrome.
Edootjuh 15y ago
I've never liked these scrolling animations. You need too much precision to see a part of the page clearly, while with normal scrolling it wouldn't matter if the information you're reading is at the bottom or top of the screen.

lucamartinetti 15y ago

Small experiment of visualization of wikipedia articles as a graph using d3.js. Articles with more traffic are bigger. I computed the semantic similarity using LSI with python (gensim) You have to scroll down/right a bit! http://similarityapi.appspot.com/graph/?title=blade%20runner http://similarityapi.appspot.com/graph/?title=blade%20runner There is also a JSON api: http://similarityapi.appspot.com/api/v1/?limit=100&title=blade%20runner http://similarityapi.appspot.com/api/v1/?limit=100&title... All feedback is appreciated: @lucamartinetti luca@luca.io

viscanti 15y ago
The JSON api should degrade gracefully if results aren't found. I.E. There should be a JSON message explaining that that item doesn't exist.
- lucamartinetti 15y ago
  Right! It could use some input checking / normalization too. It expects the title parameter to be lower case now.
rplnt 15y ago
Option to select language version could be a good feature (defaulting to en as now).
3pt14159 15y ago
I've had much, much better results with LDA than LSI. Give that a shot if you have a chance, you'll be blown away. Stop word ratios are important, and make the max number of tokens 500,000.
Radim 15y ago
how much data did you use for the semantic analysis?
- lucamartinetti 15y ago
  The whole text of all articles from wikipedia english (then filtered those with more the 1k views last month)

ssn 15y ago

Down?

lucamartinetti 15y ago
Not for me. You need a modern browser (chrome or firefox) and scroll a bit

stephengoodwin 15y ago

Does the font size for a node represent it's similarity with the query page?

lucamartinetti 15y ago
It represents the traffic of the article. Ten most related articles are displayed for each expanded node. Articles with more inbound links are darker

lucian1900 15y ago

Blank page in Chrome.

Edootjuh 15y ago

I've never liked these scrolling animations. You need too much precision to see a part of the page clearly, while with normal scrolling it wouldn't matter if the information you're reading is at the bottom or top of the screen.