3 ms·

I like the idea, but it would be more meaningful if you could link for semantic reasons - that would be a very nice tool, could even be used by authors who want

I like the idea, but it would be more meaningful if you could link for semantic reasons - that would be a very nice tool, could even be used by authors who want to put a book together from various essays they've written, etc. Or help you focus on an area of writing you're interested in by the author. I mention this because looking at the graph you can see that things that are related are not necessarily linked.

RiderOfGiraffes 18y ago
This would be really cool. Can you suggest a way, given two essays, to decide if, semantically, the first should point to the second? If an author could use a tool like that it would mean that links could be discovered rather than inserted by hand. That would be valuable ... you've made me think ... I may have a way of doing something close enough. Hmm.
- zupatol 18y ago
  Latent semantic analysis can be used to measure the similarity of two documents. http://en.wikipedia.org/wiki/Latent_semantic_analysis http://en.wikipedia.org/wiki/Latent_semantic_analysis I found examples in ruby and python in this blog, which is unfortunately down just at the moment http://blog.josephwilk.net/ruby/latent-semantic-analysis-in-ruby.html http://blog.josephwilk.net/ruby/latent-semantic-analysis-in-...
  - Prrometheus 18y ago
    Gregor Heinrich has a good paper on Latent Dirichlet Allocation, which I believe is an extension of Latent Semantic Analysis. It is a model which can be used to group documents based on semantic content. He gives the mathematical details and the "punchlines" for implementation. The model takes as input a collection of documents and outputs a topic label for each word in each document. The documents can be plotted in K-dimensional space, where K is the number of possible topics, by using the proportion of each topic in a document as its coordinates. Documents which are closer to each other have topics more similar to each other. You could then use your favorite clustering algorithm or a simple distance threshold to decide which should documents should link to each other. The paper is here [PDF]: http://www.arbylon.net/publications/text-est.pdf http://www.arbylon.net/publications/text-est.pdf C++ implementation: http://gibbslda.sourceforge.net/ http://gibbslda.sourceforge.net/ (note, I haven't used the C++ implementation).
    - zupatol 18y ago
      That looks even better. I'm going to try this out in my pet project. Thank you.
aswanson 18y ago
In a way, there is a gross semantic linking in a Q/A type of way with the titles, e.g., Why do Startups Condense in America --> The Power of the Marginal. What Can Business Learn from Open Source -->How To Start A Startup + Inequality and Risk.

RiderOfGiraffes 18y ago

This would be really cool. Can you suggest a way, given two essays, to decide if, semantically, the first should point to the second? If an author could use a tool like that it would mean that links could be discovered rather than inserted by hand. That would be valuable ... you've made me think ... I may have a way of doing something close enough. Hmm.

zupatol 18y ago
Latent semantic analysis can be used to measure the similarity of two documents. http://en.wikipedia.org/wiki/Latent_semantic_analysis http://en.wikipedia.org/wiki/Latent_semantic_analysis I found examples in ruby and python in this blog, which is unfortunately down just at the moment http://blog.josephwilk.net/ruby/latent-semantic-analysis-in-ruby.html http://blog.josephwilk.net/ruby/latent-semantic-analysis-in-...
- Prrometheus 18y ago
  Gregor Heinrich has a good paper on Latent Dirichlet Allocation, which I believe is an extension of Latent Semantic Analysis. It is a model which can be used to group documents based on semantic content. He gives the mathematical details and the "punchlines" for implementation. The model takes as input a collection of documents and outputs a topic label for each word in each document. The documents can be plotted in K-dimensional space, where K is the number of possible topics, by using the proportion of each topic in a document as its coordinates. Documents which are closer to each other have topics more similar to each other. You could then use your favorite clustering algorithm or a simple distance threshold to decide which should documents should link to each other. The paper is here [PDF]: http://www.arbylon.net/publications/text-est.pdf http://www.arbylon.net/publications/text-est.pdf C++ implementation: http://gibbslda.sourceforge.net/ http://gibbslda.sourceforge.net/ (note, I haven't used the C++ implementation).
  - zupatol 18y ago
    That looks even better. I'm going to try this out in my pet project. Thank you.

aswanson 18y ago

In a way, there is a gross semantic linking in a Q/A type of way with the titles, e.g., Why do Startups Condense in America --> The Power of the Marginal. What Can Business Learn from Open Source -->How To Start A Startup + Inequality and Risk.