Techmeme's algorithm seems to rely on a combination of clustering similar articles and an analysis of the interlinking of various articles. So if you a textual

Techmeme's algorithm seems to rely on a combination of clustering similar articles and an analysis of the interlinking of various articles. So if you a textual analysis of two articles and discover that both use very similar words -- e.g. both discuss possible merger talks between yahoo and microsoft, they would be placed in a meme together. Also, if an article links to the main article of a meme, odds are it's on the same subject as the main article of the meme. The larger the meme and the more clicks it gets, the higher on the page it rises. And I'm sure there's a number of other factors considered. Google News seems less sophisticated. I haven't worked as hard at reverse engineering it but it seems to rely solely on clustering. I wrote the <a href="http://drupal.org/project/memetracker http://drupal.org/project/memetracker">Drupal Memetracker module</a> last summer as part of Google Summer of Code. At this point it's a usable but fairly simple + unsophisticated memetracker (but patches are highly welcome!). It relies solely on clustering right now to find memes. To step a bit into machine learning concepts -- to cluster items, you first find the "distance" between different items. Then the clustering algorithm runs through the different items bring together items close to each other until it reaches the distance threshold you've set (i.e. at this point items are too far apart to bring together). This type of clustering is called Hierarchical agglomerative clustering see http://nlp.stanford.edu/IR-book/html/htmledition/hierarchical-agglomerative-clustering-1.html http://nlp.stanford.edu/IR-book/html/htmledition/hierarchica... Memetracker uses two tools to do the clustering. It uses the MySQL FullText search to find the distance between news articles. Basically it searches every article against every other article which returns a very accurate distance score. Then for clustering, it uses the PyCluster library as thesethings mentioned. It uses PyCluster as thesethings mentioned.