4 ms·

My experience building http://esciencenews.com http://esciencenews.com , which is automated news for science, tells me otherwise. About the 'instantly obsolete

My experience building http://esciencenews.com http://esciencenews.com , which is automated news for science, tells me otherwise. About the 'instantly obsolete' news - just put the latest of the cluster at the top. We see that a lot with shuttle missions to space : see http://esciencenews.com/sources/space.com/2008/11/12/space.shuttle.endeavour.go.friday.launch http://esciencenews.com/sources/space.com/2008/11/12/space.s... In the cluster ('More sources from other sites') you can see the mission`s coverage from start to finish. As for bad grouping, just accept that you won't be able to find ALL stories belonging to a cluster and set your cutoff to accept the minimum of false positive while keeping the most stories belonging to a cluster.

codeismightier 18y ago
Cool. One feature that I wish that you would add is RSS feeds for each sub-topic. Other than that it looks great.
- FiReaNG3L 18y ago
  There is already; http://esciencenews.com/syndication http://esciencenews.com/syndication I just noticed the topic pages still link to the main rss though, will change that as soon as I get back from the lab!
  - aaronblohowiak 18y ago
    Your site is very well done. I will be returning soon.
brandnewlow 18y ago
Hey FiReaNG3L, I've been scratching my head about this for a while. Over on eScienceNews, some of your news items are full stories. Some are just summaries. When I follow your source links below the full stories I find sites offering summary-only RSS feeds. How are you getting the full versions of those stories? Are you scraping the HTML from the feed item URL?
- FiReaNG3L 18y ago
  The full versions are press releases we get from universities and the like. The summaries with links to full stories are copyrighted (or not, as some site editors do the same press release cut and paste as our automated service does).
  - brandnewlow 18y ago
    What's the workflow look like on those releases? Do they e-mail them to you and get "sucked in?" Do they have a special feed? Do they fill in a form?
    - FiReaNG3L 18y ago
      We scrape them, and the universities and other organizations are pretty happy about it, we often get email saying thanks, links from professors and emails asking how much views they received, etc. These would get no exposure at all without sites like e! science news.
bbgm 18y ago
First thing I thought about when I saw Gabe's post was your site and how well it works, even though it is fully automated. Your last point is probably the most important one, and in my book that's an acceptable trade off. One question. What's the speed of updates? I know techmeme wants to be the place where you go immediately when a news story breaks out. Would that be a problem (only computational I suppose)?
- FiReaNG3L 18y ago
  Updates are every 5 minutes, which is plenty. Faster than that I don't see the point.
deleted 18y ago
[deleted]
shafqat 18y ago
How do you do the clustering? Open source tool/software? Nice site!
- FiReaNG3L 18y ago
  http://search.cpan.org/~mdehoon/Algorithm-Cluster/perl/Cluster.pm http://search.cpan.org/~mdehoon/Algorithm-Cluster/perl/Clust... Full write-up on Drupal.org, awesome CMS we used to build this : http://drupal.org/node/261340 http://drupal.org/node/261340

codeismightier 18y ago

Cool. One feature that I wish that you would add is RSS feeds for each sub-topic. Other than that it looks great.

FiReaNG3L 18y ago
There is already; http://esciencenews.com/syndication http://esciencenews.com/syndication I just noticed the topic pages still link to the main rss though, will change that as soon as I get back from the lab!
- aaronblohowiak 18y ago
  Your site is very well done. I will be returning soon.

brandnewlow 18y ago

Hey FiReaNG3L, I've been scratching my head about this for a while. Over on eScienceNews, some of your news items are full stories. Some are just summaries. When I follow your source links below the full stories I find sites offering summary-only RSS feeds. How are you getting the full versions of those stories? Are you scraping the HTML from the feed item URL?

FiReaNG3L 18y ago
The full versions are press releases we get from universities and the like. The summaries with links to full stories are copyrighted (or not, as some site editors do the same press release cut and paste as our automated service does).
- brandnewlow 18y ago
  What's the workflow look like on those releases? Do they e-mail them to you and get "sucked in?" Do they have a special feed? Do they fill in a form?
  - FiReaNG3L 18y ago
    We scrape them, and the universities and other organizations are pretty happy about it, we often get email saying thanks, links from professors and emails asking how much views they received, etc. These would get no exposure at all without sites like e! science news.

bbgm 18y ago

First thing I thought about when I saw Gabe's post was your site and how well it works, even though it is fully automated. Your last point is probably the most important one, and in my book that's an acceptable trade off. One question. What's the speed of updates? I know techmeme wants to be the place where you go immediately when a news story breaks out. Would that be a problem (only computational I suppose)?

FiReaNG3L 18y ago
Updates are every 5 minutes, which is plenty. Faster than that I don't see the point.

deleted 18y ago

[deleted]

shafqat 18y ago

How do you do the clustering? Open source tool/software? Nice site!

FiReaNG3L 18y ago
http://search.cpan.org/~mdehoon/Algorithm-Cluster/perl/Cluster.pm http://search.cpan.org/~mdehoon/Algorithm-Cluster/perl/Clust... Full write-up on Drupal.org, awesome CMS we used to build this : http://drupal.org/node/261340 http://drupal.org/node/261340