4 ms·
The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields
by dunham 13y ago
The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages.
The software is abandoned, but their algorithms are described in a paper:
http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf
- pranade 13y agoThanks a ton for sharing... the association algorithms have been where we've been spending a good chunk of time. Will read through this
- tlianza 13y agoIf you're interested in hosted solutions that try to do automatic identification of pages, diffbot is worth a look. We've had some good experiences: http://diffbot.com/ http://diffbot.com/
- losvedir 13y agoOh, hey, memories. I worked one summer with David Huynh (who you're linking to there) and David Karger (his thesis advisor) on one of the Simile projects. I vaguely remember playing around with this tool you mentioned. I thiiiiink it was this one[0], although it seems to be superseded by this one[1] now. [0] http://simile.mit.edu/wiki/Piggy_Bank http://simile.mit.edu/wiki/Piggy_Bank [1] http://simile.mit.edu/wiki/Sifter http://simile.mit.edu/wiki/Sifter
- danso 13y agoJust had to chime in and say that David Huynh and his fellow programmers will be forever heroes to me and a small group of data journalists who depended on Gridworks/Google Refine/OpenRefine