> use some kind of percentage to determine whether or not it's a dupe That's the problem I intended to solve by 'grouping', as I called it: Don't delete the si

> use some kind of percentage to determine whether or not it's a dupe That's the problem I intended to solve by 'grouping', as I called it: Don't delete the similar items, but group them together (see my GP comment regarding the functionality). Also, it might not be as hard as it seems with this data. There would be maybe thousands of records (though you'd want it to scale higher); how many would have the same sets of proper nouns, for example, and not be about the same topic?