4 ms·
"Brad Edwards" and "Bradley Edwards" might be the same individual.
by liotier 11mo ago
"Brad Edwards" and "Bradley Edwards" might be the same individual.
- GuinansEyebrows 11mo agoLikewise for instances of "Larry" and "Lawrence" Summers... probably a lot of those.
- tovej 11mo agoYes, the dataset also has three entries for Virginia Giuffre, "Virginia L. Giuffre", "Virginia Roberts Giuffre", and "Jane Doe Number 3 (Virginia Roberts)"
- DrewADesign 11mo agoI’m sure some developer/archivist is working on a name authority as we speak.
- cyrusradfar 11mo agogreat use case for using AI to suggest mergers and clean up.
- specproc 11mo agoLLMs are awful for this. I've got a project that's doing structured extraction and half the work is deduplication. I didn't go down the route of LLMs for the clean up, as you're getting into scale and context issues with larger datasets. I got into semantic similarity networks for this use case. You can do efficient pairwise matching with Annoy, set a cutoff threshold, and your isolated subgraphs are merger candidates. I wrapped up my code in a little library if you're into this sort of thing. github.com/specialprocedures/semnet
- mvATM99 11mo agoNice looking library! Might try it for one of my own projects.
- adolph 11mo agoI read a recent observation that people subject to discovery are often making purposeful typos in key names in order for the communication to remain under the radar.
- potato3732842 11mo agoEveryone is potentially subject to discovery. Some people are just more aware of it.