2 ms·

> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. O

> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of rules for letters and letter combinations which may or may not result in the same sounds, hand-coded as a huge if tree by a linguist not a programmer)? What about their relatives, do they match by the same rules? Should we incorporate domain knowledge about local naming customs? Etc etc. I pointed a coding agent to this problem, and it aggressively started coming up with complex scoring rules and testing them against real datasets. Which led to sort-of acceptable results, but it still missed lots of cases which were obvious to a human, and had false positives which were obvious to a human. Which I could trade off, and slightly improve, with more back and forth with the coding agent. Pointing a good LLM to all the information about two people, would of course give great results. Maybe even better than human judgment. But I can't do that for 100000^2 people, it would be too expensive in all sorts of ways. I need a fast, reliable scorer. I could maybe train an embedding, but that would be a huge job and where would I get the quality data?

camdenclark 10d ago
You need blocking! Fundamentally this is an entity resolution problem. An LLM can score pairwise really well but scoring all the pairs would be insanely computationally difficult. If you can constrain the set of potential matches up front by querying the dataset for things that could be matches it gets a lot more tractable to use an LLM for this. Are there any heuristics you can use to reduce the search space? You mentioned soundex transformation and maybe prefixes of last names could work? Even if you get the number of potential matches down by a few orders of magnitude this gets more reasonable! Check out https://moj-analytical-services.github.io/splink/index.html https://moj-analytical-services.github.io/splink/index.html
- vintermann 10d ago
  The coding agent was pretty good at coming up with heuristics for matching - even more than the dozen I suggested from domain experience. And it used some of them sensibly for blocking, too. I'm sure I could get it to perform a little better and a lot faster with more agent wrangling. I did consider using the heuristics just for blocking, and letting a local LLM do the actual evaluation, but if Jev or Jev-like models work as advertised, maybe we can have the best of both worlds. Thanks for the link, it is an interesting topic.
RobinL 10d ago
There's also two other important limitations to using an LLM and just providing it with pairs of records. It does not know enough about the records in the context of the overall dataset: - what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but rare in others. I've written in more detail about this here: https://www.robinlinacre.com/fellegi_sunter_accuracy/ https://www.robinlinacre.com/fellegi_sunter_accuracy/
- vintermann 10d ago
  Another great article, thanks. I thought a lot about that too, and basically realized the same thing (this wasn't the coding agent), that frequency in the dataset mattered a lot for how useful a match is, and made frequency lists from my datasets, both on how common the literal names were and the phoneme-reduced names. But I'm worried about that "bitter lesson" the TypeSafe CEO refers to, and that we're retreading the steps of natural language processing and a lot of other fields, trying to come up with clever rules, when the rule-based approach simply never gets good enough. I need matching to be good, it's absolutely central to genealogy, and I've seen the damage blindly linking by hand-crafted matching formulas can do.
  - RobinL 10d ago
    I agree, and I think LLMs can potentially do a better job than more conventional methods, so long as they're provided with enough content. Another aspect of context that could be relevant to your work may be how people are nested within households. In some record linkage scenarios you can run a linkage on household membership and use this to make person linkages more precise. I imagine an LLM agent could also use this information effectively if it were explicitly provided, but sometimes they miss logical leaps like this
satvikpendem 10d ago
Maybe you need Prolog instead.

camdenclark 10d ago

You need blocking! Fundamentally this is an entity resolution problem. An LLM can score pairwise really well but scoring all the pairs would be insanely computationally difficult. If you can constrain the set of potential matches up front by querying the dataset for things that could be matches it gets a lot more tractable to use an LLM for this. Are there any heuristics you can use to reduce the search space? You mentioned soundex transformation and maybe prefixes of last names could work? Even if you get the number of potential matches down by a few orders of magnitude this gets more reasonable! Check out https://moj-analytical-services.github.io/splink/index.html https://moj-analytical-services.github.io/splink/index.html

vintermann 10d ago
The coding agent was pretty good at coming up with heuristics for matching - even more than the dozen I suggested from domain experience. And it used some of them sensibly for blocking, too. I'm sure I could get it to perform a little better and a lot faster with more agent wrangling. I did consider using the heuristics just for blocking, and letting a local LLM do the actual evaluation, but if Jev or Jev-like models work as advertised, maybe we can have the best of both worlds. Thanks for the link, it is an interesting topic.

RobinL 10d ago

There's also two other important limitations to using an LLM and just providing it with pairs of records. It does not know enough about the records in the context of the overall dataset: - what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but rare in others. I've written in more detail about this here: https://www.robinlinacre.com/fellegi_sunter_accuracy/ https://www.robinlinacre.com/fellegi_sunter_accuracy/

vintermann 10d ago
Another great article, thanks. I thought a lot about that too, and basically realized the same thing (this wasn't the coding agent), that frequency in the dataset mattered a lot for how useful a match is, and made frequency lists from my datasets, both on how common the literal names were and the phoneme-reduced names. But I'm worried about that "bitter lesson" the TypeSafe CEO refers to, and that we're retreading the steps of natural language processing and a lot of other fields, trying to come up with clever rules, when the rule-based approach simply never gets good enough. I need matching to be good, it's absolutely central to genealogy, and I've seen the damage blindly linking by hand-crafted matching formulas can do.
- RobinL 10d ago
  I agree, and I think LLMs can potentially do a better job than more conventional methods, so long as they're provided with enough content. Another aspect of context that could be relevant to your work may be how people are nested within households. In some record linkage scenarios you can run a linkage on household membership and use this to make person linkages more precise. I imagine an LLM agent could also use this information effectively if it were explicitly provided, but sometimes they miss logical leaps like this

satvikpendem 10d ago

Maybe you need Prolog instead.