3 ms·

from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was

by trhway 5y ago

from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was working for tens of millions of entities on regular, though beefy, RDBMS more than a decade ago) - the issue isn't that much technological, it is that each customer/client has custom everything when it comes to ER, and thus scaling that business is extremely hard (that specific business collapsed primarily for that reason)

Major_Grooves 5y ago
the real technical challenge is the "transitive hop" problem that we describe. The matching of the data is not so complicated - that can be done with any technology - but searching with Data A, and getting result Z - that was the tricky bit that took us years to solve and was only possible thanks to serverless tech.
- trhway 5y ago
  in our case we didn't have explicit "transitive hop". Instead in this example it would be just one entity with a bunch of addresses (with related dates if configured so), and multiple known names, etc. attached. Granted in order to get to that state the data loading process would include a massive batch matching which would be performed on the array of worker nodes - the number could be configured for given run, so there is kind of rough technological similarity to your serverless.
  - skafoi 5y ago
    That sounds mostly like deduplication which is often used in marketing contexts. There are indeed some good solutions out there, but from our experience they have difficulties handling huge amounts of data (>1 billion data sets) and they are often batch based, so your data is always outdated, whereas we constantly add new data in near real-time.
    - trhway 5y ago
      Typically deduplication is more like clean, standartize and match (using relatively simple match approaches). Our process was much more complex (more along the lines of what another commenter linked https://blog.acolyer.org/2020/12/14/entity-resolution/ https://blog.acolyer.org/2020/12/14/entity-resolution/) with more rich resulting functionality, and, yes, marketing was the segment where it started though i'd say it was only about 3rd of the business at the peak of it. Yep, performance for most of the solutions is an issue. We had to do significant re-engineering at one point to parallelize at much finer granularity and thus were able to scale much more. Our biggest number of entities counting across all the sources in the largest implementation was just under 200M. It was right pre-Nehalem hardware. The batch mode had naturally orders of magnitude higher throughput. We did have real-time single-record mode which was pretty fast as long as the stream of the incoming single-records wouldn't saturate the worker array capacity (here is the difference from serverless as the worker array was limited by whatever was statically configured at the moment as adding/removing nodes wasn't an instant on the fly operation) Couple years later i worked at another company on a similar, though somewhat simpler, project when it was in the process of total rewrite for performance reason - the old version was really slow - that rewrite failed spectacularly for a lot of reasons. So, yes, performance is a kind of a noticeable factor in the domain.
skafoi 5y ago
I would really love to hear more about your experience regarding the client customizations. So far the two things I can see are domain model customization and rule customization. Obviously with the rule customizations being the more challenging one.
- trhway 5y ago
  the rule customization is match rules and index generation rules, and all these customizations are data source specific. And a large company with a lot of departments and divisions, some of them being former acquisitions, would have a number of different data sources, as well some external ones too, reference ones in particular. Beside the pure basic issue of connecting those data sources and pulling data from it, the data in them have different reliability/quality, different standard of maintenance, etc. as well as different role in the system - some of those sources are master sources, some are only used to extract matches without update, some are [also] consumers of the match/deduplication results, so they have to be updated back. As as result any of such actually implemented ER systems is totally one-off.
  - skafoi 5y ago
    Thanks a lot. Sounds similar to the issues we had in our previous company. We provide some basic ETL functionalities to tackle these. Will be interessting to see where this will eventually end up and how much our solution does there or where it is better to use the probably in those companies existing ETL tools.

by trhway 5y ago

Major_Grooves 5y ago

the real technical challenge is the "transitive hop" problem that we describe. The matching of the data is not so complicated - that can be done with any technology - but searching with Data A, and getting result Z - that was the tricky bit that took us years to solve and was only possible thanks to serverless tech.

trhway 5y ago
in our case we didn't have explicit "transitive hop". Instead in this example it would be just one entity with a bunch of addresses (with related dates if configured so), and multiple known names, etc. attached. Granted in order to get to that state the data loading process would include a massive batch matching which would be performed on the array of worker nodes - the number could be configured for given run, so there is kind of rough technological similarity to your serverless.
- skafoi 5y ago
  That sounds mostly like deduplication which is often used in marketing contexts. There are indeed some good solutions out there, but from our experience they have difficulties handling huge amounts of data (>1 billion data sets) and they are often batch based, so your data is always outdated, whereas we constantly add new data in near real-time.
  - trhway 5y ago
    Typically deduplication is more like clean, standartize and match (using relatively simple match approaches). Our process was much more complex (more along the lines of what another commenter linked https://blog.acolyer.org/2020/12/14/entity-resolution/ https://blog.acolyer.org/2020/12/14/entity-resolution/) with more rich resulting functionality, and, yes, marketing was the segment where it started though i'd say it was only about 3rd of the business at the peak of it. Yep, performance for most of the solutions is an issue. We had to do significant re-engineering at one point to parallelize at much finer granularity and thus were able to scale much more. Our biggest number of entities counting across all the sources in the largest implementation was just under 200M. It was right pre-Nehalem hardware. The batch mode had naturally orders of magnitude higher throughput. We did have real-time single-record mode which was pretty fast as long as the stream of the incoming single-records wouldn't saturate the worker array capacity (here is the difference from serverless as the worker array was limited by whatever was statically configured at the moment as adding/removing nodes wasn't an instant on the fly operation) Couple years later i worked at another company on a similar, though somewhat simpler, project when it was in the process of total rewrite for performance reason - the old version was really slow - that rewrite failed spectacularly for a lot of reasons. So, yes, performance is a kind of a noticeable factor in the domain.

skafoi 5y ago

I would really love to hear more about your experience regarding the client customizations. So far the two things I can see are domain model customization and rule customization. Obviously with the rule customizations being the more challenging one.

trhway 5y ago
the rule customization is match rules and index generation rules, and all these customizations are data source specific. And a large company with a lot of departments and divisions, some of them being former acquisitions, would have a number of different data sources, as well some external ones too, reference ones in particular. Beside the pure basic issue of connecting those data sources and pulling data from it, the data in them have different reliability/quality, different standard of maintenance, etc. as well as different role in the system - some of those sources are master sources, some are only used to extract matches without update, some are [also] consumers of the match/deduplication results, so they have to be updated back. As as result any of such actually implemented ER systems is totally one-off.
- skafoi 5y ago
  Thanks a lot. Sounds similar to the issues we had in our previous company. We provide some basic ETL functionalities to tackle these. Will be interessting to see where this will eventually end up and how much our solution does there or where it is better to use the probably in those companies existing ETL tools.