3 ms·
> There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thi
by skohan 5y ago
> There are strategies like entity resolution[2] that attempt to take items from dataset A and items from dataset B and figure out if they refer to the same thing
The more code I have written, the more convinced I become that "general" solutions almost never pay for themselves when you compare the implementation and/or performance complexity required to make them work.
It's almost always better to "just write the damn code" and solve the exact problem you actually have.
- cratermoon 5y agoIf every time you do it is a one-off, yes. If your business is essentially nothing but aggregating datasets (which may themselves be aggregates of still more datasets), then by the third time you've solved the exact problem, you're going to see commonalities. You abstract those out so that the fourth, fifth, and sixth problems don't have to re-write that part. After a while, you've built out enough abstractions that you're programming the migration process itself, not each migration. Add in the fact that your business depends on doing it at scale, ingesting live data as it is generated, and developing an entity resolution system is almost a no-brainer.
- arthur_sav 5y agoYes and no. They don't pay if you try to generilize before you have a problem. If you have a frequent use case that follows the same patterns, then you probably can benefit from an abstraction layer. But trying to build an abstraction layer before even having a use case then yes, it usually doesn't pay off.