3 ms·
If every time you do it is a one-off, yes. If your business is essentially nothing but aggregating datasets (which may themselves be aggregates of still more da
by cratermoon 5y ago
If every time you do it is a one-off, yes. If your business is essentially nothing but aggregating datasets (which may themselves be aggregates of still more datasets), then by the third time you've solved the exact problem, you're going to see commonalities. You abstract those out so that the fourth, fifth, and sixth problems don't have to re-write that part. After a while, you've built out enough abstractions that you're programming the migration process itself, not each migration.
Add in the fact that your business depends on doing it at scale, ingesting live data as it is generated, and developing an entity resolution system is almost a no-brainer.