4 ms·
Makes sense for low-entropy data. Though I can see that approach choking on some datasets. What happens if every entry has a GUID, for example? May be better t
by TheLoneWolfling 11y ago
Makes sense for low-entropy data. Though I can see that approach choking on some datasets. What happens if every entry has a GUID, for example?
May be better to do a best-effort deduplication instead of an exhaustive approach.
- victorNicollet 11y agoAt some point, we considered tweaking by dropping any strings longer than a certain length from the deduplication (it also helps with memory usage when streaming the data). Our method makes most sense for many-to-many data (several orders per product, several orders per day), which happens to be the largest data sets we manipulate (by 3 orders of magnitude). I can certainly see situations where this would not be the case (e.g. web crawler logs).