3 ms·
I think the ultimate cause of this kind of inconsistency is that the panopticon-style correlation of customer data is happening offline in a business intelligen
by jcrawfordor 5y ago
I think the ultimate cause of this kind of inconsistency is that the panopticon-style correlation of customer data is happening offline in a business intelligence system that has a one-way, stale data feed from various ETLs, while making a return happens in an online system that doesn't get anything back from the analytics system (because the analytics data is too loosely structured and low-quality to feed it back into a line of business system).
This kind of dichotomy exists in just about any organization that gets into BI/data science/whatever you want to call it, and it can be absolutely maddening. The state of ETLs and analytics systems, for example, means that we have basically gone back to "overnight batch processing" and a lot of reporting data only becomes available the day after the transaction, since all the ETLs need to run and then whatever analytics system, data-cube assembly or etc. But since the analytics system exists, it is taken as good enough and no one wants to re-implement the functionality in an online, system-of-record fashion.
Or another way to put it: the GP says "The second you walk into the store based on your face they can pull up your entire purchase history and contact." That's being pretty generous... most of the time they don't know any of that until an hour or a day later, because it comes from multiple disparate systems (and outside vendors) and importing that data into a data warehouse (in batches) and then correlating it later is far easier to implement, and the dominant method used in this kind of analytics.