4 ms·

In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not. Multiple legacy systems with

by mswen 10y ago

In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not. Multiple legacy systems with no consistent cross reference to unambiguously identify the same customer. Assured that systems have been gone through and all the names made consistent. Consistency for a human is not consistency for a computer. "Commers Ltd" is not the same as "Commers Ltd." And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Business process discipline is often lacking and will mess you up. Subscription data sources that change their schema with no notification to paying customers. And, when you are scraping data from websites you need to constantly be checking that your scrapers are still working properly. Source websites change regularly. Crazy processes like entering a negative invoice to indicate a refund to customers but forgetting to zero out the cost of goods related to the invoice. We may have refunded the money but we didn't do the work twice. Arggh! Errors abound.

marcosdumay 10y ago
> And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Good luck modeling data to not need this. I have caught myself actively telling people to do this more than once, and I'm really thinking about replicating some data so those changes are less disrupting. That "Commers Ltd Dallas" probably has differing billing and delivery addresses, points of contact, invoice formats, customers representatives and preferred sellers, product selection, and probably everything else you have on your DB.
- zo1 10y ago
  It's really the only option if your input/managing software can not model the complexity you require "after the fact". E.g. Company has multiple offices. Delivery address is per office, not per company, etc. The real problem is when there is chaotic, and organic mixing, matching and re-purposing. I've seen it many times with "non-technical" individuals. They don't know what their software can do. E.g. Redmine. So the support individuals just log everything under the same "IssueType. They then "categorize" it using Category custom field, instead of the standard category which has enumerations. And then they then use that Category field to drive reports/process. Instead of using a different IssueType or Tracker, which is what it was designed for, and has tools that help you leverage/manage the complexity of different standardized processes. Then, they decide to to add "Sub-categories" into the category field, instead of using a project-hierarchy or something. Then they want to do billing reports from the time logged per X and of types A,B,C, and at that point it's a giant mess and I stop caring. If they want to not use the software as intended, then do "fixing" by filtering and fiddling with Redmine CSV exports in Excel afterwards, that's their problem. Oh, and they ask that everyone has permissions to everything, allowing all users to change the status of each IssueType as they please, without any process. I just feel sorry for the poor individual that get's a raw extract of that data and has to use it for something.
- mswen 10y ago
  That is fine but don't later ask why the system has 10 different customers for Commers Ltd I ended up building another table and logic just to do roll-ups and account for name variations. But I told the client that they really need to invest in a serious cross-referencing middleware that tracks identities across all these systems and uses ID numbers to coordinate all the legacy systems. But yes trying to develop models or even simple aggregations that rely on this kind of data can be quite frustrating.

by mswen 10y ago

marcosdumay 10y ago

> And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Good luck modeling data to not need this. I have caught myself actively telling people to do this more than once, and I'm really thinking about replicating some data so those changes are less disrupting. That "Commers Ltd Dallas" probably has differing billing and delivery addresses, points of contact, invoice formats, customers representatives and preferred sellers, product selection, and probably everything else you have on your DB.

zo1 10y ago
It's really the only option if your input/managing software can not model the complexity you require "after the fact". E.g. Company has multiple offices. Delivery address is per office, not per company, etc. The real problem is when there is chaotic, and organic mixing, matching and re-purposing. I've seen it many times with "non-technical" individuals. They don't know what their software can do. E.g. Redmine. So the support individuals just log everything under the same "IssueType. They then "categorize" it using Category custom field, instead of the standard category which has enumerations. And then they then use that Category field to drive reports/process. Instead of using a different IssueType or Tracker, which is what it was designed for, and has tools that help you leverage/manage the complexity of different standardized processes. Then, they decide to to add "Sub-categories" into the category field, instead of using a project-hierarchy or something. Then they want to do billing reports from the time logged per X and of types A,B,C, and at that point it's a giant mess and I stop caring. If they want to not use the software as intended, then do "fixing" by filtering and fiddling with Redmine CSV exports in Excel afterwards, that's their problem. Oh, and they ask that everyone has permissions to everything, allowing all users to change the status of each IssueType as they please, without any process. I just feel sorry for the poor individual that get's a raw extract of that data and has to use it for something.
mswen 10y ago
That is fine but don't later ask why the system has 10 different customers for Commers Ltd I ended up building another table and logic just to do roll-ups and account for name variations. But I told the client that they really need to invest in a serious cross-referencing middleware that tracks identities across all these systems and uses ID numbers to coordinate all the legacy systems. But yes trying to develop models or even simple aggregations that rely on this kind of data can be quite frustrating.