3 ms·
What was the point of creating the "canonical form" if you already had reports being generated in-app? Was it just someone's pet project, or were there supposed
by icedchai 4mo ago
What was the point of creating the "canonical form" if you already had reports being generated in-app? Was it just someone's pet project, or were there supposed to be other benefits?
- ealready_value 4mo agoI've not gotten a straight answer. I assume it is a pet project kind of situation, or trying to justify the data warehouse project as a whole, but I really don't know the real driver to do this.
- icedchai 4mo agoThese sorts of odd projects are relatively common. A few years ago I was brought on near the end of a data engineering project where somebody had decided they needed multiple databases, a crap load of JSON exports, and dozens of python, R, and shell scripts running inside some job orchestrator to support what amounted to a few megabytes of data being processed each day. Maybe 5 megabytes max. There wasn't even a lot of transformation going on. It was just... strange. I witnessed some true eldritch horrors like Python calling R calling a shell script that called the mysql client, which wrote data to a temporary file that was eventually read by the great-grand-parent python script.