2 ms·
I work on an ETL service that interacts with a good few, semi diverse third party systems. We organize these by individual jobs that get run and a job will proc
by jjice 3y ago
I work on an ETL service that interacts with a good few, semi diverse third party systems. We organize these by individual jobs that get run and a job will process certain types of data and spawn different sub jobs in specific situations.
We wanted to have a compile time map of this data and a graph of the sub jobs. To do this, I tossed together a thirty minute script that took the source of our job functions and then ran a few regular expressions on them to extract the data we needed. It was filthy. Regex on source files just feels wrong. Problem is, it's worked great for the last six months and it's still going strong, so we can't justify changing it. The generated data has been extremely valuable for optimizing data fetching paths for customers.