In my case it lead to a reverse on that reasoning. The script ended up being an explicit step to import the data when it changed instead of running implicitly w

In my case it lead to a reverse on that reasoning. The script ended up being an explicit step to import the data when it changed instead of running implicitly whenever the data was needed. Of course that choice was based on the initial version which was even slower and didn't use multiprocessing to handle the files in parallel.