Have you done much data analysis? In my experience adding more steps to your pipeline (E.g. database, deserializing, etc) is a pain when you are figuring thing

Have you done much data analysis? In my experience adding more steps to your pipeline (E.g. database, deserializing, etc) is a pain when you are figuring things out because nothing has solidified yet, so you're literally just adding overhead that requires even more work to remove/alter later on. If you're not careful you end up with something unmaintainable extremely quickly. Only analyzing a subset of your data is usually not a magic bullet either. Unless your data is extremely well cleaned and standardized you're probably going to run into edge cases on the full dataset that were not in your subset. Being able to run your full pipeline on the entire dataset in a short period of time is very useful for testing on the full dataset and seeing realistic analysis results. If you're doing any sort of aggregate analysis it becomes even more important, if not required. I now believe a relatively fast clean run is one of the most important things for performing data analysis. It increases your velocity tremendously.