3 ms·
This mirrors how many big data / data warehousing pipelines work. Originally a lot of transformations tended to get pulled into the same flow as moving data aro
by skeeter2020 2y ago
This mirrors how many big data / data warehousing pipelines work. Originally a lot of transformations tended to get pulled into the same flow as moving data around, but storage has gotten progressively cheaper and this is so much simpler and durable.
- hiAndrewQuinn 2y agoThat's essentially why I gravitated to it, yeah. To me, Git submodules are a pretty generalized and elegant solution to the question of "how do I model a directory of files as an immutable data structure". If you know the commit hash of the submodule, you know exactly what is going to be in there. That kind of certainty is very helpful when trying to code up tricky data cleaning and the like.