albertstanley

4 ms·

1. 1.▲
   by albertstanley 3y ago

   Sure, I’ll clarify some of the terms used in that one-liner in case it’s helpful for anyone else as well. ETL is the process of extracting transforming and loading data from a source to a destination in a data pipeline. Spark, an engine for
2. 2.▲
   by albertstanley 3y ago

   Sure, our approach is to define Python classes to handle reusable steps for reading, transforming or loading data. For example, we have a MapTransformer, CastColumnsTransformer, GeoDistanceTransformer. Each class specifies some configuratio
3. 3.▲
   by albertstanley 3y ago

   This is a completely valid point, we'll be changing the readers to directly read into Spark. Thank for the comment!