3 ms·

>ETL / data transformations. Python is a big player here - but for stable and performant data pipelines, I strongly believe Scala is the better choice. For expl

>ETL / data transformations. Python is a big player here - but for stable and performant data pipelines, I strongly believe Scala is the better choice. For explorative things, python has the edge though. I disagree. And I've done a lot of Scala ETL work. The issue is that in Scala ETL work your data structures are either untyped (Spark Dataframe), case classes or typed tuples. If they're untyped then there's little to gain from Scala vs. Python in terms of type safety and a lot of overhead. If they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself. If it's typed tuples then it's on you to remember which field is what as nothing is named. Some sort of intelligent compile time case class subset generation would have helped the situation immensely. I think frameless (https://github.com/typelevel/frameless https://github.com/typelevel/frameless) gives you that power but it's a lot of shapeless black magic overhead and I've never seen it used in production (and probably various maximum field limitations and much slower compile times). edit: Case classes also have (or had) a steeply increasing compile time memory requirement as you add more fields which is fun when your data is 200+ fields long. Almost as fun as having to make a 200 field case class to read in a single file of which you only need 50 fields but want to be type safe. edit2: Scala is popular in ETL because the Hadoop ecosystem is JVM and Java used to be atrocious. So Spark (which wanted Hadoop compatibility) picked Scala because it was better than the alternatives. However nowadays Databricks is investing a lot more into their Python support from what I can tell and there's Python native competitors out there (Dask, Ray, etc.).

codygman 6y ago
> they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself. Haskell developer here... What about case classes and lenses? Do they solve this?
- marcinzm 6y ago
  As I understand it lenses don't change the underlying data structure. For ETL you need a way to basically say "the code only uses fields X, Y and Z so we will only load X, Y and Z during runtime." Automatically based on usage without having to keep updating your lens definition. Modern on disk file formats are columnar so they can very efficiently read subsets of the data. If your data has 200 columns than reading the 199 unnecessary ones can be very slow. They could help with the intermediate data structure but some of them aren't subsets or trivial derivatives. So you really need an inline way to create single use case classes. I think frameless in Scala can do some of this for standard transformations but that requires the black magic of shapeless. Spark in Python (and the untyped Dataframe API in Scala) compiles everything internally before running it to achieve the above. So it's trivial to have unit tests on empty data structures which "type check" your Spark code.
funcDropShadow 6y ago
Another possibility is to use "typesafe heterogenuous containers" as described in Joshua Bloch, "Effective Java", 2nd edition and probably 3rd edition. The idea is to use maps with specially constructed keys, that give you type safe access to their accompanied values. That is better than completely untyped, you have a place to document stuff at the declaration of those keys and you can easily create runtime checks, that check for the presence of a set of well defined keys in the map. The fact that they are described in a Java book does not mean that they aren't a nice pattern in Scala and several other statically typed language with generics.

codygman 6y ago

> they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself. Haskell developer here... What about case classes and lenses? Do they solve this?

marcinzm 6y ago
As I understand it lenses don't change the underlying data structure. For ETL you need a way to basically say "the code only uses fields X, Y and Z so we will only load X, Y and Z during runtime." Automatically based on usage without having to keep updating your lens definition. Modern on disk file formats are columnar so they can very efficiently read subsets of the data. If your data has 200 columns than reading the 199 unnecessary ones can be very slow. They could help with the intermediate data structure but some of them aren't subsets or trivial derivatives. So you really need an inline way to create single use case classes. I think frameless in Scala can do some of this for standard transformations but that requires the black magic of shapeless. Spark in Python (and the untyped Dataframe API in Scala) compiles everything internally before running it to achieve the above. So it's trivial to have unit tests on empty data structures which "type check" your Spark code.

funcDropShadow 6y ago

Another possibility is to use "typesafe heterogenuous containers" as described in Joshua Bloch, "Effective Java", 2nd edition and probably 3rd edition. The idea is to use maps with specially constructed keys, that give you type safe access to their accompanied values. That is better than completely untyped, you have a place to document stuff at the declaration of those keys and you can easily create runtime checks, that check for the presence of a set of well defined keys in the map. The fact that they are described in a Java book does not mean that they aren't a nice pattern in Scala and several other statically typed language with generics.