4 ms·

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a trans

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a transport such as Avro, for example). While in many cases that is somebody else's problem -- it is solved in a library -- when it isn't the strengths and facility of a dynamic language can save you considerable code complexity and maintenance. Type control is often central to reporting as well, and it is, again, more awkward in a strong typing context. I would tend to argue that insistence upon type frameworks such as pydantic in a data engineering framework is naive and imposed by academic rather than industry experience. There is a reason that python is chosen for data processing applications, and it certainly isn't typing.

waffletower 2y ago
Giving this some more thought: I do know that Nike has a revolving door for developers. It seems that a framework like Koheesio allows Nike to essentially hire for scala from a pool of candidates that only have python experience. Once hired, as they use pyspark and koheesio daily, they don't even know they are scala developers. Much easier to hire/fire python developers these days.
IshKebab 2y ago
Python is chosen for data processing because it's one of the most popular languages in the world and it has a passable REPL (which is more than most languages) so you can use it for experimentation. From the readme they describe it as "robust" and having a high level of type safety, so I'm guessing they're just leaning towards the "learn about bugs up before they hit production" end of the spectrum than you. Then again I don't do any of this data engineering stuff so maybe it doesn't matter too much if it doesn't work reliably?
sixdimensional 2y ago
You're right, but this problem doesn't go away when you use Python - because you still have to interact with the type systems of the platforms you are integrating. It's a much more difficult and nuanced problem than I think most people realize.
AtlasBarfed 2y ago
"the report just has to look right"

waffletower 2y ago

Giving this some more thought: I do know that Nike has a revolving door for developers. It seems that a framework like Koheesio allows Nike to essentially hire for scala from a pool of candidates that only have python experience. Once hired, as they use pyspark and koheesio daily, they don't even know they are scala developers. Much easier to hire/fire python developers these days.

IshKebab 2y ago

Python is chosen for data processing because it's one of the most popular languages in the world and it has a passable REPL (which is more than most languages) so you can use it for experimentation. From the readme they describe it as "robust" and having a high level of type safety, so I'm guessing they're just leaning towards the "learn about bugs up before they hit production" end of the spectrum than you. Then again I don't do any of this data engineering stuff so maybe it doesn't matter too much if it doesn't work reliably?

sixdimensional 2y ago

You're right, but this problem doesn't go away when you use Python - because you still have to interact with the type systems of the platforms you are integrating. It's a much more difficult and nuanced problem than I think most people realize.

AtlasBarfed 2y ago

"the report just has to look right"