3 ms·

I'm also a software engineer, though I've had the unofficial title "data engineer" applied to me for quite some time now. The more I work with tools like Spark

I'm also a software engineer, though I've had the unofficial title "data engineer" applied to me for quite some time now. The more I work with tools like Spark, the more dissatisfied I become with the data engineering world. Spark is a hot mess of fiddling with configuration - I've lost more productivity to dealing with memory limit, executor count, and other configuration than I think is reasonable. Pandas is another one. It was good enough to make quick processing concise that it got significant uptake and became de facto. The API is a pain, though, and processing is slow. Now, couple Pandas and Spark in your day-to-day job and you get what I see from my data science colleagues: "I'll hack together some processing in Pandas until my machine can't handle any more data, at which point I'll throw it into Spark and provision a bunch of nodes to do that work for me." I don't mean that to sound pejorative, as they're generally just trying to be productive, but there's so little attention paid to real efficiency in the frameworks and infrastructure that we're blowing through compute, memory, and person-hours unnecessarily (IMHO).

rectang 4y ago
> Pandas is another one. But at least if I write a transform in pandas it's straightforward to unit test it: create a DataFrame with some dummy data, send it through the function which wraps the transform, test that what comes out is what's expected. Validating a transform done in SQL is not nearly as straightforward. For starters it needs to be an integration test not a unit test (because you need a database). And that's assuming there's even a way to hook unit tests into your framework. I'm not a huge fan of Pandas — it's way too prone to silent failure. I've written wrappers around e.g. read_csv which are there to defeat all the magic. But at least I can do that with Python code instead of being stuck with the big-ball-of-SQL (e.g. some complicated view SELECT statement that does a bunch of JOINs, CASE tests and casts).
FridgeSeal 4y ago
Spark is the bane of my existence. Sure, you can spin out multiple executors and have it distribute work amongst them… Provided you figure out the 3 million configs necessary to do this in a reasonable way, so our dependency, networking and storage issues, only for the runtime to decide that actually, it’d quite like to run your entire workload on only 1 of your machines, or something equally frustrating. Given how frustrating it is to operate on, I’m shocked it has gained as much popularity as it has. I sort of wonder if that’s a bit by design from vested commercial interests- make it simultaneously exceedingly popular and so convoluted to run that you are basically obliged to pay for the likes of databricks just to make your life not awful. Of course, once you’ve done this, you’ve now bought into a whole suite of other issues, but that’s a different discussion…

rectang 4y ago

> Pandas is another one. But at least if I write a transform in pandas it's straightforward to unit test it: create a DataFrame with some dummy data, send it through the function which wraps the transform, test that what comes out is what's expected. Validating a transform done in SQL is not nearly as straightforward. For starters it needs to be an integration test not a unit test (because you need a database). And that's assuming there's even a way to hook unit tests into your framework. I'm not a huge fan of Pandas — it's way too prone to silent failure. I've written wrappers around e.g. read_csv which are there to defeat all the magic. But at least I can do that with Python code instead of being stuck with the big-ball-of-SQL (e.g. some complicated view SELECT statement that does a bunch of JOINs, CASE tests and casts).

FridgeSeal 4y ago

Spark is the bane of my existence. Sure, you can spin out multiple executors and have it distribute work amongst them… Provided you figure out the 3 million configs necessary to do this in a reasonable way, so our dependency, networking and storage issues, only for the runtime to decide that actually, it’d quite like to run your entire workload on only 1 of your machines, or something equally frustrating. Given how frustrating it is to operate on, I’m shocked it has gained as much popularity as it has. I sort of wonder if that’s a bit by design from vested commercial interests- make it simultaneously exceedingly popular and so convoluted to run that you are basically obliged to pay for the likes of databricks just to make your life not awful. Of course, once you’ve done this, you’ve now bought into a whole suite of other issues, but that’s a different discussion…