7 ms·

I'm a software dev who's been bumping up against the data engineering field lately, and I've been dismayed as to how many popular tools shunt you towards unmain

I'm a software dev who's been bumping up against the data engineering field lately, and I've been dismayed as to how many popular tools shunt you towards unmaintainable, unevolvable system design. - A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. - Poor support for unit testing. - Poor support for version control. - Frameworks over libraries (because the vendors want to lock you in). > data engineers have been increasingly adopting software engineering best practices We can only hope. I think it's more likely that in the near term data engineers will get better and better at prototyping within low-code frameworks, and that transitioning from the prototype to an evolvable system will get harder.

dang 4y ago
Please stick to plain text - formatting gimmicks are distracting, not the convention here, and tend to start an arms race of copycats. I've put hyphens there now. Your comment is otherwise fine of course.
- nmarinov 4y ago
  Out of curiosity, could you give or point to an example of "formatting gimmicks"? I tried searching the FAQ but only found the formatdoc guide: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
  - rectang 4y ago
    OT... I used one of the Unicode bullets instead of a hyphen (◉). I understand what dang is getting at here — it's in the same spirit as stripping emojis (because otherwise we'd be drowning in them). I'm a past graphic designer so I'm accustomed to working with the medium to achieve maximum communicative impact, but I don't mind operating within constraints. The convention of using hyphens to start a bulleted list isn't officially documented AFAIK. Having to separate bulleted list items with double newlines is a little weird, but it's fine.
    - dang 4y ago
      Thanks for the thoughtful response. You don't have to use hyphens for that. Asterisks are fine (see https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc for how to get a *) and no doubt there are other ways within the plaintext convention.
MrPowers 4y ago
> A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. Yea, thankfully some frameworks have Python / Scala APIs that let you abstract "SQL logic" into programatic functions that can be chained and reused independently to avoid the big-ball-of-SQL problem. The best ones also allow for SQL because that's the best way to express some logic. > Poor support for unit testing. I've written pandas, PySpark, Scala Spark, and Dask testing libs. Not sure which framework you're referring to. > Poor support for version control. The execution platform should make it easy to package up code in JAR files / Python wheel files and attach them to your cluster, so you can version control the business logic. If not, yea, that's a huge limitation. > Frameworks over libraries (because the vendors want to lock you in) Not sure what you're referring to, but interested in learning more.
- forgetfulness 4y ago
  They were talking about the "modern data stack" no doubt. The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark. It makes it very accessible to write data pipelines then using dbt (which outcompeted Dataform, though the latter is still kicking), but then you don't have the richer programming facilities, stricter type systems, tooling and the practices of Python or Scala programming, you're in the world of SQL, set back decade or two in testing, checking, and a culture of using them, and with little tools to organize your code. That, if the team has rebuked the siren songs of a myriad of cloud, low-code platforms for this or the other, with even fewer facilities to keep the data pipelines under control, be it that we call control any of: access, versioning, monitoring, data quality, anything really.
  - MrPowers 4y ago
    > The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark. I feel like there there are some data professionals that only want to use SQL. Other data professionals only want to use Python. I feel like the trend is to provide users with interfaces that let them be productive. I could be misreading the trend of course.
mynameisash 4y ago
I'm also a software engineer, though I've had the unofficial title "data engineer" applied to me for quite some time now. The more I work with tools like Spark, the more dissatisfied I become with the data engineering world. Spark is a hot mess of fiddling with configuration - I've lost more productivity to dealing with memory limit, executor count, and other configuration than I think is reasonable. Pandas is another one. It was good enough to make quick processing concise that it got significant uptake and became de facto. The API is a pain, though, and processing is slow. Now, couple Pandas and Spark in your day-to-day job and you get what I see from my data science colleagues: "I'll hack together some processing in Pandas until my machine can't handle any more data, at which point I'll throw it into Spark and provision a bunch of nodes to do that work for me." I don't mean that to sound pejorative, as they're generally just trying to be productive, but there's so little attention paid to real efficiency in the frameworks and infrastructure that we're blowing through compute, memory, and person-hours unnecessarily (IMHO).
- rectang 4y ago
  > Pandas is another one. But at least if I write a transform in pandas it's straightforward to unit test it: create a DataFrame with some dummy data, send it through the function which wraps the transform, test that what comes out is what's expected. Validating a transform done in SQL is not nearly as straightforward. For starters it needs to be an integration test not a unit test (because you need a database). And that's assuming there's even a way to hook unit tests into your framework. I'm not a huge fan of Pandas — it's way too prone to silent failure. I've written wrappers around e.g. read_csv which are there to defeat all the magic. But at least I can do that with Python code instead of being stuck with the big-ball-of-SQL (e.g. some complicated view SELECT statement that does a bunch of JOINs, CASE tests and casts).
- FridgeSeal 4y ago
  Spark is the bane of my existence. Sure, you can spin out multiple executors and have it distribute work amongst them… Provided you figure out the 3 million configs necessary to do this in a reasonable way, so our dependency, networking and storage issues, only for the runtime to decide that actually, it’d quite like to run your entire workload on only 1 of your machines, or something equally frustrating. Given how frustrating it is to operate on, I’m shocked it has gained as much popularity as it has. I sort of wonder if that’s a bit by design from vested commercial interests- make it simultaneously exceedingly popular and so convoluted to run that you are basically obliged to pay for the likes of databricks just to make your life not awful. Of course, once you’ve done this, you’ve now bought into a whole suite of other issues, but that’s a different discussion…
bobbruno 4y ago
Let me see if I can at least justify some of these things: - SQL became the de facto standard for data manipulation a few decades ago, and I’m still to see a worthy contender. While the language itself shows its age, the concept of it being (almost) totally declarative of “what” data to get and “what” to do with it, instead of “how”, is unbeatable. A better syntax for achieving this is certainly possible, but you won’t beat decades of SQL code being generated easily. About “get the answer right once”, for analysis it’s often all that’s needed, and good SW maintence doesn’t apply that hard (it just won’t be needed again). For actual repeatable data engineering, I’d suggest either using generated SQL or writing in some abstraction (like PySpark). But don’t underestimate the productivity of SQL and the ability of modern parsers; - Unit testing: while I generally agree, after 28 years, I have to say that unit testing is not as useful in data engineering. Essentially, it doesn’t matter how much time and effort you put on designing test cases, you’ll never beat the chaotic creativity of real users making real systems ingest all kinds of wrong data in patterns you’d never imagine. Now multiply that by having to cross-reference data from systems that were designed independently, and think of all the possible data errors that might come out of that. You won’t, you won’t cover even 20% of the ways things can go wrong. So, instead of “testing for all ways things go wrong”, a much better pattern is “write code that can capture unexpected errors and make reasonable decisions on whether to continue or abort - with some nice messaging and data debugging, please”; - Poor support for version control: I 100% agree with you. DataOps is still not nearly as popular as it should be; - Frameworks over libraries: let me tell you one thing - data is coupled. Uncoupled data has less value than coupled data, because it can’t answer more complex questions. Having different libraries that represent, manipulate and expose data in different ways will just make things much harder for the DE trying to generate the data. And the focus is handling the data, not the code. So, while you may have a bit of a point, I’d suggest that you’re looking at this from the wrong perspective; Low-code: it’s one of those things that come and go. When I started, I’d use Oracle’s PL/SQL and C. Then came the first generation of visual ETL tools (Informatica, Datastage, etc) and they looked great with the graphic flows. Until you actually had to use them and all the clicking just to get to a point where you could actually add some logic showed. Then came the Hadoop era, and data engineering split into the DW people (who prefer visual, low-code to this day) and the Hadoop people, who went back to coding. This split stays to this day, with each side not recognising what the other does as essentially the same thing. Personally, I prefer code, but I don’t care what others prefer. The visual x coding discussion is, in my opinion, the wrong problem. The real issue is being able to create logic that’s clear enough that something as dumb as a computer will unambiguously understand. STEM learning environments for kids show that this is totally possible in a visual way - the thing is, most companies choose the visual tool in hopes that a person without the right mindset will generate good logic with it. That’s what fails. The rest is a matter of preferred UX - relevant, but not core or one-sided.

dang 4y ago

Please stick to plain text - formatting gimmicks are distracting, not the convention here, and tend to start an arms race of copycats. I've put hyphens there now. Your comment is otherwise fine of course.

nmarinov 4y ago
Out of curiosity, could you give or point to an example of "formatting gimmicks"? I tried searching the FAQ but only found the formatdoc guide: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- rectang 4y ago
  OT... I used one of the Unicode bullets instead of a hyphen (◉). I understand what dang is getting at here — it's in the same spirit as stripping emojis (because otherwise we'd be drowning in them). I'm a past graphic designer so I'm accustomed to working with the medium to achieve maximum communicative impact, but I don't mind operating within constraints. The convention of using hyphens to start a bulleted list isn't officially documented AFAIK. Having to separate bulleted list items with double newlines is a little weird, but it's fine.
  - dang 4y ago
    Thanks for the thoughtful response. You don't have to use hyphens for that. Asterisks are fine (see https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc for how to get a *) and no doubt there are other ways within the plaintext convention.

MrPowers 4y ago

> A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. Yea, thankfully some frameworks have Python / Scala APIs that let you abstract "SQL logic" into programatic functions that can be chained and reused independently to avoid the big-ball-of-SQL problem. The best ones also allow for SQL because that's the best way to express some logic. > Poor support for unit testing. I've written pandas, PySpark, Scala Spark, and Dask testing libs. Not sure which framework you're referring to. > Poor support for version control. The execution platform should make it easy to package up code in JAR files / Python wheel files and attach them to your cluster, so you can version control the business logic. If not, yea, that's a huge limitation. > Frameworks over libraries (because the vendors want to lock you in) Not sure what you're referring to, but interested in learning more.

forgetfulness 4y ago
They were talking about the "modern data stack" no doubt. The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark. It makes it very accessible to write data pipelines then using dbt (which outcompeted Dataform, though the latter is still kicking), but then you don't have the richer programming facilities, stricter type systems, tooling and the practices of Python or Scala programming, you're in the world of SQL, set back decade or two in testing, checking, and a culture of using them, and with little tools to organize your code. That, if the team has rebuked the siren songs of a myriad of cloud, low-code platforms for this or the other, with even fewer facilities to keep the data pipelines under control, be it that we call control any of: access, versioning, monitoring, data quality, anything really.
- MrPowers 4y ago
  > The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark. I feel like there there are some data professionals that only want to use SQL. Other data professionals only want to use Python. I feel like the trend is to provide users with interfaces that let them be productive. I could be misreading the trend of course.

mynameisash 4y ago

I'm also a software engineer, though I've had the unofficial title "data engineer" applied to me for quite some time now. The more I work with tools like Spark, the more dissatisfied I become with the data engineering world. Spark is a hot mess of fiddling with configuration - I've lost more productivity to dealing with memory limit, executor count, and other configuration than I think is reasonable. Pandas is another one. It was good enough to make quick processing concise that it got significant uptake and became de facto. The API is a pain, though, and processing is slow. Now, couple Pandas and Spark in your day-to-day job and you get what I see from my data science colleagues: "I'll hack together some processing in Pandas until my machine can't handle any more data, at which point I'll throw it into Spark and provision a bunch of nodes to do that work for me." I don't mean that to sound pejorative, as they're generally just trying to be productive, but there's so little attention paid to real efficiency in the frameworks and infrastructure that we're blowing through compute, memory, and person-hours unnecessarily (IMHO).

rectang 4y ago
> Pandas is another one. But at least if I write a transform in pandas it's straightforward to unit test it: create a DataFrame with some dummy data, send it through the function which wraps the transform, test that what comes out is what's expected. Validating a transform done in SQL is not nearly as straightforward. For starters it needs to be an integration test not a unit test (because you need a database). And that's assuming there's even a way to hook unit tests into your framework. I'm not a huge fan of Pandas — it's way too prone to silent failure. I've written wrappers around e.g. read_csv which are there to defeat all the magic. But at least I can do that with Python code instead of being stuck with the big-ball-of-SQL (e.g. some complicated view SELECT statement that does a bunch of JOINs, CASE tests and casts).
FridgeSeal 4y ago
Spark is the bane of my existence. Sure, you can spin out multiple executors and have it distribute work amongst them… Provided you figure out the 3 million configs necessary to do this in a reasonable way, so our dependency, networking and storage issues, only for the runtime to decide that actually, it’d quite like to run your entire workload on only 1 of your machines, or something equally frustrating. Given how frustrating it is to operate on, I’m shocked it has gained as much popularity as it has. I sort of wonder if that’s a bit by design from vested commercial interests- make it simultaneously exceedingly popular and so convoluted to run that you are basically obliged to pay for the likes of databricks just to make your life not awful. Of course, once you’ve done this, you’ve now bought into a whole suite of other issues, but that’s a different discussion…

bobbruno 4y ago

Let me see if I can at least justify some of these things: - SQL became the de facto standard for data manipulation a few decades ago, and I’m still to see a worthy contender. While the language itself shows its age, the concept of it being (almost) totally declarative of “what” data to get and “what” to do with it, instead of “how”, is unbeatable. A better syntax for achieving this is certainly possible, but you won’t beat decades of SQL code being generated easily. About “get the answer right once”, for analysis it’s often all that’s needed, and good SW maintence doesn’t apply that hard (it just won’t be needed again). For actual repeatable data engineering, I’d suggest either using generated SQL or writing in some abstraction (like PySpark). But don’t underestimate the productivity of SQL and the ability of modern parsers; - Unit testing: while I generally agree, after 28 years, I have to say that unit testing is not as useful in data engineering. Essentially, it doesn’t matter how much time and effort you put on designing test cases, you’ll never beat the chaotic creativity of real users making real systems ingest all kinds of wrong data in patterns you’d never imagine. Now multiply that by having to cross-reference data from systems that were designed independently, and think of all the possible data errors that might come out of that. You won’t, you won’t cover even 20% of the ways things can go wrong. So, instead of “testing for all ways things go wrong”, a much better pattern is “write code that can capture unexpected errors and make reasonable decisions on whether to continue or abort - with some nice messaging and data debugging, please”; - Poor support for version control: I 100% agree with you. DataOps is still not nearly as popular as it should be; - Frameworks over libraries: let me tell you one thing - data is coupled. Uncoupled data has less value than coupled data, because it can’t answer more complex questions. Having different libraries that represent, manipulate and expose data in different ways will just make things much harder for the DE trying to generate the data. And the focus is handling the data, not the code. So, while you may have a bit of a point, I’d suggest that you’re looking at this from the wrong perspective; Low-code: it’s one of those things that come and go. When I started, I’d use Oracle’s PL/SQL and C. Then came the first generation of visual ETL tools (Informatica, Datastage, etc) and they looked great with the graphic flows. Until you actually had to use them and all the clicking just to get to a point where you could actually add some logic showed. Then came the Hadoop era, and data engineering split into the DW people (who prefer visual, low-code to this day) and the Hadoop people, who went back to coding. This split stays to this day, with each side not recognising what the other does as essentially the same thing. Personally, I prefer code, but I don’t care what others prefer. The visual x coding discussion is, in my opinion, the wrong problem. The real issue is being able to create logic that’s clear enough that something as dumb as a computer will unambiguously understand. STEM learning environments for kids show that this is totally possible in a visual way - the thing is, most companies choose the visual tool in hopes that a person without the right mindset will generate good logic with it. That’s what fails. The rest is a matter of preferred UX - relevant, but not core or one-sided.