10 ms·

The evolution of the data engineer role

167 points by Arimbr 4y ago

swyx 4y ago
> Titles and responsibilities will also morph, potentially deeming the “data engineer” term obsolete in favor of more specialized and specific titles. "analytics engineer" is mentioned but also just had its first conference at dbt's conference. all the talks are already up https://coalesce.getdbt.com/agenda/keynote-the-end-of-the-road-for-the-modern-data-stack-you-know https://coalesce.getdbt.com/agenda/keynote-the-end-of-the-ro...
- pjot 4y ago
  Just to clarify, last week was dbt’s first _in person_ conference. Third overall.
  - acjohnson55 4y ago
    And in my opinion, it was an incredible time, and had great content.
buscoquadnary 4y ago
Business Analyst, Big Data Specialist, Data Mining Engineer, Data Scientist, Data Engineer. Why is this field so prone to hype and repeating the same things with a new coat of paint. I mean what ever happened to OLAP, data cubes, Big Data, and whatever other super big next thing that has happened in the past 2 decades? Methinks the problem with Business Intelligence solving problems is the firdt part of the term and not the second.
- MonkeyMalarky 4y ago
  I dunno, I have to first put my data somewhere though. But where.. In a warehouse? Silo? Data lake? Lake house? (I really despise that last one, who could coin that phase with a straight face..)
  - PubliusMI 4y ago
    IMHO, "data lake" = collection of all data sources: HDFS, S3, SQL DB, etc "lake house" = tool to query and combine results from all such sources: Dremio
    - buscoquadnary 4y ago
      But what lake house is complete without a boat. That's why my company is looking for investors who are interested in being at the forefront of the data revolution, using our data rowboat that will allow you to proactively leverage your data synergies to break down organizational data silos and use analytics to address your core competencies in order to leverage a strategic advantage to become the platform of choice in a holistic environment. Tell me if this sounds familiar, your company has tons of data but it is spread out all over the place and you can't seem to get good info, you end up hounding engineers to get your reports and provide you information so you can look like you are making data driven decisions. Maybe you've implemented a data lake but now have no idea how to use it. We've got you covered with our patent pending data rowboat solution. This will allow you to impressive everyone else in the mid level staff meetings by allowing you to say you are doing something around the "data revolution" in your org. The best part is that every implementation will come with a team of our in house consultants that will allow the project to drag on forever so that you always have something to report on in staff meetings and make you look good to your higher ups. Now you may be an engineer looking to revolutionize your career and get involved in the next step of the glorious October data revolution. Well we've got you covered for a very reasonable price you can enroll in our "data rowboat boot camp", where you will spend hours locked in a room where someone who barely speaks English will read documentation to you. But act quick otherwise you'll end up as one of the data kulaks as the new data rowboat revolution proceeds into a glorious future with our 5 year plan.
      - jdgoesmarching 4y ago
        Brb, running to trademark every nautical data metaphor I can get my hands on. What happens when your data rowboat runs ashore? Introducing Data Tugboat™, your single pane of glass solution for shoring up undocumented ETLs and reeling your data lineage safely into harbor.
- Avicebron 4y ago
  I think the real interesting point is slapping the title of engineer/scientiest on to anything and everything regardless of the accreditation actual handed out. soon coming up.."cafeteria engineer", "janitorial engineer"...
  - Test0129 4y ago
    The difference of course being that other types of engineers have to take a PE. The idea of requiring a PE to have that title is protectionism no different than limiting the number of graduating doctors to keep salaries high. No one will ask a software engineer to build a bridge - relax. Your protection racket is safe. Software engineer is a title conferred on someone who builds systems. It is fitting. And, if we're being honest, the average "job threat level" of a so-called "real" engineer is about the same as a software engineer these days anyway. With the exception of some niche jobs every engineer I know is just a CADIA/SW/etc jocky and the real work is gatekept by greybeards. No one will call someone a cafeteria engineer or janitorial engineer. The premise is ridiculous. There is a title called "operations engineer" that uses math to optimize processes. Does this one bother you too?
    - Avicebron 4y ago
      i assume youre a micro service plumber :)
      - Test0129 4y ago
        I like that title better to be honest.
  - the-printer 4y ago
    Woah, woah, woah. Cool it, buddy. That’s already begun.
- hatware 4y ago
  > Why is this field so prone to hype and repeating the same things with a new coat of paint. Money and Marketing. It's no different from how Hadoop was a big deal around 2010, or how Functional Programming became the new thing from 2015 onwards. Personally I think this is a failure of regulatory agencies.
- rustyboy 4y ago
  Because data "is the new oil" and those titles are considered cost jobs not revenue jobs. source: https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/ https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr...
jimcavel888 4y ago
MrPowers 4y ago
Great article. > data engineers have been increasingly adopting software engineering best practices I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days. I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot. > The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer. I think the Lakehouse architecture is the real future of data engineering, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision. It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).
- victor106 4y ago
  > It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. What are some good resources that can help educate folks on these differences?
  - MrPowers 4y ago
    I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg https://youtu.be/9LYYOdIwQXg Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.
  - claytonjy 4y ago
    Short version: - data warehouse: schema on write. you have to know the end form before you load it. breaks every time upstream changes (a lot, in this world) - data lake: schema on read. load everything into S3 and deal with it later. Mongo for data platforms - data lakehouse: something in between. store everything loosely like a lake, but have in-lakehouse processes present user-friendly transforms or views like a warehouse. Made possible by cheap storage (parquet on S3), reduces schema breakage by keeping both sides of the T in the same place
iamjk 4y ago
I won't be surprised if DE ends up just falling under the "software engineering" umbrella as the jobs grow closer together. With hybrid OLAP/OLTP databases becoming more popular, the skillset delta is definitely smaller than it used to be. Data Engineers are higher leverage assets to an organization than they ever have been before.
- buscoquadnary 4y ago
  Well my understanding is that a Data Engineer is basically just a DevOps engineer but instead of building infra to run applications they build infra to process, sanitize and normalize data.
  - hbarka 4y ago
    Or to denormalize data, the distinction of which the data engineer would be most familiar with why and how.
  - Avalaxy 4y ago
    Imho that is absolutely not doing the role justice. For some people that may hold true, but I would expect a data engineer to know everything about distributed systems, database indexes, how different databases work and why you pick them, partitioning, replication, transactions/locking. These are topics a software engineer is typically familiar with. A DevOps engineer wouldn't be.
  - tbarrera 4y ago
    Author here - Of course, data engineering involves building infra and being knowledgeable about DevOps practices, but that’s not the only area data engineers should be familiar with. There are many, many more! In my personal experience, sometimes we end up not using DevOps best practices because we spread too thin. That’s why I believe in specialization within data engineering and the surge of “data reliability engineer” or similar
- snapetom 4y ago
  I think it's mostly already there, but your big, enterprise houses were late getting the memo. About 12 years ago, I switched to a DE role/title and held it ever since. I worked in a variety of startups doing DE - moving data from over here to over there, with a variety of tools from orchestration frameworks to homegrown code in a variety of languages. About six years ago, I walked into a local hospital to interview for a DE role and it was very clear that their definition of DE was different than mine. The whole dept worked in nothing but SQL. I thought I was good with SQL, but they absolutely crucified me on SQL and data architecture theories. I ended up getting kicked over to a software engineering role, doing DE in another capacity, which made more sense for me. Only now I'm hearing that they're migrating to other tools like dbt and requiring their DEs to learn programming languages.
rectang 4y ago
I'm a software dev who's been bumping up against the data engineering field lately, and I've been dismayed as to how many popular tools shunt you towards unmaintainable, unevolvable system design. - A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. - Poor support for unit testing. - Poor support for version control. - Frameworks over libraries (because the vendors want to lock you in). > data engineers have been increasingly adopting software engineering best practices We can only hope. I think it's more likely that in the near term data engineers will get better and better at prototyping within low-code frameworks, and that transitioning from the prototype to an evolvable system will get harder.
- dang 4y ago
  Please stick to plain text - formatting gimmicks are distracting, not the convention here, and tend to start an arms race of copycats. I've put hyphens there now. Your comment is otherwise fine of course.
  - nmarinov 4y ago
    Out of curiosity, could you give or point to an example of "formatting gimmicks"? I tried searching the FAQ but only found the formatdoc guide: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
    - rectang 4y ago
      OT... I used one of the Unicode bullets instead of a hyphen (◉). I understand what dang is getting at here — it's in the same spirit as stripping emojis (because otherwise we'd be drowning in them). I'm a past graphic designer so I'm accustomed to working with the medium to achieve maximum communicative impact, but I don't mind operating within constraints. The convention of using hyphens to start a bulleted list isn't officially documented AFAIK. Having to separate bulleted list items with double newlines is a little weird, but it's fine.
usgroup 4y ago
I live in the world of data lakes and elaborate pipelines. Now and again I get to use a traditional star schema data warehouse and … it is an absolute pleasure to use in contrast to modern data access patterns.
z3c0 4y ago
For those of you who are genuinely curious why this field has so many similarly-named roles, here's a sincere, non-snarky, non-ironic explanation: A Data Analyst is distinct from a Systems Analyst or a Business Analyst. They may perform both systems and business analysis tasks, but their distinction comes from their understanding of statistics and how they apply that to other forms of analysis. A ML specialist is not a Data Scientist. Did you successfully build and deploy an ML model to production? Great! That's still uncommon, despite the hype. However, that would land you in the former position. You can claim the latter once you've walked that model through the scientific method, complete with hypothesis verification and democratization of your methodology. A BI Engineer and a Data Engineer are going to overlap a lot, but the former is going to lean more towards report development, where the latter will spend more time with ELTs/ETLs. As a data engineer, most of the report development that I do is to report on the state of data pipelines. BI BI, I like to call it. A Big Data Engineer or Specialist is a subset of data engineers and architects angled towards the problems of big data. This distinction actually matters now, because I'm encountering data professionals these day who have never worked outside the cloud or with small enterprise datasets (unthinkable only half-a-decade ago.) It doesn't help that lack of understanding often leads to misnomer positions, but anybody who has spent time in this field gets used to the subtle differences quickly.
- itsoktocry 4y ago
  What about Analytics Engineer, the hypiest-of-the-hyped right now?
  - claytonjy 4y ago
    BI engineer that knows dbt
  - acjohnson55 4y ago
    A data pipeline person who does their work in SQL, rather than batch/stream processing tools. It's not hype, it's just a role built around a different data architecture. It's less powerful than the old big data toolkit, but it's also probably perfectly suitable for many businesses.
unholyguy001 4y ago
It’s fascinating that the conclusion / forecast is that tools will abstract engineering problems and DE will move closer to the business . While over the last 20 years the exact opposite has happened and the toolset has actually become harder (not easier to use) but orders of magnitude more powerful and DE has moved closer to engineering, to the point where a good data engineer basically is a specialized software engineer. The absolute pinnacle of “easy to use” was probably the Informatica / Oracle stack of the late 90’s and early 00’s. It just wasn’t powerful or scalable enough to meet the needs of the Big Data shift Of course I guess this makes sense given the author works for a company with a vested interest in reversing that trend.
- hnews_account_1 4y ago
  I think those days were easy to use within the zeitgeist. Even advanced versions of those tools would struggle against the data needs today which have become incredibly bespoke. My skill set extends all the way from my actual industry (finance) to the boundary of software development. I also have data, big data and cluster usage skills (slurm etc). I don’t use everything every day and obviously I cannot be a specialist in most of this stuff (I concentrate on finance more than anything else) considering the incredible range, but this is just the past 2 years for me. I cannot imagine a less specialized future looking around today where some nice tool does 80% of my work. Not because the work I do is difficult to automate. But because the work I do won’t match the work other industries may do (beyond existing generalizations of pandas, regression toolkits and other low level stuff). There’s no point building a full automation suite just for my single work profile which itself will differ from other areas of finance.
- rustyboy 4y ago
  DE's at my company actually spend all their time making easy to use interfaces ontop of these tools (Github workflows, IaC/CI-CD solutions, Dev/SecOps features, etc.) so that DW/BI/DS Developers, or even analysts in some cases, can create new pipelines quite easily.
  - snidane 4y ago
    That sounds like infra ops, not data engineering. It's a common theme, though, for DEs to slip into maintaining infra and analysts taking on the role of engineering data pipelines.
iblaine 4y ago
A full history of DE should include some original low code tools (Cognos, Informatica, SSIS). To some extent, the failure of these tools to adopt to the evolution of the DE role has lead to our modern data stack.
- Eumenes 4y ago
  Agreed. This is the first thing I thought about - the evolution from reporting systems to ETL code to Hadoop to Spark, etc.
- gregw2 4y ago
  Informatica and SSIS, in practice, in my experience, are not really low code tools. I'd call them... GUI-UI based coding tools?
sherifnada 4y ago
In some sense, Data engineering today is where software engineering was a decade ago: - Infrastructure as code is not the norm. Most tools are UI-focused. It's the equivalent of setting up your infra via the AWS UI. - Prod/Staging/Dev environments are not the norm - Version Control is not a first class concept - DRY and component re-use is exceedingly difficult (how many times did you walk into a meeting where 3 people had 3 different definitions of the same metric?) - API Interfaces are rarely explicitly defined, and fickle when they are (the hot name for this nowadays is "data contracts") - unit/integration/acceptance testing is not as nearly as ubiquitous as it is in software On the bright side, I think this means DE doesn't need to re-invent the wheel on a lot of these issues. We can borrow a lot from software engineering.
- mywittyname 4y ago
  My DE team has all of these, and I've never worked on a team without them. I speak as someone whose official title has been Data Engineer since 2015 and I've consulted for lots of F500 companies. Unit testing is the only thing we tend to skip, mainly because it's more reliable to allow for fluidity in the data that's being ingested. Which is really easy now that so many databases can support automatic schema detection. External APIs can change without notice, so it's better to just design for that, then use the time you would spend on unit tests to build alerts around automated data validation.
- beckingz 4y ago
  You're talking about analytics not data engineering. But yes, Data Analysis still needs more of this, though the smarter folks are getting on the Analytics Engineering / DataOps trains.
- artwr 4y ago
  > Infrastructure as code is not the norm. Most tools are UI-focused. It's the equivalent of setting up your infra via the AWS UI. > Version Control is not a first class concept Of course, I may have worked in all of the wrong places but all but one of the places I've worked for the past ten years had source control for data pipelines or the ability to setup via config/source control code as opposed to UIs. > - Prod/Staging/Dev environments are not the norm Fairly true, though in some cases, staging/dev has a bit more footprint/investment required than for backend or frontend development. > DRY and component re-use is exceedingly difficult (how many times did you walk into a meeting where 3 people had 3 different definitions of the same metric?) That's a hard one and I agree that's where a lot of opportunity is. There are several efforts to get at a more semantic layer / metric catalog where the people who care about the metrics can agree on the definition, but that's more of an organizational issue, not a data engineering issue. Proper data modeling to ensure you can more easily reuse the metric as needed is also core here. > - API Interfaces are rarely explicitly defined, and fickle when they are (the hot name for this nowadays is "data contracts") That's another hard issue. The way I see it, it's still going to be a mix between nicely defined contracts and much looser logging that the DE still has to try to shape into something useful, sometimes even successfully. > - unit/integration/acceptance testing is not as nearly as ubiquitous as it is in software I take a slight issue with ubiquitous. The amount of software (from paid vendors no less) I have interacted with which does not have proper acceptance/integration testing is just plain sad.
webshit2 4y ago
As someone who knows nothing about this stuff, I'm looking at the "Data Mart" wiki page: https://en.wikipedia.org/wiki/Data_mart https://en.wikipedia.org/wiki/Data_mart. Ok, so the entire diagram here is labelled "Data Warehouse", and within that there's a "Data Warehouse" block which seems to be solely comprised of a "Data Vault". Do you need a special data key to get into the data vault in the data warehouse? Okay, naturally the data marts are divided into normal marts and strategic marts - seems smart. But all the arrows between everything are labelled "ETL". Seems redundant. What does it mean anyway? Ok apparently it's just... moving data. Now I look at https://en.wikipedia.org/wiki/Online_analytical_processing https://en.wikipedia.org/wiki/Online_analytical_processing. What's that? First sentence: "is an approach to answer multi-dimensional analytical (MDA)". I click through to https://en.wikipedia.org/wiki/Multidimensional_analysis https://en.wikipedia.org/wiki/Multidimensional_analysis ... MDA "is a data analysis process that groups data into two categories: data dimensions and measurements". What the fuck? Who wrote this? Alright, back on the OLAP wiki page... "The measures are placed at the intersections of the hypercube, which is spanned by the dimensions as a vector space." Ah yes, the intersections... why not keep math out of it if you have no idea how to talk about it? Also, there's no actual mention of why this is considered "online" in the first place. I feel like I'm in a nightmare where the pandas documentation was rewritten in MBA-speak.
- rjbwork 4y ago
  It's a difficult sphere of knowledge to penetrate. All of that is perfectly coherent to me, FWIW. From first principals, I can highly recommend Ralph Kimball's primer, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling.[1] [1]https://www.amazon.com/gp/product/B00DRZX6XS https://www.amazon.com/gp/product/B00DRZX6XS
- Shugarl 4y ago
  Thank you, as a data engineer, that's exactly how I feel about the various concept surrounding data engineering. Or at the very least, the way they're being explained.
- acjohnson55 4y ago
  As someone who just arrived in data world, I feel your pain. You didn't even mention the data lake and the data warehouse are set for merger into the data lakehouse. Not to mention where data mesh and data fabric fit into all of this. It's hard for me to say why this all seems so much more confusing than the software dev world. My guess is because data is a thing that a business accumulates and processes, often as a side channel to its actual work. There's an inherent meta-ness to it, and both the business and tech people have had a hand in shaping the approaches. So it's kind of a mess, and for whatever reason, even more susceptible to buzzwordery than the rest of tech.
prions 4y ago
IMO Data engineering is already a specialized form of software engineering. However what people interpret as DE's being slow to adopt best practices from traditional software engineering is more about the unique difficulties of working with data (especially at scale) and less about the awareness or desire to use best practices. Speaking from my DE experience at Spotify and previously in startup land, the biggest challenge is the slow and distant feedback loop. The vast majority of data pipelines don't run on your machine and don't behave like they do on a local machine. They run as massively distributed processes and their state is opaque to the developer. Validating the correctness of a large scale data pipeline can be incredibly difficult as the successful operation of a pipeline doesn't conclusively determine whether the data is actually correct for the end user. People working seriously in this space understand that traditional practices here like unit testing only go so far. And integration testing really needs to work at scale with easily recyclable infrastructure (and data) to not be a massive drag on developer productivity. Even getting the correct kind of data to be fed into a test can be very difficult if the ops/infra of the org isn't designed for it. The best data tooling isn't going to look exactly like traditional swe tooling. Tools that vastly reduce the feedback loop of developing (and debugging) distributed pipelines running in the cloud and also provide means of validating the output on meaningful data is where tooling should be going. Trying to shoehorn traditional SWE best practices will really only take off once that kind of developer experience is realized.
- azurezyq 4y ago
  This is actually a great observation. Data pipelines are often written in various languages, running on heterogenous systems, with different time alignment schemes. I always found it tricky to "fully trust" a piece of result. Hmm, any best practice from your side?
  - oa335 4y ago
    Not OP, but a Data Engineer with 4 years experience in the space - I think the key is to first build the feedback loop - i.e. any thing that helps you answer how do you know the data pipeline is flowing and that the data is correct - then getting sign-off from both the producers and consumers of the data. Actually getting the data flowing is usually pretty easy after both parties agree about what that actually means.
mywittyname 4y ago
> The declarative concept is highly tied to the trend of moving away from data pipelines and embracing data products – Of course an Airbyte article would say this, because they are selling these tools, but my experience has been the opposite. People buy these tools because they claim to make it easier for non-software people to build pipelines. But the problem is that these tools seem to end up being far more complicated and less reliable than pipelines built in code. There's a reason that this domain is saturated with so. many. tools. None of them do a great job. And when a company invariably hits the limits of one, they start shopping for a replacement, which will have it's own set of limitations. Lather-rinse-repeat. I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. You can get surprisingly far in the data world with Airflow + a large scale database, but all of the major cloud providers have great tool offerings in this space. Plus, for platforms that these tools don't interface with, you're going to have to write code anyway.
- Arimbr 4y ago
  Oh, declarative doesn't necessarily mean no-code. Airbyte data integration connectors are built with an SDK in Python, Java, and a low-code SDK that was just released... You can then build custom connectors on top of these and many users actually need to modify an existing connector, but would rather start from a template than from scratch. Airbyte also provides a CLI and YAML configuration language that you can use to declare sources, destinations and connections without the UI: https://github.com/airbytehq/airbyte/blob/master/octavia-cli/README.md https://github.com/airbytehq/airbyte/blob/master/octavia-cli... I agree with you that code is here to stay and power users need to see the code and modify it. That's why Airbyte code is open-source.
- muspimerol 4y ago
  > I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. I'm sure you earn a nice living doing this, but surely this is not a convincing argument against using off-the-shelf data products. It will always come down to the cost (including ongoing maintenance) for the business. Bespoke in-house software is always the most flexible route, but rarely the cheaper one.
Tycho 4y ago
I would describe myself as a dataframe engineer.
antipaul 4y ago
Good readable historical overview, IMO. The referenced "The Rise of the Data Engineer" article proved quite prescient. The present overview seems less good on future predictions (it's hard!). Also, some mention of "data science" could've been a propos – though it's difficult to find consensus on what DS actually entails. Overall, I think a good framework to navigate this space is to think of 2 overarching disciplines: 1) Engineering: scaling data & software. 2) Science: getting actionable insights from data combined with subject matter/domain expertise Usually, #1 is a platform for #2 – but getting #1 good seems just as, if not more, important, and is perhaps harder. EDIT: formatting
Dowwie 4y ago
Have you seen what is possible with Elixir and its Broadway library? You can set up a fault tolerant, concurrent worker pool utilizing all of the necessary feedback mechanisms involved with message processing.

167 points by Arimbr 4y ago

swyx 4y ago

> Titles and responsibilities will also morph, potentially deeming the “data engineer” term obsolete in favor of more specialized and specific titles. "analytics engineer" is mentioned but also just had its first conference at dbt's conference. all the talks are already up https://coalesce.getdbt.com/agenda/keynote-the-end-of-the-road-for-the-modern-data-stack-you-know https://coalesce.getdbt.com/agenda/keynote-the-end-of-the-ro...

pjot 4y ago
Just to clarify, last week was dbt’s first _in person_ conference. Third overall.
- acjohnson55 4y ago
  And in my opinion, it was an incredible time, and had great content.

buscoquadnary 4y ago

Business Analyst, Big Data Specialist, Data Mining Engineer, Data Scientist, Data Engineer. Why is this field so prone to hype and repeating the same things with a new coat of paint. I mean what ever happened to OLAP, data cubes, Big Data, and whatever other super big next thing that has happened in the past 2 decades? Methinks the problem with Business Intelligence solving problems is the firdt part of the term and not the second.

MonkeyMalarky 4y ago
I dunno, I have to first put my data somewhere though. But where.. In a warehouse? Silo? Data lake? Lake house? (I really despise that last one, who could coin that phase with a straight face..)
- PubliusMI 4y ago
  IMHO, "data lake" = collection of all data sources: HDFS, S3, SQL DB, etc "lake house" = tool to query and combine results from all such sources: Dremio
  - buscoquadnary 4y ago
    But what lake house is complete without a boat. That's why my company is looking for investors who are interested in being at the forefront of the data revolution, using our data rowboat that will allow you to proactively leverage your data synergies to break down organizational data silos and use analytics to address your core competencies in order to leverage a strategic advantage to become the platform of choice in a holistic environment. Tell me if this sounds familiar, your company has tons of data but it is spread out all over the place and you can't seem to get good info, you end up hounding engineers to get your reports and provide you information so you can look like you are making data driven decisions. Maybe you've implemented a data lake but now have no idea how to use it. We've got you covered with our patent pending data rowboat solution. This will allow you to impressive everyone else in the mid level staff meetings by allowing you to say you are doing something around the "data revolution" in your org. The best part is that every implementation will come with a team of our in house consultants that will allow the project to drag on forever so that you always have something to report on in staff meetings and make you look good to your higher ups. Now you may be an engineer looking to revolutionize your career and get involved in the next step of the glorious October data revolution. Well we've got you covered for a very reasonable price you can enroll in our "data rowboat boot camp", where you will spend hours locked in a room where someone who barely speaks English will read documentation to you. But act quick otherwise you'll end up as one of the data kulaks as the new data rowboat revolution proceeds into a glorious future with our 5 year plan.
    - jdgoesmarching 4y ago
      Brb, running to trademark every nautical data metaphor I can get my hands on. What happens when your data rowboat runs ashore? Introducing Data Tugboat™, your single pane of glass solution for shoring up undocumented ETLs and reeling your data lineage safely into harbor.
Avicebron 4y ago
I think the real interesting point is slapping the title of engineer/scientiest on to anything and everything regardless of the accreditation actual handed out. soon coming up.."cafeteria engineer", "janitorial engineer"...
- Test0129 4y ago
  The difference of course being that other types of engineers have to take a PE. The idea of requiring a PE to have that title is protectionism no different than limiting the number of graduating doctors to keep salaries high. No one will ask a software engineer to build a bridge - relax. Your protection racket is safe. Software engineer is a title conferred on someone who builds systems. It is fitting. And, if we're being honest, the average "job threat level" of a so-called "real" engineer is about the same as a software engineer these days anyway. With the exception of some niche jobs every engineer I know is just a CADIA/SW/etc jocky and the real work is gatekept by greybeards. No one will call someone a cafeteria engineer or janitorial engineer. The premise is ridiculous. There is a title called "operations engineer" that uses math to optimize processes. Does this one bother you too?
  - Avicebron 4y ago
    i assume youre a micro service plumber :)
    - Test0129 4y ago
      I like that title better to be honest.
- the-printer 4y ago
  Woah, woah, woah. Cool it, buddy. That’s already begun.
hatware 4y ago
> Why is this field so prone to hype and repeating the same things with a new coat of paint. Money and Marketing. It's no different from how Hadoop was a big deal around 2010, or how Functional Programming became the new thing from 2015 onwards. Personally I think this is a failure of regulatory agencies.
rustyboy 4y ago
Because data "is the new oil" and those titles are considered cost jobs not revenue jobs. source: https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/ https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr...

jimcavel888 4y ago

MrPowers 4y ago

Great article. > data engineers have been increasingly adopting software engineering best practices I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days. I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot. > The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer. I think the Lakehouse architecture is the real future of data engineering, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision. It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).

victor106 4y ago
> It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. What are some good resources that can help educate folks on these differences?
- MrPowers 4y ago
  I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg https://youtu.be/9LYYOdIwQXg Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.
- claytonjy 4y ago
  Short version: - data warehouse: schema on write. you have to know the end form before you load it. breaks every time upstream changes (a lot, in this world) - data lake: schema on read. load everything into S3 and deal with it later. Mongo for data platforms - data lakehouse: something in between. store everything loosely like a lake, but have in-lakehouse processes present user-friendly transforms or views like a warehouse. Made possible by cheap storage (parquet on S3), reduces schema breakage by keeping both sides of the T in the same place

iamjk 4y ago

I won't be surprised if DE ends up just falling under the "software engineering" umbrella as the jobs grow closer together. With hybrid OLAP/OLTP databases becoming more popular, the skillset delta is definitely smaller than it used to be. Data Engineers are higher leverage assets to an organization than they ever have been before.

buscoquadnary 4y ago
Well my understanding is that a Data Engineer is basically just a DevOps engineer but instead of building infra to run applications they build infra to process, sanitize and normalize data.
- hbarka 4y ago
  Or to denormalize data, the distinction of which the data engineer would be most familiar with why and how.
- Avalaxy 4y ago
  Imho that is absolutely not doing the role justice. For some people that may hold true, but I would expect a data engineer to know everything about distributed systems, database indexes, how different databases work and why you pick them, partitioning, replication, transactions/locking. These are topics a software engineer is typically familiar with. A DevOps engineer wouldn't be.
- tbarrera 4y ago
  Author here - Of course, data engineering involves building infra and being knowledgeable about DevOps practices, but that’s not the only area data engineers should be familiar with. There are many, many more! In my personal experience, sometimes we end up not using DevOps best practices because we spread too thin. That’s why I believe in specialization within data engineering and the surge of “data reliability engineer” or similar
snapetom 4y ago
I think it's mostly already there, but your big, enterprise houses were late getting the memo. About 12 years ago, I switched to a DE role/title and held it ever since. I worked in a variety of startups doing DE - moving data from over here to over there, with a variety of tools from orchestration frameworks to homegrown code in a variety of languages. About six years ago, I walked into a local hospital to interview for a DE role and it was very clear that their definition of DE was different than mine. The whole dept worked in nothing but SQL. I thought I was good with SQL, but they absolutely crucified me on SQL and data architecture theories. I ended up getting kicked over to a software engineering role, doing DE in another capacity, which made more sense for me. Only now I'm hearing that they're migrating to other tools like dbt and requiring their DEs to learn programming languages.

rectang 4y ago

I'm a software dev who's been bumping up against the data engineering field lately, and I've been dismayed as to how many popular tools shunt you towards unmaintainable, unevolvable system design. - A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. - Poor support for unit testing. - Poor support for version control. - Frameworks over libraries (because the vendors want to lock you in). > data engineers have been increasingly adopting software engineering best practices We can only hope. I think it's more likely that in the near term data engineers will get better and better at prototyping within low-code frameworks, and that transitioning from the prototype to an evolvable system will get harder.

dang 4y ago
Please stick to plain text - formatting gimmicks are distracting, not the convention here, and tend to start an arms race of copycats. I've put hyphens there now. Your comment is otherwise fine of course.
- nmarinov 4y ago
  Out of curiosity, could you give or point to an example of "formatting gimmicks"? I tried searching the FAQ but only found the formatdoc guide: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
  - rectang 4y ago
    OT... I used one of the Unicode bullets instead of a hyphen (◉). I understand what dang is getting at here — it's in the same spirit as stripping emojis (because otherwise we'd be drowning in them). I'm a past graphic designer so I'm accustomed to working with the medium to achieve maximum communicative impact, but I don't mind operating within constraints. The convention of using hyphens to start a bulleted list isn't officially documented AFAIK. Having to separate bulleted list items with double newlines is a little weird, but it's fine.

usgroup 4y ago

I live in the world of data lakes and elaborate pipelines. Now and again I get to use a traditional star schema data warehouse and … it is an absolute pleasure to use in contrast to modern data access patterns.

z3c0 4y ago

For those of you who are genuinely curious why this field has so many similarly-named roles, here's a sincere, non-snarky, non-ironic explanation: A Data Analyst is distinct from a Systems Analyst or a Business Analyst. They may perform both systems and business analysis tasks, but their distinction comes from their understanding of statistics and how they apply that to other forms of analysis. A ML specialist is not a Data Scientist. Did you successfully build and deploy an ML model to production? Great! That's still uncommon, despite the hype. However, that would land you in the former position. You can claim the latter once you've walked that model through the scientific method, complete with hypothesis verification and democratization of your methodology. A BI Engineer and a Data Engineer are going to overlap a lot, but the former is going to lean more towards report development, where the latter will spend more time with ELTs/ETLs. As a data engineer, most of the report development that I do is to report on the state of data pipelines. BI BI, I like to call it. A Big Data Engineer or Specialist is a subset of data engineers and architects angled towards the problems of big data. This distinction actually matters now, because I'm encountering data professionals these day who have never worked outside the cloud or with small enterprise datasets (unthinkable only half-a-decade ago.) It doesn't help that lack of understanding often leads to misnomer positions, but anybody who has spent time in this field gets used to the subtle differences quickly.

itsoktocry 4y ago
What about Analytics Engineer, the hypiest-of-the-hyped right now?
- claytonjy 4y ago
  BI engineer that knows dbt
- acjohnson55 4y ago
  A data pipeline person who does their work in SQL, rather than batch/stream processing tools. It's not hype, it's just a role built around a different data architecture. It's less powerful than the old big data toolkit, but it's also probably perfectly suitable for many businesses.

unholyguy001 4y ago

It’s fascinating that the conclusion / forecast is that tools will abstract engineering problems and DE will move closer to the business . While over the last 20 years the exact opposite has happened and the toolset has actually become harder (not easier to use) but orders of magnitude more powerful and DE has moved closer to engineering, to the point where a good data engineer basically is a specialized software engineer. The absolute pinnacle of “easy to use” was probably the Informatica / Oracle stack of the late 90’s and early 00’s. It just wasn’t powerful or scalable enough to meet the needs of the Big Data shift Of course I guess this makes sense given the author works for a company with a vested interest in reversing that trend.

hnews_account_1 4y ago
I think those days were easy to use within the zeitgeist. Even advanced versions of those tools would struggle against the data needs today which have become incredibly bespoke. My skill set extends all the way from my actual industry (finance) to the boundary of software development. I also have data, big data and cluster usage skills (slurm etc). I don’t use everything every day and obviously I cannot be a specialist in most of this stuff (I concentrate on finance more than anything else) considering the incredible range, but this is just the past 2 years for me. I cannot imagine a less specialized future looking around today where some nice tool does 80% of my work. Not because the work I do is difficult to automate. But because the work I do won’t match the work other industries may do (beyond existing generalizations of pandas, regression toolkits and other low level stuff). There’s no point building a full automation suite just for my single work profile which itself will differ from other areas of finance.
rustyboy 4y ago
DE's at my company actually spend all their time making easy to use interfaces ontop of these tools (Github workflows, IaC/CI-CD solutions, Dev/SecOps features, etc.) so that DW/BI/DS Developers, or even analysts in some cases, can create new pipelines quite easily.
- snidane 4y ago
  That sounds like infra ops, not data engineering. It's a common theme, though, for DEs to slip into maintaining infra and analysts taking on the role of engineering data pipelines.

iblaine 4y ago

A full history of DE should include some original low code tools (Cognos, Informatica, SSIS). To some extent, the failure of these tools to adopt to the evolution of the DE role has lead to our modern data stack.

Eumenes 4y ago
Agreed. This is the first thing I thought about - the evolution from reporting systems to ETL code to Hadoop to Spark, etc.
gregw2 4y ago
Informatica and SSIS, in practice, in my experience, are not really low code tools. I'd call them... GUI-UI based coding tools?

sherifnada 4y ago

In some sense, Data engineering today is where software engineering was a decade ago: - Infrastructure as code is not the norm. Most tools are UI-focused. It's the equivalent of setting up your infra via the AWS UI. - Prod/Staging/Dev environments are not the norm - Version Control is not a first class concept - DRY and component re-use is exceedingly difficult (how many times did you walk into a meeting where 3 people had 3 different definitions of the same metric?) - API Interfaces are rarely explicitly defined, and fickle when they are (the hot name for this nowadays is "data contracts") - unit/integration/acceptance testing is not as nearly as ubiquitous as it is in software On the bright side, I think this means DE doesn't need to re-invent the wheel on a lot of these issues. We can borrow a lot from software engineering.

mywittyname 4y ago
My DE team has all of these, and I've never worked on a team without them. I speak as someone whose official title has been Data Engineer since 2015 and I've consulted for lots of F500 companies. Unit testing is the only thing we tend to skip, mainly because it's more reliable to allow for fluidity in the data that's being ingested. Which is really easy now that so many databases can support automatic schema detection. External APIs can change without notice, so it's better to just design for that, then use the time you would spend on unit tests to build alerts around automated data validation.
beckingz 4y ago
You're talking about analytics not data engineering. But yes, Data Analysis still needs more of this, though the smarter folks are getting on the Analytics Engineering / DataOps trains.
artwr 4y ago
> Infrastructure as code is not the norm. Most tools are UI-focused. It's the equivalent of setting up your infra via the AWS UI. > Version Control is not a first class concept Of course, I may have worked in all of the wrong places but all but one of the places I've worked for the past ten years had source control for data pipelines or the ability to setup via config/source control code as opposed to UIs. > - Prod/Staging/Dev environments are not the norm Fairly true, though in some cases, staging/dev has a bit more footprint/investment required than for backend or frontend development. > DRY and component re-use is exceedingly difficult (how many times did you walk into a meeting where 3 people had 3 different definitions of the same metric?) That's a hard one and I agree that's where a lot of opportunity is. There are several efforts to get at a more semantic layer / metric catalog where the people who care about the metrics can agree on the definition, but that's more of an organizational issue, not a data engineering issue. Proper data modeling to ensure you can more easily reuse the metric as needed is also core here. > - API Interfaces are rarely explicitly defined, and fickle when they are (the hot name for this nowadays is "data contracts") That's another hard issue. The way I see it, it's still going to be a mix between nicely defined contracts and much looser logging that the DE still has to try to shape into something useful, sometimes even successfully. > - unit/integration/acceptance testing is not as nearly as ubiquitous as it is in software I take a slight issue with ubiquitous. The amount of software (from paid vendors no less) I have interacted with which does not have proper acceptance/integration testing is just plain sad.

webshit2 4y ago

As someone who knows nothing about this stuff, I'm looking at the "Data Mart" wiki page: https://en.wikipedia.org/wiki/Data_mart https://en.wikipedia.org/wiki/Data_mart. Ok, so the entire diagram here is labelled "Data Warehouse", and within that there's a "Data Warehouse" block which seems to be solely comprised of a "Data Vault". Do you need a special data key to get into the data vault in the data warehouse? Okay, naturally the data marts are divided into normal marts and strategic marts - seems smart. But all the arrows between everything are labelled "ETL". Seems redundant. What does it mean anyway? Ok apparently it's just... moving data. Now I look at https://en.wikipedia.org/wiki/Online_analytical_processing https://en.wikipedia.org/wiki/Online_analytical_processing. What's that? First sentence: "is an approach to answer multi-dimensional analytical (MDA)". I click through to https://en.wikipedia.org/wiki/Multidimensional_analysis https://en.wikipedia.org/wiki/Multidimensional_analysis ... MDA "is a data analysis process that groups data into two categories: data dimensions and measurements". What the fuck? Who wrote this? Alright, back on the OLAP wiki page... "The measures are placed at the intersections of the hypercube, which is spanned by the dimensions as a vector space." Ah yes, the intersections... why not keep math out of it if you have no idea how to talk about it? Also, there's no actual mention of why this is considered "online" in the first place. I feel like I'm in a nightmare where the pandas documentation was rewritten in MBA-speak.

rjbwork 4y ago
It's a difficult sphere of knowledge to penetrate. All of that is perfectly coherent to me, FWIW. From first principals, I can highly recommend Ralph Kimball's primer, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling.[1] [1]https://www.amazon.com/gp/product/B00DRZX6XS https://www.amazon.com/gp/product/B00DRZX6XS
Shugarl 4y ago
Thank you, as a data engineer, that's exactly how I feel about the various concept surrounding data engineering. Or at the very least, the way they're being explained.
acjohnson55 4y ago
As someone who just arrived in data world, I feel your pain. You didn't even mention the data lake and the data warehouse are set for merger into the data lakehouse. Not to mention where data mesh and data fabric fit into all of this. It's hard for me to say why this all seems so much more confusing than the software dev world. My guess is because data is a thing that a business accumulates and processes, often as a side channel to its actual work. There's an inherent meta-ness to it, and both the business and tech people have had a hand in shaping the approaches. So it's kind of a mess, and for whatever reason, even more susceptible to buzzwordery than the rest of tech.

prions 4y ago

IMO Data engineering is already a specialized form of software engineering. However what people interpret as DE's being slow to adopt best practices from traditional software engineering is more about the unique difficulties of working with data (especially at scale) and less about the awareness or desire to use best practices. Speaking from my DE experience at Spotify and previously in startup land, the biggest challenge is the slow and distant feedback loop. The vast majority of data pipelines don't run on your machine and don't behave like they do on a local machine. They run as massively distributed processes and their state is opaque to the developer. Validating the correctness of a large scale data pipeline can be incredibly difficult as the successful operation of a pipeline doesn't conclusively determine whether the data is actually correct for the end user. People working seriously in this space understand that traditional practices here like unit testing only go so far. And integration testing really needs to work at scale with easily recyclable infrastructure (and data) to not be a massive drag on developer productivity. Even getting the correct kind of data to be fed into a test can be very difficult if the ops/infra of the org isn't designed for it. The best data tooling isn't going to look exactly like traditional swe tooling. Tools that vastly reduce the feedback loop of developing (and debugging) distributed pipelines running in the cloud and also provide means of validating the output on meaningful data is where tooling should be going. Trying to shoehorn traditional SWE best practices will really only take off once that kind of developer experience is realized.

azurezyq 4y ago
This is actually a great observation. Data pipelines are often written in various languages, running on heterogenous systems, with different time alignment schemes. I always found it tricky to "fully trust" a piece of result. Hmm, any best practice from your side?
- oa335 4y ago
  Not OP, but a Data Engineer with 4 years experience in the space - I think the key is to first build the feedback loop - i.e. any thing that helps you answer how do you know the data pipeline is flowing and that the data is correct - then getting sign-off from both the producers and consumers of the data. Actually getting the data flowing is usually pretty easy after both parties agree about what that actually means.

mywittyname 4y ago

> The declarative concept is highly tied to the trend of moving away from data pipelines and embracing data products – Of course an Airbyte article would say this, because they are selling these tools, but my experience has been the opposite. People buy these tools because they claim to make it easier for non-software people to build pipelines. But the problem is that these tools seem to end up being far more complicated and less reliable than pipelines built in code. There's a reason that this domain is saturated with so. many. tools. None of them do a great job. And when a company invariably hits the limits of one, they start shopping for a replacement, which will have it's own set of limitations. Lather-rinse-repeat. I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. You can get surprisingly far in the data world with Airflow + a large scale database, but all of the major cloud providers have great tool offerings in this space. Plus, for platforms that these tools don't interface with, you're going to have to write code anyway.

Arimbr 4y ago
Oh, declarative doesn't necessarily mean no-code. Airbyte data integration connectors are built with an SDK in Python, Java, and a low-code SDK that was just released... You can then build custom connectors on top of these and many users actually need to modify an existing connector, but would rather start from a template than from scratch. Airbyte also provides a CLI and YAML configuration language that you can use to declare sources, destinations and connections without the UI: https://github.com/airbytehq/airbyte/blob/master/octavia-cli/README.md https://github.com/airbytehq/airbyte/blob/master/octavia-cli... I agree with you that code is here to stay and power users need to see the code and modify it. That's why Airbyte code is open-source.
muspimerol 4y ago
> I built a solid career over the past 8 or so years of replacing these "no code" pipeline tools with code once companies hit the ceilings of these tools. I'm sure you earn a nice living doing this, but surely this is not a convincing argument against using off-the-shelf data products. It will always come down to the cost (including ongoing maintenance) for the business. Bespoke in-house software is always the most flexible route, but rarely the cheaper one.

Tycho 4y ago

I would describe myself as a dataframe engineer.

antipaul 4y ago

Good readable historical overview, IMO. The referenced "The Rise of the Data Engineer" article proved quite prescient. The present overview seems less good on future predictions (it's hard!). Also, some mention of "data science" could've been a propos – though it's difficult to find consensus on what DS actually entails. Overall, I think a good framework to navigate this space is to think of 2 overarching disciplines: 1) Engineering: scaling data & software. 2) Science: getting actionable insights from data combined with subject matter/domain expertise Usually, #1 is a platform for #2 – but getting #1 good seems just as, if not more, important, and is perhaps harder. EDIT: formatting

Dowwie 4y ago

Have you seen what is possible with Elixir and its Broadway library? You can set up a fault tolerant, concurrent worker pool utilizing all of the necessary feedback mechanisms involved with message processing.