6 ms·
Data Engineering Design Patterns
- hdespiritu 3y agoDAMA-DMBOK2 covers this very comprehensively https://www.dama.org/cpages/body-of-knowledge https://www.dama.org/cpages/body-of-knowledge
- datadrivenangel 3y agoData engineering is cool and new while data management is old school and enterprise. Specifically, data engineering in some tech companies is truly a revenue driver, so it makes data engineering in other organizations be viewed as a cost center so much, even if it is the same work at most organizations.
- esafak 3y agoHow do you define the two terms?
- datadrivenangel 3y agoData engineering versus data management? Data engineering is nominally more pipeline oriented and less concerned with the governance & people side of things, but good data engineering people end up driving a lot of data management work because that's what makes the data engineering less painful (eliminate root cause of data errors and annoying data requests) and data overall more useful and valuable.
- tremon 3y agoData Engineering is an engineering displine -- it can involve anything from data ingestion, transformation, storage, enrichment, aggregation up to presentation in operational reports. But it's still a manufacturing process with "data" as an input and "data" as its output. Data Management is an organization discipline -- it is about how the enterprise manages data as an asset and how data is embedded in the organization. This includes data governance issues like common data models, and a chain of command (which person/role is responsible for which piece of data), but also second-tier data processes such as quality control and data valuation.
- erhserhdfd 3y agoThis may be nitpicking, but technologies being described as "cool" versus "enterprise" or "new" versus "old" I find meaningless. I don't necessarily want to have the "coolest" or "newest" tech stack; I want to have the tech stack that solves reasonably and reliably solves my business problems. If that means leveraging "old" or "enterprise" technologies and practices, that could be totally fine.
- bfors 3y agoCould be interesting once there's more content, in its current state the content is mostly just definitions.
- articsputnik 3y agoAuthor of the book here — 100%. The idea is to release early and update on the go. You might wonder if it's worth delving into the book in its current, unfinished form. I'd say it depends. There's quite a bit of content around general DE knowledge. However, the anticipated design patterns are still in the works. Suppose that's what you're most excited about. In that case, you'll find the beginnings of exploring the patterns of `caching` and `ad-hoc querying` in the first Convergent Evolution chapter, but otherwise, you need to wait for more.
- mdaniel 3y agoThe "please let me know" <https://www.dedp.online/appendix/feedback https://www.dedp.online/appendix/feedback> link at the bottom of the page gives an ugly Apache 404 heh, I would send feedback about that but ... --- update: so, it seems the Feedback in the sidebar <https://www.dedp.online/appendix/feedback.html https://www.dedp.online/appendix/feedback.html> works so it was just a missing page extension. While reading that I discovered the GitHub repo for the project is private, which explains why I couldn't find it, either
- articsputnik 3y agoAuthor of the book here — yes, I used Mdbook (https://github.com/rust-lang/mdBook https://github.com/rust-lang/mdBook), and the links strangely end with `.html. as you correctly updated. Initially, I wanted to change it but left it as it was.
- JAlexoid 3y agoIt's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background. To the point that just 2 years ago, when applying for DE roles I would be confused why majority of my screening questions came in the form of "how well do you know SQL". Today I do the same as I did 3-4 years ago, but I am no longer a data engineer.
- lysecret 3y agoYea same. Also, honestly feels like the way the field is progressing it will just be eaten up by an SWE role. Feel the same for ML engineer and many other specialized roles.
- JAlexoid 3y agoI don't think that SWEs will. The software and services are going to be getting advanced enough to just eliminate the need for a dedicated team to build ETL. People with relevant domain knowledge will have an easier time to deliver their work product, without the overhead of building phase. To get a reasonably good data platform - point-and-click ETL service, SAAS offering and the likes of Metabase - are already good enough for medium enterprises... and beat Databricks offerings for speed(setup, delivery and operation) in reporting and operational data access. I am absolutely sure that there will be a massive contraction in the DS, DE and ML opportunity market in the next few years. The major companies will consolidate and jobs in those domains will only be available at only a handful of companies... or extremely specialized startups.(much like chip design is now consolidated) Long story short, for companies - you probably don't need DS, ML and DE departments.
- stvltvs 3y agoMy experience is that many established companies are still struggling to get adequate operational reporting. Data engineers are still helpful to move the data necessary to make that happen. DS and ML become useful later once there's a more mature data culture and infrastructure. Otherwise you have analysts spending most of their time doing data engineering so they have something to analyze.
- g0xA52A2A 3y agoMaybe I'm just of a different vintage but I'm just not a fan of these e-mail newsleters that seem to be the trend these days. I'd much rather follow a Github repo for this sort of thing.
- cloakedcode 3y agoEmailNewsletters.Age > GitHub.Age ? “Showing my vintage” : “Kids these days”
- maddynator 3y agoDoes anyone know what tech is used here to create the online book?
- emlos 3y agolooks like mdBook https://github.com/rust-lang/mdBook https://github.com/rust-lang/mdBook
- maddynator 3y agoAwesome ty!!
- articsputnik 3y agoYep, it is. Super happy with it. I added some custom themes and links to my second brain, but other than that, all mdBook. I also have more notes about writing a book with Markdown and self-publish at https://www.ssp.sh/brain/create-a-book-on-markdown/ https://www.ssp.sh/brain/create-a-book-on-markdown/.
- alexyz12 3y agoAs a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken. Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like you are as a DE. The needing can be nice...
- Centigonal 3y agoI work in data, but I've seen a lot of comments from SWEs who are tired of building the same CRUD web app over and over again. The grass is always greener?
- edmundsauto 3y agoAlso a DE - the “data as a service” model is broken. Data engineers need to be upstream producing insights because they have the best feel for the warts. Companies that treat DE as IT (“service model”) do not have the right perspective and will not hire the kind of DE who are difference makers. Data teams should be partners.
- deleted 3y ago[deleted]
- thegabriele 3y agoI work as DE in a local medium business and since there's no giant all-in-one ERP my role is essentialy build solutions to improve (or create) analytics to act upon. It's working and even if i'm not at spaceX, i feel like i'm inventing something. But, since the size of it all..i also fix things (software and hardware). Now, the problem is to convince other business that they might need my services.
- moandcompany 3y agoI used to give a lightning talk at Google called "Life of a Data Engineer" (~6+ years ago), where I tried to explain to Google Software Engineers and Product Managers the concept of a "Data Engineer" in other parts of industry, as well as Google Cloud Customers. In that talk, I outlined three (3) primary archetypes which still exist today: - Data Engineering for Business Intelligence (often a rebranding of yesterday's "DBAs") - Data Engineering for Data Science - Data Engineering for Machine Learning (precursor for today's "ML Engineers") Having worn all of these hats in the past, and many others, in different contexts, one of the key points I wanted to convey was that, in practice: - The Role of a Data Engineer on a Team is Complementary and Often Defined By The Tasks That Others Don’t (Want To) Do Well There are many guides, tutorials, and courses that like to talk about tools and technologies, and yet very few that get real about organizational structures and the dynamics that cause these archetypes to emerge, peoples' desires to rebrand themselves as "Data Engineers," "ML Engineers," or "(Data) Software Engineers" instead. There was an interesting survey of ~600 "Data Engineers" by DataKitchen from 2021 ago where they had some very relatable, yet eye-opening findings: - 97% report experiencing burnout in their day-to-day jobs. - 70% say they are likely to leave their current company for another data engineering job in the next 12 months. - 79% have considered leaving the industry entirely. - 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations. - 87% say they are blamed when things go wrong. - 69% say their company’s data governance policies make their day-to-day job more difficult. - 89% report frequent disruptions to work-life balance due to unplanned work. (source: https://datakitchen.io/white_papers/2021-data-engineering-survey-burned-out-data-engineers-are-calling-for-dataops/ https://datakitchen.io/white_papers/2021-data-engineering-su...)
- _aaed 3y agohttps://f.hubspotusercontent10.net/hubfs/2866765/Whitepapers/White%20Paper%20-%202021%20Data%20Engineering%20Survey.pdf https://f.hubspotusercontent10.net/hubfs/2866765/Whitepapers... So people don't have to give their details for the survey
- anonu 3y agoThis is a pretty cool resource and lots of effort to put it together. I spent ten minutes reading the first few pages. I have to admit it's a bit hard to read. I would describe it as fairly verbose with lots of preamble before actually saying what you want to say. I also think this is very fixable with some editing and iterations. Just my 2 cents.