16 ms·

Data Engineering Cookbook

382 points by charlysl 7y ago

soobrosa 7y ago
Tempted to say how harmful things and advice is in here.
wiremine 7y ago
Looks like a great outline for an important topic! A few thoughts: 1. Given the title, I was expecting a more traditional Cookbook style book. I.e., "How do I..." followed by one or more recipes to answer the question. 2. The blueprint on page 39 is a good start: it includes the 4 main processes for data engineering. However, Display is only one use case. Training models, for example, is another. It also ignores real-time vs. batch processing. This comes up later in the book, but could be diagrammed more clearly. There are a lot of recipes for the overall architecture, and for each subprocess.
- herewego 7y ago
  This is a good example of respectful criticism. I wish others would follow your example.
nautilus12 7y ago
As a data engineer, the level of self promotion this guy is going through is insane. Good for him, if this catches on though, I hope I dont have to make myself into a data engineering celebrity/ thought leader in order to get jobs in the future.
HNisCurated 7y ago
What a weird title for a book about data science. I'm not a fan of the marketing gravy people dump on everything. The future will devolve to unhelpful titles like - "How to do Data Fking science"
- EForEndeavour 7y ago
  Are you referring to the term "data engineering" or the word "cookbook"?
- wickerman 7y ago
  Data engineering is not data science. Data engineers deliver the data for data scientists, data scientists use the data in models.
- squeaky-clean 7y ago
  This does not follow the typical programming "cookbook" structure, but it is a real thing in naming books. Like a conventional cookbook, a programming one has lots of examples and "recipes" for how to solve common problems, but doesn't necessarily provide the knowledge to be able to recognize or understand those problems. Like how almost no recipe book would tell you that it's customary to serve cake on birthdays or that you should be careful of the high sugar, or various cake alternatives, you would just find a section for cakes and it assumes you're making the correct choice in searching for a cake recipe. For example I own a cookbook on NLTK and flipping to a random page I got "How to remove insignificant words from a sentence"
lelima 7y ago
Good summary of he current technologies on the market right now, Maybe adding a few ETL tools will be great, they're still important :) I liked case of studies compilation thumps up.
CapmCrackaWaka 7y ago
Excessive self-promotion aside, I think high level documents like this are more important than people give them credit for. Computer science is: 1) Daunting to beginners 2) Changing constantly It is absolutely a field where people suffer from "I don't know what I don't know". 100-ft overviews of best practices and new buzz-words are refreshing, in my opinion. The criticisms I have of this document are all preference. I do not want to watch the podcasts or youtube videos, especially if I just want a high level overview of something. A ctrl-f of 'andreaskayy' returns 12 results. This guy's self promotions is everywhere, which is fine for some people, but it makes me think I'm getting 1 man's opinion on everything, and not an unbiased explanation of different technologies/methods.
- hdra 7y ago
  >Daunting to beginners I agree it is. But just I took a quick look at the book, while I understand the book is not done yet, looking at the table of contents, I doubt this would help. Like, do you have to cover kerberos, IP Subnetting, OSI IP model, Agile, git, REST API, docker, and many more in a single book about "data engineering"? If anything, it would confuse beginners even more. Its like the author tried to cram as many buzzwords as possible into a single book.
  - deleted 7y ago
    [deleted]
  - CapmCrackaWaka 7y ago
    I agree this author seems more interested in self-promotion than anything else. However, I don't think the purpose of this book is to cover these subjects in their entirety. Most of the time, with books like these, I would just like to see definitions, use cases, why it's used, maybe even what it replaced in the space it exists in. Materials like this aren't meant to make you an expert, they are just meant to show you what's out there, why it's out there, and (maybe) what was out there before. They give you context with which to google things. Either way, everyone expects something different from educational materials. I personally would not use this book, but I get the idea he was going for.
- 7y ago
meritt 7y ago
For anyone eager to read something now, Designing Data-Intensive Applications [1] is an excellent and completed book that covers nearly all of the same material with significant depth. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 https://www.amazon.com/Designing-Data-Intensive-Applications...
- soobrosa 7y ago
  I might even have stronger feelings than Vicky in terms of how 'useful' it is. If you want to build an other piece of tooling that we already have to muc of, then maybe. http://veekaybee.github.io/2019/04/11/attic-compsci/ http://veekaybee.github.io/2019/04/11/attic-compsci/
- ps101 7y ago
  Why is this book considered to be so good? I started it because it's been recommended on HN so much and I gave it up rather quickly because it was really dry and not all that focused on practical applications. Should I give it another go?
  - hdra 7y ago
    I highly recommend it. It does a very good job at explaining the "magic" behind all the data storage techniques, giving you a very good fundamentals and intuition of why each of them are good for certain kind of problems. After reading the book, googling for something like "mongodb v.s. cassandra" would start to feel as silly as googling for something like "javascript v.s. css" as you start to understand the fundamental differences between them. No more need to hope the vague Medium post you found while trying to decide which DB to use would match your use case closely enough.
    - laichzeit0 7y ago
      I’ve read the book too and didn’t feel like it covered much that isn’t covered by an undergraduate CS curriculum of databases and distributed systems. Perhaps the book appeals to developers without a formal education in computer science?
deleted 7y ago
[deleted]
hobbescotch 7y ago
Is there an equivalent job title for Data Engineer without the word “engineer”? As a data engineer without an engineering degree I don’t like having that word in my job title.
- red_hare 7y ago
  "Data Web Developer" But really, let the imposter syndrome go. I'm a third-generation computer engineer and the first one with a degree. I'm half the engineer my father is or my grandfather was. Whatever title you put on your resume is just you marketing the services you deliver. If you can do the work of a Data Engineer, you're a Data Engineer. If you're really self-conscious about it, get a gcloud/aws certification.
  - ska 7y ago
    Depends where you live, too. Some places having "engineer" in your title implies licensure.
- alvern 7y ago
  What part of the world are you working in? In the US it seems common to switch between software developer and software engineer interchangeably. But in the UK & AUS an engineer title means something completely different.
- wickerman 7y ago
  Oh man I'm the only data engineer I know who actually studied computer engineering. It's all right dude, it's just a title.
- randcraw 7y ago
  I suspect almost nobody who does the work of a data/ML engineer has a degree in engineering. Most are computer scientists or statisticians or mathematical scientists. Too few formal engineers choose this path for others to dwell on the term 'engineer'. The main advantage of using the data engineer label is that it describes what you do now so that others will understand. It doesn't describe your academic past, only your professional present. What's more, titles like data/ML scientist/engineer are still too new and ill-defined for anyone to worry too much about the exact work implied or the underlying credentials. 'Engineer' is just an attempt to imply that the role involves more involvement in production than exploration.
alienreborn 7y ago
This looks like blogspam or collecting a bunch of blogspam articles and turning it into a book.
madis 7y ago
How about calling it "Book of mostly empty chapters on various topics about software engineering with slight emphasis on data engineering"?
- kylek 7y ago
  How about constructive criticism...
  - squeaky-clean 7y ago
    Have you looked through the PDF? Many sections are entirely blank except a section header. Many are just links to a blog post or podcast. Data Warehouse vs Data Lake chapter is a single podcast link. The Hadoop chapter is 5 pages, mostly used by large diagrams and the docker chapter is less than 4 pages with half of the sections empty except a heading. The REST API chapter is less than 2 pages with a blank section headed OAuth Security. Data Visualization is entirely blank. The database chapter is mostly empty except for text about HDFS, and just links on MongoDB, ElasticSearch and InfluxDB. Apache Kafka gets its own mostly blank chapter. Most of the beginning chapters seem unrelated to data engineering. 3) Learn to Code. 4) Getting Started with Git. 5) Agile Development. 6) Learn how a computer works (section 1 is subtitled "CPU,RAM,GPU,HDD" but the chapter is empty). 7) Computer Networking - Data Transmission.
    - herewego 7y ago
      Where is your finished data engineering book? I would like to read it. How do you think a book gets written? Obviously you don't think that someone sits down, puts finger to keyboard, and then a book bursts into fruition. This is a work-in-progress kindly made freely available. Is it really fair to criticize the author for not having finished it yet?
      - squeaky-clean 7y ago
        > Where is your finished data engineering book? I would like to read it. So I need to have written a book to be able to download a PDF and see 85/100 pages are blank? I work as a data engineer and can tell you 50% of these chapter topics are not directly related to data engineering. There are no chapters in this book even close to 10% finished. If you want a book recommendation I'm seconding the suggestion in this thread of Designing Data-Intensive Applications. I have a copy 3 feet from me at the moment. > This is a work-in-progress kindly made freely available. Is it really fair to criticize the author for not having finished it yet? Please look through the PDF. This isn't just not done. This is not ready to share with anyone publicly. There is no useful information in this. There are probably under 20 paragraphs of original text. > Is it really fair to criticize the author for not having finished it yet? No, but I'm criticizing the fact that it's posted[0]. Not that they're working on something. I don't see the author here in this thread so my warning is to other readers. Just move on unless you're a book publisher looking for an author to pick up. The only real criticism anyone could offer about this would be about the chapter structure, because that's all that exists. I would recommend they drop all the chapters that are a CS101 equivalent. There's no need to explain git or the OSI model or grep. [0] edit, I want to clarify I mean just posted and dumped. If the author were here for questions or feedback I would feel differently. But with just this link as-is, there is no point in sharing.
closeparen 7y ago
Why is it useful to separate "data" from software engineering in general? What are some examples of programs that do not operate on data?
- squeaky-clean 7y ago
  The difference is where the difficulty of your problem lies. I work with serving and reporting ads, so most of our actual logic is simple. Glorified ETL work and fancy caching layer really. The problem comes about that we can have peaks of 20k reads and 20k writes per second with strict guarantees on response time and data consistency. All at the same time needing to be kept consistent across multiple datacenters in several regions. Your typical application won't hold up in that environment or can become extremely difficult to maintain. And I've met people at conferences who would say my use case still isn't "big data" and is pennies compared to their data streams. It really does become a class of algorithms and solutions on its own, just dealing with your real-world ability to manage that much data.
  - closeparen 7y ago
    This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech companies. Even specialists working on storage layer tech in particular are not called "data engineers," just backend engineers, although with more systems/infrastructure focus than others. People with the title "data engineer" at my company pretty much do ETL pipelines on Hadoop. I've done some ETL pipelines on Hadoop; seems like a tool that should be in the portfolio of any SWE. But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop; I have even seen them build RPC interfaces out of INSERT statements. Whereas a SWE is looking at a broader suite of options, and probably defaulting to backend services / OLTP databases. Hence my confusion. Maybe our use of the term is anachronistic? Or maybe they just know a bunch of HiveQL or pipeline design patterns that are too complex to fit into the heads of people who can also write services?
    - squeaky-clean 7y ago
spsphulse 7y ago
This is good. Although when I think of a cookbook, I'm mostly interested in some re-usable snippets of codes that can be used again and again. A good example would be Chris Albon's site https://chrisalbon.com https://chrisalbon.com For eg: - a recipe for splitting comma separated values in a column into multiple rows or - a data cleaning recipe that removes all unwanted spaces/trailing newlines/punctuations I've searched but never come across a compilation of such re-usable code snippets anywhere. Would be glad if anyone has any resources like this.
chrstphrhrt 7y ago
Anyone using non-JVM stuff in general? Nothing against the giant ecosystem around Hadoop/spark etc. I am currently using one Python process with Prefect for DAGs (https://docs.prefect.io/guide/ https://docs.prefect.io/guide/), custom API queries and Elasticsearch indexing code in a batch processing style and it seems to be going fine to ingest a 2TB dataset with the ability to generate rich errors, retry/resume etc. Besides maybe job listings, why would a real "plumber" who came from python and js NEED to dive into the whole Apache ecosystem?
ryantuck 7y ago
What is the benefit of making something like this into a PDF instead of a website? I don't think I ever see a PDF link online and think, "yes, this is the ideal format for ingesting this information."
ablekh 7y ago
While respecting the author's good intentions to share knowledge, I think that calling this set of (shallow) notes a "book" is an insult to all decent comprehensive books and their authors.
just_myles 7y ago
I see these kinds of books as good points for reference. These days I like video formats that reference the theory and show real world examples.

382 points by charlysl 7y ago

Tempted to say how harmful things and advice is in here.

Looks like a great outline for an important topic! A few thoughts: 1. Given the title, I was expecting a more traditional Cookbook style book. I.e., "How do I..." followed by one or more recipes to answer the question. 2. The blueprint on page 39 is a good start: it includes the 4 main processes for data engineering. However, Display is only one use case. Training models, for example, is another. It also ignores real-time vs. batch processing. This comes up later in the book, but could be diagrammed more clearly. There are a lot of recipes for the overall architecture, and for each subprocess.

herewego 7y ago
This is a good example of respectful criticism. I wish others would follow your example.

nautilus12 7y ago

As a data engineer, the level of self promotion this guy is going through is insane. Good for him, if this catches on though, I hope I dont have to make myself into a data engineering celebrity/ thought leader in order to get jobs in the future.

HNisCurated 7y ago

What a weird title for a book about data science. I'm not a fan of the marketing gravy people dump on everything. The future will devolve to unhelpful titles like - "How to do Data Fking science"

EForEndeavour 7y ago
Are you referring to the term "data engineering" or the word "cookbook"?
wickerman 7y ago
Data engineering is not data science. Data engineers deliver the data for data scientists, data scientists use the data in models.
squeaky-clean 7y ago
This does not follow the typical programming "cookbook" structure, but it is a real thing in naming books. Like a conventional cookbook, a programming one has lots of examples and "recipes" for how to solve common problems, but doesn't necessarily provide the knowledge to be able to recognize or understand those problems. Like how almost no recipe book would tell you that it's customary to serve cake on birthdays or that you should be careful of the high sugar, or various cake alternatives, you would just find a section for cakes and it assumes you're making the correct choice in searching for a cake recipe. For example I own a cookbook on NLTK and flipping to a random page I got "How to remove insignificant words from a sentence"

lelima 7y ago

Good summary of he current technologies on the market right now, Maybe adding a few ETL tools will be great, they're still important :) I liked case of studies compilation thumps up.

CapmCrackaWaka 7y ago

Excessive self-promotion aside, I think high level documents like this are more important than people give them credit for. Computer science is: 1) Daunting to beginners 2) Changing constantly It is absolutely a field where people suffer from "I don't know what I don't know". 100-ft overviews of best practices and new buzz-words are refreshing, in my opinion. The criticisms I have of this document are all preference. I do not want to watch the podcasts or youtube videos, especially if I just want a high level overview of something. A ctrl-f of 'andreaskayy' returns 12 results. This guy's self promotions is everywhere, which is fine for some people, but it makes me think I'm getting 1 man's opinion on everything, and not an unbiased explanation of different technologies/methods.

hdra 7y ago
>Daunting to beginners I agree it is. But just I took a quick look at the book, while I understand the book is not done yet, looking at the table of contents, I doubt this would help. Like, do you have to cover kerberos, IP Subnetting, OSI IP model, Agile, git, REST API, docker, and many more in a single book about "data engineering"? If anything, it would confuse beginners even more. Its like the author tried to cram as many buzzwords as possible into a single book.
- deleted 7y ago
  [deleted]
- CapmCrackaWaka 7y ago
  I agree this author seems more interested in self-promotion than anything else. However, I don't think the purpose of this book is to cover these subjects in their entirety. Most of the time, with books like these, I would just like to see definitions, use cases, why it's used, maybe even what it replaced in the space it exists in. Materials like this aren't meant to make you an expert, they are just meant to show you what's out there, why it's out there, and (maybe) what was out there before. They give you context with which to google things. Either way, everyone expects something different from educational materials. I personally would not use this book, but I get the idea he was going for.
7y ago

meritt 7y ago

For anyone eager to read something now, Designing Data-Intensive Applications [1] is an excellent and completed book that covers nearly all of the same material with significant depth. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 https://www.amazon.com/Designing-Data-Intensive-Applications...

soobrosa 7y ago
I might even have stronger feelings than Vicky in terms of how 'useful' it is. If you want to build an other piece of tooling that we already have to muc of, then maybe. http://veekaybee.github.io/2019/04/11/attic-compsci/ http://veekaybee.github.io/2019/04/11/attic-compsci/
ps101 7y ago
Why is this book considered to be so good? I started it because it's been recommended on HN so much and I gave it up rather quickly because it was really dry and not all that focused on practical applications. Should I give it another go?
- hdra 7y ago
  I highly recommend it. It does a very good job at explaining the "magic" behind all the data storage techniques, giving you a very good fundamentals and intuition of why each of them are good for certain kind of problems. After reading the book, googling for something like "mongodb v.s. cassandra" would start to feel as silly as googling for something like "javascript v.s. css" as you start to understand the fundamental differences between them. No more need to hope the vague Medium post you found while trying to decide which DB to use would match your use case closely enough.
  - laichzeit0 7y ago
    I’ve read the book too and didn’t feel like it covered much that isn’t covered by an undergraduate CS curriculum of databases and distributed systems. Perhaps the book appeals to developers without a formal education in computer science?

deleted 7y ago

[deleted]

hobbescotch 7y ago

Is there an equivalent job title for Data Engineer without the word “engineer”? As a data engineer without an engineering degree I don’t like having that word in my job title.

red_hare 7y ago
"Data Web Developer" But really, let the imposter syndrome go. I'm a third-generation computer engineer and the first one with a degree. I'm half the engineer my father is or my grandfather was. Whatever title you put on your resume is just you marketing the services you deliver. If you can do the work of a Data Engineer, you're a Data Engineer. If you're really self-conscious about it, get a gcloud/aws certification.
- ska 7y ago
  Depends where you live, too. Some places having "engineer" in your title implies licensure.
alvern 7y ago
What part of the world are you working in? In the US it seems common to switch between software developer and software engineer interchangeably. But in the UK & AUS an engineer title means something completely different.
wickerman 7y ago
Oh man I'm the only data engineer I know who actually studied computer engineering. It's all right dude, it's just a title.
randcraw 7y ago
I suspect almost nobody who does the work of a data/ML engineer has a degree in engineering. Most are computer scientists or statisticians or mathematical scientists. Too few formal engineers choose this path for others to dwell on the term 'engineer'. The main advantage of using the data engineer label is that it describes what you do now so that others will understand. It doesn't describe your academic past, only your professional present. What's more, titles like data/ML scientist/engineer are still too new and ill-defined for anyone to worry too much about the exact work implied or the underlying credentials. 'Engineer' is just an attempt to imply that the role involves more involvement in production than exploration.

alienreborn 7y ago

This looks like blogspam or collecting a bunch of blogspam articles and turning it into a book.

madis 7y ago

How about calling it "Book of mostly empty chapters on various topics about software engineering with slight emphasis on data engineering"?

kylek 7y ago
How about constructive criticism...
- squeaky-clean 7y ago
  Have you looked through the PDF? Many sections are entirely blank except a section header. Many are just links to a blog post or podcast. Data Warehouse vs Data Lake chapter is a single podcast link. The Hadoop chapter is 5 pages, mostly used by large diagrams and the docker chapter is less than 4 pages with half of the sections empty except a heading. The REST API chapter is less than 2 pages with a blank section headed OAuth Security. Data Visualization is entirely blank. The database chapter is mostly empty except for text about HDFS, and just links on MongoDB, ElasticSearch and InfluxDB. Apache Kafka gets its own mostly blank chapter. Most of the beginning chapters seem unrelated to data engineering. 3) Learn to Code. 4) Getting Started with Git. 5) Agile Development. 6) Learn how a computer works (section 1 is subtitled "CPU,RAM,GPU,HDD" but the chapter is empty). 7) Computer Networking - Data Transmission.
  - herewego 7y ago
    Where is your finished data engineering book? I would like to read it. How do you think a book gets written? Obviously you don't think that someone sits down, puts finger to keyboard, and then a book bursts into fruition. This is a work-in-progress kindly made freely available. Is it really fair to criticize the author for not having finished it yet?
    - squeaky-clean 7y ago
      > Where is your finished data engineering book? I would like to read it. So I need to have written a book to be able to download a PDF and see 85/100 pages are blank? I work as a data engineer and can tell you 50% of these chapter topics are not directly related to data engineering. There are no chapters in this book even close to 10% finished. If you want a book recommendation I'm seconding the suggestion in this thread of Designing Data-Intensive Applications. I have a copy 3 feet from me at the moment. > This is a work-in-progress kindly made freely available. Is it really fair to criticize the author for not having finished it yet? Please look through the PDF. This isn't just not done. This is not ready to share with anyone publicly. There is no useful information in this. There are probably under 20 paragraphs of original text. > Is it really fair to criticize the author for not having finished it yet? No, but I'm criticizing the fact that it's posted[0]. Not that they're working on something. I don't see the author here in this thread so my warning is to other readers. Just move on unless you're a book publisher looking for an author to pick up. The only real criticism anyone could offer about this would be about the chapter structure, because that's all that exists. I would recommend they drop all the chapters that are a CS101 equivalent. There's no need to explain git or the OSI model or grep. [0] edit, I want to clarify I mean just posted and dumped. If the author were here for questions or feedback I would feel differently. But with just this link as-is, there is no point in sharing.

closeparen 7y ago

Why is it useful to separate "data" from software engineering in general? What are some examples of programs that do not operate on data?

squeaky-clean 7y ago
The difference is where the difficulty of your problem lies. I work with serving and reporting ads, so most of our actual logic is simple. Glorified ETL work and fancy caching layer really. The problem comes about that we can have peaks of 20k reads and 20k writes per second with strict guarantees on response time and data consistency. All at the same time needing to be kept consistent across multiple datacenters in several regions. Your typical application won't hold up in that environment or can become extremely difficult to maintain. And I've met people at conferences who would say my use case still isn't "big data" and is pennies compared to their data streams. It really does become a class of algorithms and solutions on its own, just dealing with your real-world ability to manage that much data.
- closeparen 7y ago
  This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech companies. Even specialists working on storage layer tech in particular are not called "data engineers," just backend engineers, although with more systems/infrastructure focus than others. People with the title "data engineer" at my company pretty much do ETL pipelines on Hadoop. I've done some ETL pipelines on Hadoop; seems like a tool that should be in the portfolio of any SWE. But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop; I have even seen them build RPC interfaces out of INSERT statements. Whereas a SWE is looking at a broader suite of options, and probably defaulting to backend services / OLTP databases. Hence my confusion. Maybe our use of the term is anachronistic? Or maybe they just know a bunch of HiveQL or pipeline design patterns that are too complex to fit into the heads of people who can also write services?
  - squeaky-clean 7y ago

spsphulse 7y ago

This is good. Although when I think of a cookbook, I'm mostly interested in some re-usable snippets of codes that can be used again and again. A good example would be Chris Albon's site https://chrisalbon.com https://chrisalbon.com For eg: - a recipe for splitting comma separated values in a column into multiple rows or - a data cleaning recipe that removes all unwanted spaces/trailing newlines/punctuations I've searched but never come across a compilation of such re-usable code snippets anywhere. Would be glad if anyone has any resources like this.

chrstphrhrt 7y ago

Anyone using non-JVM stuff in general? Nothing against the giant ecosystem around Hadoop/spark etc. I am currently using one Python process with Prefect for DAGs (https://docs.prefect.io/guide/ https://docs.prefect.io/guide/), custom API queries and Elasticsearch indexing code in a batch processing style and it seems to be going fine to ingest a 2TB dataset with the ability to generate rich errors, retry/resume etc. Besides maybe job listings, why would a real "plumber" who came from python and js NEED to dive into the whole Apache ecosystem?

ryantuck 7y ago

What is the benefit of making something like this into a PDF instead of a website? I don't think I ever see a PDF link online and think, "yes, this is the ideal format for ingesting this information."

ablekh 7y ago

While respecting the author's good intentions to share knowledge, I think that calling this set of (shallow) notes a "book" is an insult to all decent comprehensive books and their authors.

just_myles 7y ago

I see these kinds of books as good points for reference. These days I like video formats that reference the theory and show real world examples.