4 ms·

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), an

by C4stor 6y ago

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is either 0% or 100%. In turn, you quickly learn to write good tests. - You will learn distributed processing at a macro level, which in turn enlighten your thinking at a micro level. For example, even though the order of magnitudes are different, hitting data over network versus on disk is very much like hitting data on disk versus in cache. Except that when the difference ends up being in hours or days, you become much more sensible to that, so it's good training for your thoughts. - Data engineering is full of product decisions. What's often called data "cleaning" is in fact one of the import product decisions made in a company, and a data engineer will be consistently exposed to his company product, which I think makes for great personal development - Data engineering is fascinating. In adtech for example, logs of where ads are displayed are an unfiltered window on the rest of humanity, for the better or the worse. But it definitely expands your views on what the "average" person actually does on its computer (spoiler : it's mainly watching porn...), and challenges quite a bit what you might think is "normal" - You'll be plumbing technologies from all over the web, which might or might not be good news for you. So yeah, data engineering is great ! It's not harder than other specialties for developers, but imo, it's one of the fun ones !

secondcoming 6y ago
Indeed, adtech is a great place to work for anyone interesting in working with data. And yes, people working in adtech hate, and block, ads too.
alexpetralia 6y ago
The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.
- darksaints 6y ago
  > As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Exactly. You can't magically make a stateful problem stateless, you can merely move that state around. Sometimes moving state around means moving it somewhere that is appropriate and capable of expertly handling that data. But if you make those choices wrong, it makes every aspect of your application more complex. UI programming tried going down this idea of stateless programming, and for a while it was trendy to do so stuff like redux. The problem is that UIs are state machines. That's not an analogy, that is a literal statement. And it is true of all UI's...it's just as true of the transmission lever in your car as it is for your saas dashboard. You can't program stateless UIs...they would cease to be a UI. So at best, you can move that state around. And with most of these solutions (eg. redux), you end up pushing that state into a massive global singleton, where even simple things like the state of a single radio button needs to be fed through dozens of tightly coupled components in order to "statelessly" render. And even worse, you lose the extremely helpful distinction between UI state and domain state, mixing them both together into a gigantic shit stew.
- jsinai 6y ago
  >The other thing I'd emphasize here is dealing with "state". Data is effectively state. It gets even more complicated. It’s not just the current state that matters, but also the history (sometimes the entire history) up to that state.
pricci 6y ago
And where would you recommend someone to start a data engineering path. Any book, learning source?
- thecolorgreen 6y ago
  I have the same question and I believe the answer is in the same vein as someone who asks about software engineering. Books/courses are great for the concepts, but your goal should be to build something ASAP since that's where actual learning will come from.
- Karrot_Kream 6y ago
  A lot of these are just "garden variety" (distributed) systems problems. Dealing with systems with differing latency distributions, recovering from failure, acceptable tradeoffs between speed and accuracy, etc
- khaledh 6y ago
  Work at a company that has a good data engineering discipline. Shopify is hiring: https://www.shopify.ca/careers/2021 https://www.shopify.ca/careers/2021
- edmundsauto 6y ago
  Long time DE here. I recommend trying to build your own data warehouse around something you're interested in. Don't worry about teh scaling - focus on the core engineering, taking data from different places, combining it into a sensible data model, update it automatically every day. Add in more sources. It's shockingly difficult, and something that only experience can teach.
- Avalaxy 6y ago
  The book "designing data-intensive applications" is really really good, and covers all the concepts (although not per sé the tools) you need to understand.
theflyinghorse 6y ago
I wonder how: 1. one finds organizations that have data engineering 2. gets hired to said organization with software engineering background.
- khaledh 6y ago
  Shopify is hiring 2,021 engineers (not just data engineers) in 2021: https://www.shopify.ca/careers/2021 https://www.shopify.ca/careers/2021
- walleeee 6y ago
  Nearly any field of computational science likely needs skilled data engineers. You could search for topics that interest you online and contact people accordingly. I cold-emailed my current lab's P.I. and just asked for work. Search for "research software engineer" or "scientific computing professional" positions. Plenty of data engineering goes on in many fields (environmental science, climate modeling, high energy physics, physical chemistry, etc), and plenty of fields desperately need to develop an engineering culture (e.g., plant biology, my field), whatever interests you. Availability and compensation will vary by discipline.
kjerzyk 6y ago
Any recommendations on how to get started? Books, courses?

by C4stor 6y ago

secondcoming 6y ago

Indeed, adtech is a great place to work for anyone interesting in working with data. And yes, people working in adtech hate, and block, ads too.

alexpetralia 6y ago

The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.

darksaints 6y ago
> As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Exactly. You can't magically make a stateful problem stateless, you can merely move that state around. Sometimes moving state around means moving it somewhere that is appropriate and capable of expertly handling that data. But if you make those choices wrong, it makes every aspect of your application more complex. UI programming tried going down this idea of stateless programming, and for a while it was trendy to do so stuff like redux. The problem is that UIs are state machines. That's not an analogy, that is a literal statement. And it is true of all UI's...it's just as true of the transmission lever in your car as it is for your saas dashboard. You can't program stateless UIs...they would cease to be a UI. So at best, you can move that state around. And with most of these solutions (eg. redux), you end up pushing that state into a massive global singleton, where even simple things like the state of a single radio button needs to be fed through dozens of tightly coupled components in order to "statelessly" render. And even worse, you lose the extremely helpful distinction between UI state and domain state, mixing them both together into a gigantic shit stew.
jsinai 6y ago
>The other thing I'd emphasize here is dealing with "state". Data is effectively state. It gets even more complicated. It’s not just the current state that matters, but also the history (sometimes the entire history) up to that state.

pricci 6y ago

And where would you recommend someone to start a data engineering path. Any book, learning source?

thecolorgreen 6y ago
I have the same question and I believe the answer is in the same vein as someone who asks about software engineering. Books/courses are great for the concepts, but your goal should be to build something ASAP since that's where actual learning will come from.
Karrot_Kream 6y ago
A lot of these are just "garden variety" (distributed) systems problems. Dealing with systems with differing latency distributions, recovering from failure, acceptable tradeoffs between speed and accuracy, etc
khaledh 6y ago
Work at a company that has a good data engineering discipline. Shopify is hiring: https://www.shopify.ca/careers/2021 https://www.shopify.ca/careers/2021
edmundsauto 6y ago
Long time DE here. I recommend trying to build your own data warehouse around something you're interested in. Don't worry about teh scaling - focus on the core engineering, taking data from different places, combining it into a sensible data model, update it automatically every day. Add in more sources. It's shockingly difficult, and something that only experience can teach.
Avalaxy 6y ago
The book "designing data-intensive applications" is really really good, and covers all the concepts (although not per sé the tools) you need to understand.

theflyinghorse 6y ago

I wonder how: 1. one finds organizations that have data engineering 2. gets hired to said organization with software engineering background.

khaledh 6y ago
Shopify is hiring 2,021 engineers (not just data engineers) in 2021: https://www.shopify.ca/careers/2021 https://www.shopify.ca/careers/2021
walleeee 6y ago
Nearly any field of computational science likely needs skilled data engineers. You could search for topics that interest you online and contact people accordingly. I cold-emailed my current lab's P.I. and just asked for work. Search for "research software engineer" or "scientific computing professional" positions. Plenty of data engineering goes on in many fields (environmental science, climate modeling, high energy physics, physical chemistry, etc), and plenty of fields desperately need to develop an engineering culture (e.g., plant biology, my field), whatever interests you. Availability and compensation will vary by discipline.

kjerzyk 6y ago

Any recommendations on how to get started? Books, courses?