3 ms·

Suffer mostly. Joke aside, data engineers do a wide variety of engineering work related to data. Often that's around data warehousing and business intelligence

Suffer mostly. Joke aside, data engineers do a wide variety of engineering work related to data. Often that's around data warehousing and business intelligence, sometimes it's work on big data streaming or storage systems, or building yet another integration for marketing. The breadth is a challenge, which makes it fun and also hard.

papandada 4y ago
No, you had it right the first time. It's fairly dull, monotonous drudgery. It's not fun , at least for any unconstrained definition of fun, for the majority of work that needs to be done. I speak as someone who does this for a living for over 10 years. I'm in such high demand as an independent contractor I think about starting a consultancy but one of my big concerns is how to recruit people with a high tolerance for blandness, ambiguity, as well as high conscientiousness. I think the subject matter is not difficult but the personal attributes required to persevere and succeed are rare. For me it's been the "muck is brass" career approach.
- e-pelaza 4y ago
  Disclaimer: I'm in the process of making the switch to software development, and I'm from a 3rd world country. Is there any chance for someone with my profile to get a job in this? Remotely of course, as any remote job would pay times better than anything I could get in my home country. If yes, where does one learn about it more? I see Coursera has some course, but better asking from someone with your experience I guess. My approach to a job is that you do it in order to get money to fulfill whatever you enjoy in life, as long as I'm paid I'll be the best I can. Thanks
  - deleted 4y ago
    [deleted]
- pphysch 4y ago
  It's digital plumbing, almost literally: the pipes aren't very interesting, and you have to often wade through shit to put em in Thankless work so that the analysts can turn on the faucet and get nice clean data for their fancy algorithms and models
  - beckingz 4y ago
    It's plumbing, but you're also expected to know where the water came from, the exact specifications of the treatment plant that cleaned it, exactly what is in the water down to the parts per billion, and have ideas for what kind of faucet will make everyone trust the water the most. Also bonus if you know how to run a business and give guidance on what type of house to buy before putting in pipes.
bogomipz 4y ago
What are the skill sets required? Proficiency in SQL seems like an obvious one but else needs to be in the toolbox so to speak?
- RhodesianHunter 4y ago
  Kafka is a very useful one.
- beckingz 4y ago
  Core is SQL + Python (or Java/Scala for the actual Engineering part). Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others. Also every cloud has it's own stack that is sometimes worth using.
  - bogomipz 4y ago
    Thanks, what's the name of the AWS stack? Is it worth using?

papandada 4y ago

No, you had it right the first time. It's fairly dull, monotonous drudgery. It's not fun , at least for any unconstrained definition of fun, for the majority of work that needs to be done. I speak as someone who does this for a living for over 10 years. I'm in such high demand as an independent contractor I think about starting a consultancy but one of my big concerns is how to recruit people with a high tolerance for blandness, ambiguity, as well as high conscientiousness. I think the subject matter is not difficult but the personal attributes required to persevere and succeed are rare. For me it's been the "muck is brass" career approach.

e-pelaza 4y ago
Disclaimer: I'm in the process of making the switch to software development, and I'm from a 3rd world country. Is there any chance for someone with my profile to get a job in this? Remotely of course, as any remote job would pay times better than anything I could get in my home country. If yes, where does one learn about it more? I see Coursera has some course, but better asking from someone with your experience I guess. My approach to a job is that you do it in order to get money to fulfill whatever you enjoy in life, as long as I'm paid I'll be the best I can. Thanks
- deleted 4y ago
  [deleted]
pphysch 4y ago
It's digital plumbing, almost literally: the pipes aren't very interesting, and you have to often wade through shit to put em in Thankless work so that the analysts can turn on the faucet and get nice clean data for their fancy algorithms and models
- beckingz 4y ago
  It's plumbing, but you're also expected to know where the water came from, the exact specifications of the treatment plant that cleaned it, exactly what is in the water down to the parts per billion, and have ideas for what kind of faucet will make everyone trust the water the most. Also bonus if you know how to run a business and give guidance on what type of house to buy before putting in pipes.

bogomipz 4y ago

What are the skill sets required? Proficiency in SQL seems like an obvious one but else needs to be in the toolbox so to speak?

RhodesianHunter 4y ago
Kafka is a very useful one.
beckingz 4y ago
Core is SQL + Python (or Java/Scala for the actual Engineering part). Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others. Also every cloud has it's own stack that is sometimes worth using.
- bogomipz 4y ago
  Thanks, what's the name of the AWS stack? Is it worth using?