7 ms·

Besides niche languages, there are also niche roles. For example, Data Engineers and DevOps Engineers are both in low supply and high demand. I've worked at tw

Besides niche languages, there are also niche roles. For example, Data Engineers and DevOps Engineers are both in low supply and high demand. I've worked at two companies (Apple and Spotify) that I'm pretty sure I would never have been able to get into as a "common" engineer or data scientist with thousands of other applicants for the same role.

mahathu 4y ago
What does a data engineer do?
- papandada 4y ago
  Previously known or associated with terms such as ETL or ELT or data warehousing or ... In my experience tends to be involved heavily with databases, data movement, data modelling. You might think of it in terms of the work that needs to be done before data analysis is possible. Though as with many things in data, labels are imprecise and considerable overlap.
- beckingz 4y ago
  Suffer mostly. Joke aside, data engineers do a wide variety of engineering work related to data. Often that's around data warehousing and business intelligence, sometimes it's work on big data streaming or storage systems, or building yet another integration for marketing. The breadth is a challenge, which makes it fun and also hard.
  - papandada 4y ago
    No, you had it right the first time. It's fairly dull, monotonous drudgery. It's not fun , at least for any unconstrained definition of fun, for the majority of work that needs to be done. I speak as someone who does this for a living for over 10 years. I'm in such high demand as an independent contractor I think about starting a consultancy but one of my big concerns is how to recruit people with a high tolerance for blandness, ambiguity, as well as high conscientiousness. I think the subject matter is not difficult but the personal attributes required to persevere and succeed are rare. For me it's been the "muck is brass" career approach.
    - e-pelaza 4y ago
      Disclaimer: I'm in the process of making the switch to software development, and I'm from a 3rd world country. Is there any chance for someone with my profile to get a job in this? Remotely of course, as any remote job would pay times better than anything I could get in my home country. If yes, where does one learn about it more? I see Coursera has some course, but better asking from someone with your experience I guess. My approach to a job is that you do it in order to get money to fulfill whatever you enjoy in life, as long as I'm paid I'll be the best I can. Thanks
      - deleted 4y ago
        [deleted]
    - pphysch 4y ago
      It's digital plumbing, almost literally: the pipes aren't very interesting, and you have to often wade through shit to put em in Thankless work so that the analysts can turn on the faucet and get nice clean data for their fancy algorithms and models
      - beckingz 4y ago
        It's plumbing, but you're also expected to know where the water came from, the exact specifications of the treatment plant that cleaned it, exactly what is in the water down to the parts per billion, and have ideas for what kind of faucet will make everyone trust the water the most. Also bonus if you know how to run a business and give guidance on what type of house to buy before putting in pipes.
  - bogomipz 4y ago
    What are the skill sets required? Proficiency in SQL seems like an obvious one but else needs to be in the toolbox so to speak?
    - RhodesianHunter 4y ago
      Kafka is a very useful one.
    - beckingz 4y ago
      Core is SQL + Python (or Java/Scala for the actual Engineering part). Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others. Also every cloud has it's own stack that is sometimes worth using.
      - bogomipz 4y ago
        Thanks, what's the name of the AWS stack? Is it worth using?
- biellls 4y ago
  Depends on where you work at. A lot of the time they do ETL/ELT mostly with airflow nowadays. Some use little more than SQL and if they're lucky something like DBT to template and schedule the queries. A lot of SQL now runs on data warehouses of which snowflake and Google bigquery stand out, but it's SQL nevertheless. A few lucky ones get to use Spark which is more interesting as it's written in scala or python, but from my experience 99% of the time businesses are better off training business analysts to use SQL and writing the queries in snowflake. Another interesting area is real time data, for existing e with Kafka. This can be challenging and fulfilling except a lot of the time businesses people will tell you it's cool but they don't want the data to keep changing so could you please aggregate it every day... So you end up with an overly complicated batch system anyways. For me the sweet spot of practicality right now is Airflow/DBT/python/snowflake.
  - marcosdumay 4y ago
    > A few lucky ones get to use Spark which is more interesting as it's written in scala or python If you want a fun language, it's really the wrong place to be. The work can be very fun, but you must take the fun from data analysis. The best software engineering can do for you is to not get on your way (and SQL excels on this). I'd say the people working with Spike are the unlucky ones. Because they have to focus more on their tools, and not on the analysis.
e-pelaza 4y ago
Can you get in as self-taught?
- biellls 4y ago
  Probably pretty easily if you're a good software engineer. I think they don't require as much technical knowledge, though I can only speak on data engineering from first hand experience. A lot of data engineers aren't very good programmers (some use little more than SQL) though there's always outliers.
- travisjungroth 4y ago
  Yes. These roles are more accepting for self-taught people than most software roles (not saying it's wide open). Anyone can email me (last name at gmail) if they want to chat.

mahathu 4y ago

What does a data engineer do?

papandada 4y ago
Previously known or associated with terms such as ETL or ELT or data warehousing or ... In my experience tends to be involved heavily with databases, data movement, data modelling. You might think of it in terms of the work that needs to be done before data analysis is possible. Though as with many things in data, labels are imprecise and considerable overlap.
beckingz 4y ago
Suffer mostly. Joke aside, data engineers do a wide variety of engineering work related to data. Often that's around data warehousing and business intelligence, sometimes it's work on big data streaming or storage systems, or building yet another integration for marketing. The breadth is a challenge, which makes it fun and also hard.
- papandada 4y ago
  No, you had it right the first time. It's fairly dull, monotonous drudgery. It's not fun , at least for any unconstrained definition of fun, for the majority of work that needs to be done. I speak as someone who does this for a living for over 10 years. I'm in such high demand as an independent contractor I think about starting a consultancy but one of my big concerns is how to recruit people with a high tolerance for blandness, ambiguity, as well as high conscientiousness. I think the subject matter is not difficult but the personal attributes required to persevere and succeed are rare. For me it's been the "muck is brass" career approach.
  - e-pelaza 4y ago
    Disclaimer: I'm in the process of making the switch to software development, and I'm from a 3rd world country. Is there any chance for someone with my profile to get a job in this? Remotely of course, as any remote job would pay times better than anything I could get in my home country. If yes, where does one learn about it more? I see Coursera has some course, but better asking from someone with your experience I guess. My approach to a job is that you do it in order to get money to fulfill whatever you enjoy in life, as long as I'm paid I'll be the best I can. Thanks
    - deleted 4y ago
      [deleted]
  - pphysch 4y ago
    It's digital plumbing, almost literally: the pipes aren't very interesting, and you have to often wade through shit to put em in Thankless work so that the analysts can turn on the faucet and get nice clean data for their fancy algorithms and models
    - beckingz 4y ago
      It's plumbing, but you're also expected to know where the water came from, the exact specifications of the treatment plant that cleaned it, exactly what is in the water down to the parts per billion, and have ideas for what kind of faucet will make everyone trust the water the most. Also bonus if you know how to run a business and give guidance on what type of house to buy before putting in pipes.
- bogomipz 4y ago
  What are the skill sets required? Proficiency in SQL seems like an obvious one but else needs to be in the toolbox so to speak?
  - RhodesianHunter 4y ago
    Kafka is a very useful one.
  - beckingz 4y ago
    Core is SQL + Python (or Java/Scala for the actual Engineering part). Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others. Also every cloud has it's own stack that is sometimes worth using.
    - bogomipz 4y ago
      Thanks, what's the name of the AWS stack? Is it worth using?
biellls 4y ago
Depends on where you work at. A lot of the time they do ETL/ELT mostly with airflow nowadays. Some use little more than SQL and if they're lucky something like DBT to template and schedule the queries. A lot of SQL now runs on data warehouses of which snowflake and Google bigquery stand out, but it's SQL nevertheless. A few lucky ones get to use Spark which is more interesting as it's written in scala or python, but from my experience 99% of the time businesses are better off training business analysts to use SQL and writing the queries in snowflake. Another interesting area is real time data, for existing e with Kafka. This can be challenging and fulfilling except a lot of the time businesses people will tell you it's cool but they don't want the data to keep changing so could you please aggregate it every day... So you end up with an overly complicated batch system anyways. For me the sweet spot of practicality right now is Airflow/DBT/python/snowflake.
- marcosdumay 4y ago
  > A few lucky ones get to use Spark which is more interesting as it's written in scala or python If you want a fun language, it's really the wrong place to be. The work can be very fun, but you must take the fun from data analysis. The best software engineering can do for you is to not get on your way (and SQL excels on this). I'd say the people working with Spike are the unlucky ones. Because they have to focus more on their tools, and not on the analysis.

e-pelaza 4y ago

Can you get in as self-taught?

biellls 4y ago
Probably pretty easily if you're a good software engineer. I think they don't require as much technical knowledge, though I can only speak on data engineering from first hand experience. A lot of data engineers aren't very good programmers (some use little more than SQL) though there's always outliers.
travisjungroth 4y ago
Yes. These roles are more accepting for self-taught people than most software roles (not saying it's wide open). Anyone can email me (last name at gmail) if they want to chat.