3 ms·

AI problems turn into data problems. The happiest and best compensated people I know in that area have gone into data engineering, because data engineers are t

AI problems turn into data problems. The happiest and best compensated people I know in that area have gone into data engineering, because data engineers are the ones selling shovels in this gold rush.

sk11001 2y ago
I would give the exact opposite advice - data engineering is one of the least rewarding or respected career paths.
- mistrial9 2y ago
  a large organization that can pay stable wages over time is going to look for formal credentials when making a hiring decision. Further from that kind of (big,bureaucratic) group, are quickly assembling and then dissolving entities .. in other words "nice work if you can get it" .. with far less stability. Another term for this is "the Wild West" environment. There are some applicable analogies to the way Hollywood makes movies also perhaps, but with far less grounding. In the Hollywood example from what I know of.. small "tiger teams" assemble with fundamentals, then quickly farm out the sexy work to disposable contracting firms, who then hire even more disposable people with various skill levels. In other words, lots of fun and excitement but also lots of work place abuses and low stability. Over time, Hollywood formed unions for a surprisingly large number of roles (like writers) because the real truth of business is not pretty. Needless to say, Silicon Valley has moved very quickly, and the Hollywood stories are not exactly applicable.
- sevensor 2y ago
  If you find that this is the case in your organization, watch out. It's a sign that you're trying to build performative models that aren't grounded in reality. Depending on your organization's goals, this may take a long time to catch up with you.
- jakeoverflow 2y ago
  Data engineering feels like the DevOps of AI/ML. Important, but you have to have the right interests for it. This comes from someone doing web dev after studying data science, so I don't know how well it reflects reality as I've never worked as a Data Eng. myself
  - sk11001 2y ago
    > Data engineering feels like the DevOps of AI/ML. I wouldn't label it that way, AI/ML has its own ops and devops which is very distinct from data engineering.
akudha 2y ago
Could you please elaborate a bit more? I am your typical web-dev/frontend/backend developer, don't really know much about what is involved in data engineering. Do you mean collecting, cleaning data? Or setting up databases (if yes, how is it different from me managing my employer's databases, except for size)? In other words, what does a data engineer do all day?
- sgaur 2y ago
  Several data engineers that I know in this space (AI for business applications) are doing the usual ETL + mapping pipeline related work. Nowadays a lot of them are also having to deal with unstructured data such as textual reviews of products, service quality feedback, policy documents. Data engineers are developing the pipelines for chunking, vectorization, ingestion into RAG pipeline and for LLM training fine tuning. So it's still collecting, cleaning data, but quite different at the next level of detail, in my opinion.
  - akudha 2y ago
    Thank you. How does one get into this field? Web dev isn’t that exciting anymore
    - sevensor 2y ago
      The data engineers I know started out as data scientists. They discovered that their ability to make models that did anything useful was hampered by low-quality or absent data. Then they discovered that there was a lot more job satisfaction (not to mention much better pay) in making useful data available than there was in clicking the "train" button over and over in slightly different configurations.
    - gaws 2y ago
      > How does one get into this field? Network and meet someone who will give you, a web developer (a profession that is heavily stigmatized in the industry), a chance in a very competitive field.

sk11001 2y ago

I would give the exact opposite advice - data engineering is one of the least rewarding or respected career paths.

mistrial9 2y ago
a large organization that can pay stable wages over time is going to look for formal credentials when making a hiring decision. Further from that kind of (big,bureaucratic) group, are quickly assembling and then dissolving entities .. in other words "nice work if you can get it" .. with far less stability. Another term for this is "the Wild West" environment. There are some applicable analogies to the way Hollywood makes movies also perhaps, but with far less grounding. In the Hollywood example from what I know of.. small "tiger teams" assemble with fundamentals, then quickly farm out the sexy work to disposable contracting firms, who then hire even more disposable people with various skill levels. In other words, lots of fun and excitement but also lots of work place abuses and low stability. Over time, Hollywood formed unions for a surprisingly large number of roles (like writers) because the real truth of business is not pretty. Needless to say, Silicon Valley has moved very quickly, and the Hollywood stories are not exactly applicable.
sevensor 2y ago
If you find that this is the case in your organization, watch out. It's a sign that you're trying to build performative models that aren't grounded in reality. Depending on your organization's goals, this may take a long time to catch up with you.
jakeoverflow 2y ago
Data engineering feels like the DevOps of AI/ML. Important, but you have to have the right interests for it. This comes from someone doing web dev after studying data science, so I don't know how well it reflects reality as I've never worked as a Data Eng. myself
- sk11001 2y ago
  > Data engineering feels like the DevOps of AI/ML. I wouldn't label it that way, AI/ML has its own ops and devops which is very distinct from data engineering.

akudha 2y ago

Could you please elaborate a bit more? I am your typical web-dev/frontend/backend developer, don't really know much about what is involved in data engineering. Do you mean collecting, cleaning data? Or setting up databases (if yes, how is it different from me managing my employer's databases, except for size)? In other words, what does a data engineer do all day?

sgaur 2y ago
Several data engineers that I know in this space (AI for business applications) are doing the usual ETL + mapping pipeline related work. Nowadays a lot of them are also having to deal with unstructured data such as textual reviews of products, service quality feedback, policy documents. Data engineers are developing the pipelines for chunking, vectorization, ingestion into RAG pipeline and for LLM training fine tuning. So it's still collecting, cleaning data, but quite different at the next level of detail, in my opinion.
- akudha 2y ago
  Thank you. How does one get into this field? Web dev isn’t that exciting anymore
  - sevensor 2y ago
    The data engineers I know started out as data scientists. They discovered that their ability to make models that did anything useful was hampered by low-quality or absent data. Then they discovered that there was a lot more job satisfaction (not to mention much better pay) in making useful data available than there was in clicking the "train" button over and over in slightly different configurations.
  - gaws 2y ago
    > How does one get into this field? Network and meet someone who will give you, a web developer (a profession that is heavily stigmatized in the industry), a chance in a very competitive field.