3 ms·
You know that's usually the hard part of ML projects, right? It may be boring, but it also tends to be the less least replaceable component.
by deepnotderp 8y ago
You know that's usually the hard part of ML projects, right? It may be boring, but it also tends to be the less least replaceable component.
- bitL 8y agoYes, and there is zero prestige within company, awful lot of high-pressure work, no control over bugs in infrastructure pieces that tend to blow up unexpectedly in production, getting stuck on a certain stack that tends to age badly (who remembers Hadoop or Spark 1.3 these days?), making one quickly replaceable by younger folks with the latest hyped tech, small understanding of the data science part, no joy from constructing great algorithms either. It's a needed job of course; it might pay well at this very moment but it's a perfect target for automation once latest cognitive automation research bubbles down to production. If you want to shine, do Deep Learning (research if possible), low-level distributed systems you have control of as an author of an important piece of infrastructure or the actual data modeling and predictions.
- amznthrowaway2 8y agoHe's right about how data pipeline work is regarded by most ML scientists. It often ends up being extremely hard work and is regarded by others as useless grunt work. There is no big reward, constant overtime due to unexpected issues and lots of stress.
- closeparen 8y agoSQL skills are expected of every data scientist and backend engineer, but are also found in a much larger and cheaper labor pool. As far as I can tell, the data engineering role exists to take advantage of those economics. ETL pipeline queries and configs are absolutely critical, make or break an ML project, and account for most of its labor hours. But they are the most commoditized part of it.