3 ms·
> I've read over and over that the most time-consuming part of ML work is data collection & cleanup, and that matches my own experience. It is the same thing th
by Bartweiss 10y ago
> I've read over and over that the most time-consuming part of ML work is data collection & cleanup, and that matches my own experience. It is the same thing that killed so many data warehouse projects in the 90s. You don't need a Ph.D. to do it, but it is a tough and costly prerequisite. So it seems like you'll need non-ML programmers even for specifically ML projects.
Yeah, this is a very good point. I trained a simple RNN to play games a while back, and the "RNN" part was something anyone with a decent math background could self-teach pretty quickly. There are lots of tricks and speedups and complications I could have added, but the baseline product was pretty easy.
The "trained" part, though. That took huge effort, and research, and tinkering. Building up a proper dataset with usable input features was 90%+ of my effort. Most of it was too specific to standardize in any useful way, so working with an off-the-shelf system for the learner wouldn't have saved any decent amount of effort.