3 ms·
The first big chunk of time is getting data into a usable format. Customers (internal or external) all have different data formats, and nothing is consistent. S
by cityhall 14y ago
The first big chunk of time is getting data into a usable format. Customers (internal or external) all have different data formats, and nothing is consistent. So you write parsers, deal with the corner cases, and get data into a form you can analyze.
Then you have to write software that lets you both experiment and release your models to production. That involves writing pipeline architectures to apply things like feature extraction and pruning in a consistent way, and to make sure the result can be serialized and deployed. Off the shelf packages typically haven't solved these problems very well, so you have to make sure the thing that looks good in the scripting environment is reproducible on new data.
Then when you have a model and can deploy it, you start working on automating the training process so the model automatically adapts as new data comes in. Usually the customer has gotten the impression this was happening on day one, so you have to rush to deliver it.
Then, you deal with customer complaints that something that would have been obvious to a human is wrong even after they corrected it.
At some point you try to measure the gains you're offering over the non-ML system you replaced, and try to tweak those metrics until they make you look good.
If you're lucky, you got to experiment with some interesting algorithms somewhere in the middle, but you probably got the best results from something fairly standard like random forests and not the latent bayesian slice sampler you dreamed up when you first heard about the problem.