4 ms·

Hi Mike, Can you talk about the tradeoffs or relationship between active learning and weak supervision from your point of view?

nailer 6y ago
Hey Luke! I'm a full stack / devops person so this is going to be high level but getting a couple of my ML researcher colleagues in for a deeper dive now. Short version: 1) we can get to the same level of accuracy with around 10% of the data points. Getting (and managing) a big enough data set to train a supervised learning model is the biggest thing slowing down ML deployments. 2) the model contacts a human when it can't label a data point with a high degree of confidence. You'll never have people with a bunch of specialist knowledge being asked to perform mundane data labelling tasks
razcle 6y ago
Hi lwhsiao, Raza here, author of the post. My high-level answer is weak-labelling overcomes cold starts and active learning helps with the last mile. More detail: We see weak learning as very complementary to active learning. By using labelling functions, you can quickly overcome the cold start problem and also better leverage external resources like knowledge bases. But most of the work in training ML systems often comes in getting the last few percentage points of performance. Going from good to good enough. This is where active learning can really shine because it guides you as to what data you really need to move model performance. At Humanloop, we've started with active learning tools but are also doing a lot of work on weak labelling.
- lwhsiao 6y ago
  Thanks for the detailed response :)
peadarohaodha 6y ago
adding to what Raza said - a consideration for both active learning and weak supervision is the need to construct a gold standard labelled dataset for model validation and testing purposes. At Humanloop, in addition to collecting training data, we are also using a form of active learning to speed up the collection of an unbiased test data set. Another consideration on the weak supervision side (for the snorkel style approach of labelling functions) is that creating labelling functions can be a relatively technical task, which may not be well suited for non-technical domain expert annotators for providing feedback to the model

nailer 6y ago

Hey Luke! I'm a full stack / devops person so this is going to be high level but getting a couple of my ML researcher colleagues in for a deeper dive now. Short version: 1) we can get to the same level of accuracy with around 10% of the data points. Getting (and managing) a big enough data set to train a supervised learning model is the biggest thing slowing down ML deployments. 2) the model contacts a human when it can't label a data point with a high degree of confidence. You'll never have people with a bunch of specialist knowledge being asked to perform mundane data labelling tasks

razcle 6y ago

Hi lwhsiao, Raza here, author of the post. My high-level answer is weak-labelling overcomes cold starts and active learning helps with the last mile. More detail: We see weak learning as very complementary to active learning. By using labelling functions, you can quickly overcome the cold start problem and also better leverage external resources like knowledge bases. But most of the work in training ML systems often comes in getting the last few percentage points of performance. Going from good to good enough. This is where active learning can really shine because it guides you as to what data you really need to move model performance. At Humanloop, we've started with active learning tools but are also doing a lot of work on weak labelling.

lwhsiao 6y ago
Thanks for the detailed response :)

peadarohaodha 6y ago

adding to what Raza said - a consideration for both active learning and weak supervision is the need to construct a gold standard labelled dataset for model validation and testing purposes. At Humanloop, in addition to collecting training data, we are also using a form of active learning to speed up the collection of an unbiased test data set. Another consideration on the weak supervision side (for the snorkel style approach of labelling functions) is that creating labelling functions can be a relatively technical task, which may not be well suited for non-technical domain expert annotators for providing feedback to the model