4 ms·

Myself and a few colleagues attended the _absolutely packed_ presentation the Snorkel folks did at ODSC in Boston last year and came away so convinced by this a

Myself and a few colleagues attended the _absolutely packed_ presentation the Snorkel folks did at ODSC in Boston last year and came away so convinced by this approach that we actually constructed our interns summer project around using Snorkel for multi-label classification on complex very domain specific financial documents. The project was successful and both us and our intern were very happy. The main lessons learned from this exercise helped us identify where our efforts would be shifted when using Snorkel. Of course there's never any free lunch, but Snorkel has what I believe to be a very reasonable and effective trade off. Snorkel provides both a huge decrease in overall costs, but critically it shifts costs towards the front of the development process. Writing a good set of labeling functions is a non-trivial piece of work. It requires the data scientist to have deep domain experience or a few fairly large blocks of time collaborating with and learning from a business user that is a already a domain expert. It has the upside though of forcing the data scientist to get a solid foundation of this domain knowledge, which I feel often times is underestimated in many ML projects. Anyways, congrats to the team! Looking forward to checking out your future work.

eggie5 6y ago
can you educate me on how you do multi-label w/ snorkel? As far as I can understand that's one it's largest drawbacks.
- ZeroCool2u 6y ago
  Fair question. I didn't personally supervise our last intern, it was my turn the summer before, so I'm not as deeply familiar with it. Now that you bring this up though, I think perhaps I may have misspoken. When I said muti-label, I think that was our goal originally, but because of the constraints of Snorkel you mentioned, we ended up reframing the problem into many single class models instead. They would both work, but because of how our business users worked, multi-label wasn't super important. For example, not all business users are interested in every label, so I think what happened was more than one model was trained, one for each label, and then ensembled based on the business users interests. Our final output allowed users to effectively sort, filter, and search documents based on any combination of these labels. Keep in mind too, some of these labels are fairly abstract, so just one of them was fairly powerful by itself and could perhaps power an entire team in some cases. I hope that helps, I'm sorry I can't go into too much more detail.
  - eggie5 6y ago
    yeah, you can do single-label w/ snorkel, but not multi-label. Multi-label snorkel would be the killer feature bc making the negatives (ie for a softmax) is very hard especially when you work w/ user-interaction systems with an unknown negative distribution.
    - ajratner 6y ago
      You can always do multi-label as a multi-task learning model (or just a set of binary models), which is something we (and many others) have explored before! A lot of the adjustments for mainline Snorkel have to do with (A) the semantics of the labeling functions (need to be able to express that something is not class A and/or have a general per-class prior) and (B) all the infra to support what is just now a bunch of independent per-label binary tasks, at base
      - eggie5 6y ago
        Snorkel has a label mutual exclusion assumption right? My core problem is a multi-label problem, but my snorkel data, from the LabelModel is inherently single-label (mutually exclusive). What is the prevailing recommendation to do multi-label w/ Snorkel? Is the below what you are currently recommending? For a given, k-wise multi-label problem: 1. Generate k binary datasets w/ LabelModel 2. Train k separate binary classifiers for each respective dataset 3. At inference/prediction time pass input though the k classifiers and get scores. Is this what the current recommendation is? Create a set of binary classifiers?
ajratner 6y ago
Thanks for this kind note! Would love to chat sometime and hear about your findings working with Snorkel- pros, cons, feature requests, etc. Multi-label is one on our list and under development (see comments below and elsewhere), would be interested to trade notes!

eggie5 6y ago

can you educate me on how you do multi-label w/ snorkel? As far as I can understand that's one it's largest drawbacks.

ZeroCool2u 6y ago
Fair question. I didn't personally supervise our last intern, it was my turn the summer before, so I'm not as deeply familiar with it. Now that you bring this up though, I think perhaps I may have misspoken. When I said muti-label, I think that was our goal originally, but because of the constraints of Snorkel you mentioned, we ended up reframing the problem into many single class models instead. They would both work, but because of how our business users worked, multi-label wasn't super important. For example, not all business users are interested in every label, so I think what happened was more than one model was trained, one for each label, and then ensembled based on the business users interests. Our final output allowed users to effectively sort, filter, and search documents based on any combination of these labels. Keep in mind too, some of these labels are fairly abstract, so just one of them was fairly powerful by itself and could perhaps power an entire team in some cases. I hope that helps, I'm sorry I can't go into too much more detail.
- eggie5 6y ago
  yeah, you can do single-label w/ snorkel, but not multi-label. Multi-label snorkel would be the killer feature bc making the negatives (ie for a softmax) is very hard especially when you work w/ user-interaction systems with an unknown negative distribution.
  - ajratner 6y ago
    You can always do multi-label as a multi-task learning model (or just a set of binary models), which is something we (and many others) have explored before! A lot of the adjustments for mainline Snorkel have to do with (A) the semantics of the labeling functions (need to be able to express that something is not class A and/or have a general per-class prior) and (B) all the infra to support what is just now a bunch of independent per-label binary tasks, at base
    - eggie5 6y ago
      Snorkel has a label mutual exclusion assumption right? My core problem is a multi-label problem, but my snorkel data, from the LabelModel is inherently single-label (mutually exclusive). What is the prevailing recommendation to do multi-label w/ Snorkel? Is the below what you are currently recommending? For a given, k-wise multi-label problem: 1. Generate k binary datasets w/ LabelModel 2. Train k separate binary classifiers for each respective dataset 3. At inference/prediction time pass input though the k classifiers and get scores. Is this what the current recommendation is? Create a set of binary classifiers?

ajratner 6y ago

Thanks for this kind note! Would love to chat sometime and hear about your findings working with Snorkel- pros, cons, feature requests, etc. Multi-label is one on our list and under development (see comments below and elsewhere), would be interested to trade notes!