5 ms·

I can see how this would work for tabular and text data, where the labeling functions are well-defined. I don't understand, at all, how this would work with com

I can see how this would work for tabular and text data, where the labeling functions are well-defined. I don't understand, at all, how this would work with computer vision tasks where heuristics are pretty much impossible to define. That said, I don't see anything here that would prevent you from using a pre-trained conv net as a labeling function, but I expect that multiple conv nets trained on a small corpus of data would be biased and make correlated errors, which violate their assumptions. This looks super powerful in some cases, but I'm just not seeing how it can possibly generalize to every ML problem.

ZeroCool2u 6y ago
At the ODSC presentation I went to last year where the team presented they actually used a vision problem as their canonical example. It's hard to grasp without a concrete example, but it makes a lot of sense the way they explained it. For example, lets assume you want to identify something like a lung tumor. So you have many MRI images and they're all largely the same template of image. Using traditional image processing software like open CV, it's suprisingly easy to do more coarse grained tasks programmatically, like say, search this image for any circle that's brighter than the surrounding tissue and has a radius greater than say x.yz mm. If you find one, that function returns True if not False. That x.yz mm number is what you get from your radiologists that you work with to help you develop the labeling functions and this is just _one_ of the labeling functions. But basically it turns out if you construct a few of these functions with the help of domain experts and then use those functions all together with the information theory research the Snorkel folks do, you get pretty damn good performance!
- woeirua 6y ago
  Yeah, but for some tasks, like recognizing a car, or something else you'd have to write pretty sophisticated code to even get to a reasonable result. It's far easier in those cases to use supervised learning to have a NN learn to do it for you.
  - ajratner 6y ago
    Agreed! As noted in other answer, Snorkel certainly does not work for everything :) And indeed, in many cases it may be easier to express what you know extensionally (label examples) vs intensionally (write functions). A lot comes down to the unit cost per label over time- and whether it's more economical to label a bunch of data by hand vs. write LFs or similar. That's btw why a lot of examples of ML today are ones where data is (i) simple for non-experts to label, (ii) non-private and therefore easy to outsource for labeling, and (iii) low rate of change (e.g. images for self-driving, basic NLP stuff for chat bots, etc)- this kind of data can be labeled cheaply and once, so hand-labeled training sets are (barely) economically feasible to build manually. However, most data is not that easy or cheap to label, needs to be relabeled constantly to adapt to change, and thus the investment in a programmatic approach is often far better even if certainly not push-button!
ajratner 6y ago
First: Snorkel Flow absolutely does not generalize to every ML problem :). IMO defining where different systems and approaches do and don't work best is one of the most important and most challenging problems in ML systems research- as noted, we've worked to give detail on this for Snorkel over the years... no perfect answers, but some notes below: - As you imply, a lot has to do with the available sources of input signal- whether labeling functions, or 'transformation functions' for data augmentation, or other ops we've worked on... the input is obviously key. - For data modalities like image, video, etc: Often the most successful approach is to (A) rely on some pre-processed features or "primitives" and write labeling functions over these- as my co-founder Paroma in particular has published about over the years- and/or (B) use metadata - External models are definitely expressable as labeling functions, and we've worked on exactly that problem of modeling (local) biases and correlations!
- woeirua 6y ago
  Does it work for semantic segmentation? That's really where I'm struggling to see how this could work.
  - ajratner 6y ago
    More advanced structured prediction tasks are still definitely on the cutting edge- mainly IMO down to defining the semantics of the programmatic user input like labeling functions for these kinds of tasks. Some recent work (http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf) has extended these semantics for sequence tagging, as an example- so some exciting moves in this direction!

ZeroCool2u 6y ago

At the ODSC presentation I went to last year where the team presented they actually used a vision problem as their canonical example. It's hard to grasp without a concrete example, but it makes a lot of sense the way they explained it. For example, lets assume you want to identify something like a lung tumor. So you have many MRI images and they're all largely the same template of image. Using traditional image processing software like open CV, it's suprisingly easy to do more coarse grained tasks programmatically, like say, search this image for any circle that's brighter than the surrounding tissue and has a radius greater than say x.yz mm. If you find one, that function returns True if not False. That x.yz mm number is what you get from your radiologists that you work with to help you develop the labeling functions and this is just _one_ of the labeling functions. But basically it turns out if you construct a few of these functions with the help of domain experts and then use those functions all together with the information theory research the Snorkel folks do, you get pretty damn good performance!

woeirua 6y ago
Yeah, but for some tasks, like recognizing a car, or something else you'd have to write pretty sophisticated code to even get to a reasonable result. It's far easier in those cases to use supervised learning to have a NN learn to do it for you.
- ajratner 6y ago
  Agreed! As noted in other answer, Snorkel certainly does not work for everything :) And indeed, in many cases it may be easier to express what you know extensionally (label examples) vs intensionally (write functions). A lot comes down to the unit cost per label over time- and whether it's more economical to label a bunch of data by hand vs. write LFs or similar. That's btw why a lot of examples of ML today are ones where data is (i) simple for non-experts to label, (ii) non-private and therefore easy to outsource for labeling, and (iii) low rate of change (e.g. images for self-driving, basic NLP stuff for chat bots, etc)- this kind of data can be labeled cheaply and once, so hand-labeled training sets are (barely) economically feasible to build manually. However, most data is not that easy or cheap to label, needs to be relabeled constantly to adapt to change, and thus the investment in a programmatic approach is often far better even if certainly not push-button!

ajratner 6y ago

First: Snorkel Flow absolutely does not generalize to every ML problem :). IMO defining where different systems and approaches do and don't work best is one of the most important and most challenging problems in ML systems research- as noted, we've worked to give detail on this for Snorkel over the years... no perfect answers, but some notes below: - As you imply, a lot has to do with the available sources of input signal- whether labeling functions, or 'transformation functions' for data augmentation, or other ops we've worked on... the input is obviously key. - For data modalities like image, video, etc: Often the most successful approach is to (A) rely on some pre-processed features or "primitives" and write labeling functions over these- as my co-founder Paroma in particular has published about over the years- and/or (B) use metadata - External models are definitely expressable as labeling functions, and we've worked on exactly that problem of modeling (local) biases and correlations!

woeirua 6y ago
Does it work for semantic segmentation? That's really where I'm struggling to see how this could work.
- ajratner 6y ago
  More advanced structured prediction tasks are still definitely on the cutting edge- mainly IMO down to defining the semantics of the programmatic user input like labeling functions for these kinds of tasks. Some recent work (http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf) has extended these semantics for sequence tagging, as an example- so some exciting moves in this direction!