4 ms·

Question about text classification without labeled data

1 points by jesuswr 5y ago

Hello! I am working on a text classifier but at the moment im quite lost on what to do. The classes form a tree with three levels, for example: class A (level 1), class A.1 (level 2, subclass of A) and class A.1.a (level 3, subclass of A.1). I have a lot of texts but without labels, also its hard to create a set of labeled data because of the big number of classes. I've looked at word embeddings, clustering algorithms, topic modeling, keyword extraction algorithms and other approaches but im not sure whats would be a good way to proceed. Also im not sure how to evaluate the success of the classifier, maybe i could build a small dataset but i wanted to know if there was any other option At the moment the idea that i like most is classifying by the level of the class, for example: given a text t, get the first level class of t, lets say B, then get the second level class of t (only taking the subclasses of B), lets say B.4, and finally get the third level class of t (only taking the subclasses of B.4), lets say B.4.c, and finally we could say that the class of t is B.4.c. So i wanted to ask here for advices to continue on this classifier and evaluate its success, thanks! :)

PaulHoule 5y ago
Making a training set is what "separates the men from the boys". If you make a labeled training set it is possible your project will succeed. If you don't you'll be just another person who messed around with BERT. You probably need somewhere between 1,000 to 20,000 labeled texts depending on what you want to do. My experience is I can label somewhere between 500 to 2000 items a day. I've done more than that (with images) but find that if I look at 4000 images a day for very long my visual system starts to malfunction.
- jesuswr 5y ago
  Thanks for the reply! How many labeled texts would you recommend for class? There are more than 100 third level classes, so i was trying to avoid it but if it's the best way and it's possible i could do it
  - PaulHoule 5y ago
    Well, if you have a large number of categories you have a difficult problem. One trouble is that as the number of classes expands the more likely your categorization is bad from an ontological perspective. For instance, something in class #34 might also be in class #79, you might have a hard time telling if something is in #55 or #86, etc. Even if your categorization can be implemented perfectly you will still have the problem of class imbalance. If we number classes by prevalence it might be that class #1 occurs 20% of the time, and class #99 occurs at a 1-in-a-million rate and you might not be able to find one at all in your data set. If you make a set of labeled classes that is representative of how often they occur you are going to have too many instances of the most common classes and not enough of the rare classes. In principle ML algorithms learn not just the characteristics of the classes but also learn the frequency. Pretty obviously you could enrich the labeled data with more examples of rare cases and difficult cases, but the algorithm will then get the wrong idea about the prevalence. I wish I could say I knew some way to tell the system that the "prior distribution" of classes is different from what's seen in the training examples, but I don't. Your best bet is to start with a simplified problem: target either the top of hierarchy or the most common examples. You've got to get into some place where imbalances are not outrageous.
    - jesuswr 5y ago
      Ok, thanks! I'll focus on the top level first
bpiche 5y ago
Why does it have to be hierarchical? Edit: Just to be clear, you have a large unlabeled data set which you want to label and turn into a classifier? And you have considered bootstrapping a semisupervised classifier with topic modeling. I would not suggest that. If you've considered word vectors already, consider sentence vectors. Maybe cluster those and see if you get any salient clusters out of it and try to use those as machine labeled classes. You'll need to edit the results if you do that. And then at that point you can either do sentence similarity to perform classification, or k nearest neighbor, or whatever. Or just label everything yourself and train a baseline linear SVM. To echo the other user's point, you're going to end up doing a lot of manual labeling either way. Try to make sure your data set is balanced. Keep going until your evaluation numbers look good. As for evaluating the class accuracy, cross validation is a good way of measuring that. sklearn has a lot of great libraries for that and there's lots of information elsewhere on the internet.
- jesuswr 5y ago
  Yeah, i have a large dataset and I'm looking for ways to work on the classifier, with or without putting labels on the data. And the classes than I'm interested in form a tree. Thanks! Yeah i think there's no way to scape the manual labelling

1 points by jesuswr 5y ago

PaulHoule 5y ago

Making a training set is what "separates the men from the boys". If you make a labeled training set it is possible your project will succeed. If you don't you'll be just another person who messed around with BERT. You probably need somewhere between 1,000 to 20,000 labeled texts depending on what you want to do. My experience is I can label somewhere between 500 to 2000 items a day. I've done more than that (with images) but find that if I look at 4000 images a day for very long my visual system starts to malfunction.

jesuswr 5y ago
Thanks for the reply! How many labeled texts would you recommend for class? There are more than 100 third level classes, so i was trying to avoid it but if it's the best way and it's possible i could do it
- PaulHoule 5y ago
  Well, if you have a large number of categories you have a difficult problem. One trouble is that as the number of classes expands the more likely your categorization is bad from an ontological perspective. For instance, something in class #34 might also be in class #79, you might have a hard time telling if something is in #55 or #86, etc. Even if your categorization can be implemented perfectly you will still have the problem of class imbalance. If we number classes by prevalence it might be that class #1 occurs 20% of the time, and class #99 occurs at a 1-in-a-million rate and you might not be able to find one at all in your data set. If you make a set of labeled classes that is representative of how often they occur you are going to have too many instances of the most common classes and not enough of the rare classes. In principle ML algorithms learn not just the characteristics of the classes but also learn the frequency. Pretty obviously you could enrich the labeled data with more examples of rare cases and difficult cases, but the algorithm will then get the wrong idea about the prevalence. I wish I could say I knew some way to tell the system that the "prior distribution" of classes is different from what's seen in the training examples, but I don't. Your best bet is to start with a simplified problem: target either the top of hierarchy or the most common examples. You've got to get into some place where imbalances are not outrageous.
  - jesuswr 5y ago
    Ok, thanks! I'll focus on the top level first

bpiche 5y ago

Why does it have to be hierarchical? Edit: Just to be clear, you have a large unlabeled data set which you want to label and turn into a classifier? And you have considered bootstrapping a semisupervised classifier with topic modeling. I would not suggest that. If you've considered word vectors already, consider sentence vectors. Maybe cluster those and see if you get any salient clusters out of it and try to use those as machine labeled classes. You'll need to edit the results if you do that. And then at that point you can either do sentence similarity to perform classification, or k nearest neighbor, or whatever. Or just label everything yourself and train a baseline linear SVM. To echo the other user's point, you're going to end up doing a lot of manual labeling either way. Try to make sure your data set is balanced. Keep going until your evaluation numbers look good. As for evaluating the class accuracy, cross validation is a good way of measuring that. sklearn has a lot of great libraries for that and there's lots of information elsewhere on the internet.

jesuswr 5y ago
Yeah, i have a large dataset and I'm looking for ways to work on the classifier, with or without putting labels on the data. And the classes than I'm interested in form a tree. Thanks! Yeah i think there's no way to scape the manual labelling