Why does it have to be hierarchical? Edit: Just to be clear, you have a large unlabeled data set which you want to label and turn into a classifier? And you h

Why does it have to be hierarchical? Edit: Just to be clear, you have a large unlabeled data set which you want to label and turn into a classifier? And you have considered bootstrapping a semisupervised classifier with topic modeling. I would not suggest that. If you've considered word vectors already, consider sentence vectors. Maybe cluster those and see if you get any salient clusters out of it and try to use those as machine labeled classes. You'll need to edit the results if you do that. And then at that point you can either do sentence similarity to perform classification, or k nearest neighbor, or whatever. Or just label everything yourself and train a baseline linear SVM. To echo the other user's point, you're going to end up doing a lot of manual labeling either way. Try to make sure your data set is balanced. Keep going until your evaluation numbers look good. As for evaluating the class accuracy, cross validation is a good way of measuring that. sklearn has a lot of great libraries for that and there's lots of information elsewhere on the internet.