I agree the comparison could be a lot more fair. My goal was to show how adding in labels can give you better principle components, but I should've used an RBF

I agree the comparison could be a lot more fair. My goal was to show how adding in labels can give you better principle components, but I should've used an RBF kernel, or some kind of unsupervised kernel, instead of linear PCA. I really should go and make the code available. I spent a lot of time writing the RF kernel in Cython to get it to be fast. But it is so fragile and incomplete, I'm not sure I am ready to release and maintain it. Maybe I'll just post the scripts. Thanks for the links and insights about ANNs as combined kernel learners + classifiers. I'll check out the papers (I'm also surprised at what Deep Forest is. Not what I was expecting.)