4 ms·
Somewhat off-topic, but has anyone tried training a random forest at LLM scale? Like an RF with millions of trees and billions of branches? My intuition says it
by bglazer 2y ago
Somewhat off-topic, but has anyone tried training a random forest at LLM scale? Like an RF with millions of trees and billions of branches? My intuition says it could be much more efficient on CPUs, provided it works at all.
- skyde 2y agoForest are good at classification but they cannot leverage pre-training on unclassified data.
- ColonelPhantom 2y agoAren't LLMs a classification problem in a sense? "Given this text, classify it based on the next token" seems like a viable interpretation of the problem. Although there are a couple thousand or so classes, which might be a lot (but I know very little about this field).
- thesz 2y agohttps://gradientdescending.com/unsupervised-random-forest-example/ https://gradientdescending.com/unsupervised-random-forest-ex... You can cluster data using unsupervised random forests and then use these cluster indices as features.
- solresol 2y agoI'm working on it, but with a loss function that penalises hypernyms/hyponyms less than other kinds of mistakes. I'm vaguely optimistic that this is going to be more efficient.
- Der_Einzige 2y agoI’ve been talking about this for years! It’s fully possible!