3 ms·
Perhaps the reason that this approach works well is that, while the LLM gives you good general-purpose language processing, the decision tree learns about the s
by Matthyze 2y ago
Perhaps the reason that this approach works well is that, while the LLM gives you good general-purpose language processing, the decision tree learns about the specific dataset. And that combination is more powerful than either component.
- ellisv 2y agoIt’s the same reason LLMs don’t perform well on tabular data. (They can do fine but usually not was well as other models) Performing feature engineering with LLMs and then storing the embeddings in a vector database also allows you to reuse the embeddings for multiple tasks (eg clustering, nearest neighbor). Generally no one uses plain decision trees since random forest or gradient boosted trees perform better and are more robust.