2 ms·
There are countless ML algorithms good for classification, no need for LLMs. But the problem is on how you feed data into them. Some websites are vast depots o
by DeathArrow 4y ago
There are countless ML algorithms good for classification, no need for LLMs.
But the problem is on how you feed data into them. Some websites are vast depots of pretty different type of texts spanning a huge number of domains. If you use data from /r/programming you will classify Reddit as a programming website. If you use data from /r/food you will classify it as a culinary website.
Some websites like Pinterest or dailymotion are media heavy, so using just text might not be helpful.
What I want to say is that actual classification is the last problem to solve, the problem is feeding relevant data into it.
- nerdponx 4y agoI think they mean "categorize" instead of "classify". That is, they are looking for some form of unsupervised clustering. In that case, I agree that LLMs are not the solution in and of themselves, but the word/sentence/documents embeddings obtained from the inner layers thereof might actually be useful. Back when BERT came out, the team I was on at the time had pretty good results using BERT-derived sentence vectors for a machine learning project.