3 ms·
Is scraping and putting this in a structured format too inaccurate or expensive?
by runarberg 3mo ago
Is scraping and putting this in a structured format too inaccurate or expensive?
- motoxpro 3mo agoThat's the whole problem. If you have tons (100s of thousands or more) of labels, then you have "structured" data, but how do you correctly classify that scraped item into the correct label? Putting all the labels into the LLM is super expensive per call when you have millions of items to classify. You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.
- runarberg 3mo ago100s of thousand? In that case I would label about a 100 by hand and train a supervised learning model. This problem has also been solved for 3 decades now.
- Terr_ 3mo agoAhhh, but the LLM processing is the kind corporate leadership and investors will actually agree to buy! Orginally I started writing that as sarcasm, and now I'm not quite so sure.