2 ms·
About 21 million, and I'm crawling myself. Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turn
by marginalia_nu 5y ago
About 21 million, and I'm crawling myself.
Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.
I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
- ad404b8a372f2b9 5y agoIf you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.