3 ms·
Very cool project! How many websites do you have in your index? And how did you go about building it? I've been working on an engine for personal websites, cur
by ad404b8a372f2b9 5y ago
Very cool project! How many websites do you have in your index? And how did you go about building it?
I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
- marginalia_nu 5y agoAbout 21 million, and I'm crawling myself. Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text. I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
- ad404b8a372f2b9 5y agoIf you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.