2 ms·
Yeah, you do need ML. But a tweakable one. I’ve built a crawler, scraped the first 10 pages of all public domains I could find, then made those billions of page
by dstick 6y ago
Yeah, you do need ML. But a tweakable one. I’ve built a crawler, scraped the first 10 pages of all public domains I could find, then made those billions of pages freely queryable with ElasticSearch. The result: moderately good. It was great for discovering hidden gems. But you needed a lot of longtail keywords and exclusions.
There is a massive amount of textual garbage out there!
- axaxs 6y agoI gave it more thought in passing and am a bit curious what real, heavy ML might look like. And if there's value in such things. Example: I often find myself remembering bits of things, and querying Google, and the results are way off or completely irrelevent. Things a human would know in a matter of seconds, generally. I was for example trying to remember that newish Netflix movie that is very similar to the Michael Douglas movie "The Game." It feels like such a question should be easy for a sufficiently smart system. Something like Watson, I guess?, at the search engine level would be immensely mindblowing to me. The answer btw for anyone curious was "Rebirth." Nearly every review of said movie makes the comparison.
- rzzzt 6y agoThis is where semantic data would be really handy (admittedly, it's not a great example because the two movies are not linked at the moment): - https://dbpedia.org/page/The_Game_(1997_film) https://dbpedia.org/page/The_Game_(1997_film) - https://dbpedia.org/page/Rebirth_(2016_film) https://dbpedia.org/page/Rebirth_(2016_film)