5 ms·
Will try it again tomorrow. I could not find any information on site as to what will make this search engine different from what we already have? Also, the cor
by sangupta 6y ago
Will try it again tomorrow. I could not find any information on site as to what will make this search engine different from what we already have?
Also, the corpus is now 3162, roughly 250 links in last 2 hours - which is way too slow for a real world scenario. I though like the page for its simplicity and the experiment.
- shakna 6y agoI don't imagine that I, as a lone person, can actually build an engine to ever compete with Google or Bing, etc. It is not link crawling the entire web. Instead it is trying to find information that is current. You'll find relevant and current information from the BBC, CNN, and NPR for example. As well as stuff from places like LowTech magazine, the CCC, and FSF, etc. I haven't really put any information on the page at all about it. But mostly it scratches my own itch. But I wouldn't see it becoming a competitor if you're looking for "anything & everything", ever. (Though that 2 hours is actually you watching the nginx cache expire. The database updates hourly.)
- sangupta 6y agoGot it. So it sources data from a whitelisted set of sites and updates every hour. If the curated list can scratch my itch, I would love to come back. You mention you used some sort of NLP (mention of nltk before) - is it for summaries or reducing noise, or for bringing context to search terms.
- shakna 6y agoYeah, pretty much. I'm using nltk for - generating most (not all) of the summaries, and for building most of the tags that get attached to each article. It's also being used when searching the text of a summary.