3 ms·

You have a bug somewhere. Need to convert to lowercase for index and searches. For instance: - searching "houston" gives me a page full of results; - search

by MR4D 6y ago

You have a bug somewhere. Need to convert to lowercase for index and searches. For instance: - searching "houston" gives me a page full of results; - searching "Houston" gives me zero results; - searching "HOUSTON" give me zero results. Also, just a recommendation - if you are going to do a "Show HN", then you should have a good writeup about the tech used somewhere. We a re a curious bunch, and love to not only see what others are doing, but learn a bit about it. Finally, your results are fast, so that's a good start.

shakna 6y ago
Well. Forgetting lowercase but doing the rest of the search parsing was a bit of a mistake. That should now be fixed, though it might take a little while to propagate through the caches. --- There's really, really, not much to the tech behind it. Nothing surprising, and nothing really special. The website itself is 446 lines of a Python bottle app, behind an aggressive nginx cache. For example, the homepage is: @bottle.route("/") @bottle.view('home') def home(): # Get some random articles random_articles = [] for url in list(set(get_recent_urls(N=9))): random_articles.append(record(url)) random.shuffle(random_articles) random_articles = random_articles[:3] # Get the top tags tags = top_keywords(N=10) count = url_count() update = datetime.now() return {"articles": random_articles, "tags": tags, "count": count, "update": update, "tag_kind": 'Trending'} There's no... "magic" there. The link-fetcher is slightly larger at around 700 lines of Python, but most of it is taking edge-cases into account. Like finding where the timedata is hidden on a BBC page (comes as an epoch), or on CNN's lite site where it doesn't have a tag (and has to be manually parsed), etc. The vast majority, about half of the fetcher, is a series of fallbacks to keep trying until it finds something, before it finally gives up.
- MR4D 6y ago
  Thanks for sharing! How do you retrieve paywalled links (or do you skip those)?
  - shakna 6y ago
    I've tried to somewhat curate where the links are retrieved from, but the fetcher isn't javascript aware. In cases such as paywalls or heavy JS usage, you'll probably see an empty summary.

by MR4D 6y ago

shakna 6y ago

Well. Forgetting lowercase but doing the rest of the search parsing was a bit of a mistake. That should now be fixed, though it might take a little while to propagate through the caches. --- There's really, really, not much to the tech behind it. Nothing surprising, and nothing really special. The website itself is 446 lines of a Python bottle app, behind an aggressive nginx cache. For example, the homepage is: @bottle.route("/") @bottle.view('home') def home(): # Get some random articles random_articles = [] for url in list(set(get_recent_urls(N=9))): random_articles.append(record(url)) random.shuffle(random_articles) random_articles = random_articles[:3] # Get the top tags tags = top_keywords(N=10) count = url_count() update = datetime.now() return {"articles": random_articles, "tags": tags, "count": count, "update": update, "tag_kind": 'Trending'} There's no... "magic" there. The link-fetcher is slightly larger at around 700 lines of Python, but most of it is taking edge-cases into account. Like finding where the timedata is hidden on a BBC page (comes as an epoch), or on CNN's lite site where it doesn't have a tag (and has to be manually parsed), etc. The vast majority, about half of the fetcher, is a series of fallbacks to keep trying until it finds something, before it finally gives up.

MR4D 6y ago
Thanks for sharing! How do you retrieve paywalled links (or do you skip those)?
- shakna 6y ago
  I've tried to somewhat curate where the links are retrieved from, but the fetcher isn't javascript aware. In cases such as paywalls or heavy JS usage, you'll probably see an empty summary.