2 ms·
Well. Forgetting lowercase but doing the rest of the search parsing was a bit of a mistake. That should now be fixed, though it might take a little while to pro
by shakna 6y ago
Well. Forgetting lowercase but doing the rest of the search parsing was a bit of a mistake. That should now be fixed, though it might take a little while to propagate through the caches.
---
There's really, really, not much to the tech behind it. Nothing surprising, and nothing really special.
The website itself is 446 lines of a Python bottle app, behind an aggressive nginx cache.
For example, the homepage is:
@bottle.route("/")
@bottle.view('home')
def home():
# Get some random articles
random_articles = []
for url in list(set(get_recent_urls(N=9))):
random_articles.append(record(url))
random.shuffle(random_articles)
random_articles = random_articles[:3]
# Get the top tags
tags = top_keywords(N=10)
count = url_count()
update = datetime.now()
return {"articles": random_articles, "tags": tags, "count": count, "update": update, "tag_kind": 'Trending'}
There's no... "magic" there.
The link-fetcher is slightly larger at around 700 lines of Python, but most of it is taking edge-cases into account. Like finding where the timedata is hidden on a BBC page (comes as an epoch), or on CNN's lite site where it doesn't have a tag (and has to be manually parsed), etc.
The vast majority, about half of the fetcher, is a series of fallbacks to keep trying until it finds something, before it finally gives up.