3 ms·

Are incoming articles just pulled via RSS and then a sorting algorithm decides what is displayed?

• cpayne624 13y ago
  Here's the basic (scheduled, async) process... * background threads continuously check source RSS/ATOM for new entries, respecting 304s (via TTL/ETAG/LASTMOD headers if provided); there's also a configurable throttle to limit the number of times any particular source will be checked, irrespective of anything else * when new RSS/ATOM entries are found, the _full_ article from the host site is retrieved & parsed and evaluated for keywords to figure out what main topics the article's about; lots of feeds are partial, so being able to grab the full article content (w/out the header/comment/nav/footer cruft) is necessary for it to work * then the full article is checked for a "suitable" (size/dimensions) image; if one's found, it's pulled down, resized/optimized, and stored locally for display alongside the entry when browsing the site * then the (full article) text is added to the search index; again, w/partial feeds the full text is necessary to make search useful * the "research" function leverages the YouTube, Wikipedia, Bing, Reddit, Slideshare (although I may be yanking this out soon; hella-slow responses sometimes), and Amazon APIs. For display, the "grid" lists all entries in reverse chrono that had a suitable image. The "index" lists every entry.