4 ms·
Let me give some more information. First, lesson learned, do not post on HN when tired. The ups and downs after posting are incredible, even if its just a priva
by tomw1808 12y ago
Let me give some more information. First, lesson learned, do not post on HN when tired. The ups and downs after posting are incredible, even if its just a private project. I couldn't be more thankful for the powerful emotions that arrive when I see my code in the open wild after weeks and months of coding. :)
The tool here is "just a frontend" I've build as an API-sideproject of mine. So, what does it do? In one sentence: Newscombinator gathers news from different sources, downloads it, makes a screenshot, cleans it, natural language process it, and push it through a MySQL DB into a Solr index.
The Solr-select handler is passed through a REST handler. Why that, why not directly the solr select handler? Because the REST handler also exposes endpoints to the MySQL Database. Some things are just not feasible in Solr alone. For example its hard to build aggregation queries in solr (think: SELECT AVG(points), SUM(points) FROM ... WHERE date_news_found ...), I've made that available in MySQL. And its easy to shift all that to a MariaDB cluster, or to an Amazon RDS, just in case...
What is available here is a RESTful interface to a Solr-Index with all Hacker-News articles, selected Reddits, etc, starting from Oct 2014. That is around 200k of articles. Including searchable content. I think that is a good base to start with for me.
Thank you in the meantime HN, I've learned something.