I have not read all the code closely but it looks like you are scraping the HTML directly. True? It would be easier to use the JSON feed for a couple of reason

I have not read all the code closely but it looks like you are scraping the HTML directly. True? It would be easier to use the JSON feed for a couple of reasons: - the data is segregated into types of things that you otherwise have to back out from the HTML/CSS - you don't need logic for parsing multiple pages Have a look at code I wrote on this: https://github.com/hughdbrown/who-is-hiring https://github.com/hughdbrown/who-is-hiring The code to get a month of Who Is Hiring posts just hits `https://hn.algolia.com/api/v1/items/{article_id} https://hn.algolia.com/api/v1/items/{article_id}` with a GET request for a particular article_id. Finding the monthly article_ids is the only manual maintenance. The code is split into multiple commandline tools that extract JSON from HN, preprocess the JSON, and filter on what I was interested in (rust job postings). You should be able to replace downstream parts like preprocessing and filtering. See if any of this is helpful.