5 ms·

Here is the crawler project: https://github.com/MoserMichael/flagged-hn/ https://github.com/MoserMichael/flagged-hn/ . It crawls HN and creates a database recor

Here is the crawler project: https://github.com/MoserMichael/flagged-hn/ https://github.com/MoserMichael/flagged-hn/ . It crawls HN and creates a database record for each submission. The page is generated by formatting the result of the sql query. A bit boring, all written in python3. (I don't have the nerve to run the crawler and page generator on a daily basis; the page has instructions on how to install and run the script)

WalterGR 5y ago
Is the Hacker News API not useful for your project?
- MichaelMoser123 5y ago
 didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?print=pretty https://hacker-news.firebaseio.com/v0/item/29103056.json?pri... On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.
- MichaelMoser123 5y ago
 for this task I need to rescan the pages often, in order to check if the post got flagged. There is no way to know, if or when they rescan the submissions with the HN api.
 - WalterGR 5y ago
 Thanks for your responses. So you have to check the HTML (substring search) to see if a story is flagged? Edit: Answered my own question: yes: https://github.com/MoserMichael/flagged-hn/blob/master/crawl.py#L257 https://github.com/MoserMichael/flagged-hn/blob/master/crawl...
 - MichaelMoser123 5y ago
 Thanks for your feedback, it is very much appreciated! yes, it's part of the table row that includes the title, but if you take the text within the link tag only, then you don't get that indication - I think that's what the HN api might be doing. Example: <td class="title"> [flagged] <a href="https://www.dailywire.com/news/mcauliffe-education-plan-could-lead-to-1970s-style-busing" class="titlelink">McAuliffe Plan: Giving Parents a ‘Choice’ Leads to ‘Segregation’</a> (<a href="from?site=dailywire.com">dailywire.com</a>)</td> Actually if you ask me, I don't know why they are using this 'flagging' thing over here, they might have just moved a submission some pages away from the front pages, if they don't like it.
 - mattowen_uk 5y ago
 Users with enough Karma can 'vouch' for a flagged entry, and effectively de-flag it if they think the flagging is erroneous. Flagging isn't a one-way street.

WalterGR 5y ago

Is the Hacker News API not useful for your project?

MichaelMoser123 5y ago
didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?print=pretty https://hacker-news.firebaseio.com/v0/item/29103056.json?pri... On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.
MichaelMoser123 5y ago
for this task I need to rescan the pages often, in order to check if the post got flagged. There is no way to know, if or when they rescan the submissions with the HN api.
- WalterGR 5y ago
 Thanks for your responses. So you have to check the HTML (substring search) to see if a story is flagged? Edit: Answered my own question: yes: https://github.com/MoserMichael/flagged-hn/blob/master/crawl.py#L257 https://github.com/MoserMichael/flagged-hn/blob/master/crawl...
 - MichaelMoser123 5y ago
 Thanks for your feedback, it is very much appreciated! yes, it's part of the table row that includes the title, but if you take the text within the link tag only, then you don't get that indication - I think that's what the HN api might be doing. Example: <td class="title"> [flagged] <a href="https://www.dailywire.com/news/mcauliffe-education-plan-could-lead-to-1970s-style-busing" class="titlelink">McAuliffe Plan: Giving Parents a ‘Choice’ Leads to ‘Segregation’</a> (<a href="from?site=dailywire.com">dailywire.com</a>)</td> Actually if you ask me, I don't know why they are using this 'flagging' thing over here, they might have just moved a submission some pages away from the front pages, if they don't like it.
 - mattowen_uk 5y ago
 Users with enough Karma can 'vouch' for a flagged entry, and effectively de-flag it if they think the flagging is erroneous. Flagging isn't a one-way street.