3 ms·

Show HN: Page and crawler that show flagged HN submissions only

MichaelMoser123 5y ago
Here is the crawler project: https://github.com/MoserMichael/flagged-hn/ https://github.com/MoserMichael/flagged-hn/ . It crawls HN and creates a database record for each submission. The page is generated by formatting the result of the sql query. A bit boring, all written in python3. (I don't have the nerve to run the crawler and page generator on a daily basis; the page has instructions on how to install and run the script)
- WalterGR 5y ago
 Is the Hacker News API not useful for your project?
 - MichaelMoser123 5y ago
 didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?print=pretty https://hacker-news.firebaseio.com/v0/item/29103056.json?pri... On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.
 - MichaelMoser123 5y ago
 for this task I need to rescan the pages often, in order to check if the post got flagged. There is no way to know, if or when they rescan the submissions with the HN api.
 - WalterGR 5y ago
 Thanks for your responses. So you have to check the HTML (substring search) to see if a story is flagged? Edit: Answered my own question: yes: https://github.com/MoserMichael/flagged-hn/blob/master/crawl.py#L257 https://github.com/MoserMichael/flagged-hn/blob/master/crawl...
 - MichaelMoser123 5y ago
 Thanks for your feedback, it is very much appreciated! yes, it's part of the table row that includes the title, but if you take the text within the link tag only, then you don't get that indication - I think that's what the HN api might be doing. Example: <td class="title"> [flagged] <a href="https://www.dailywire.com/news/mcauliffe-education-plan-could-lead-to-1970s-style-busing" class="titlelink">McAuliffe Plan: Giving Parents a ‘Choice’ Leads to ‘Segregation’</a> (<a href="from?site=dailywire.com">dailywire.com</a>)</td> Actually if you ask me, I don't know why they are using this 'flagging' thing over here, they might have just moved a submission some pages away from the front pages, if they don't like it.
 mattowen_uk 5y ago
 Users with enough Karma can 'vouch' for a flagged entry, and effectively de-flag it if they think the flagging is erroneous. Flagging isn't a one-way street.
MichaelMoser123 5y ago
Wow! I did cause them to change: now they flag stories, but they also delete the title from the story, like here: https://news.ycombinator.com/item?id=29134228 https://news.ycombinator.com/item?id=29134228 . Now you only see [flagged] in the title. That makes my little exercise here much less useful. That's also a lesson: crawlers can be made ineffective by slight changes in the input. At least i had some fun at making this little exercise... On the other side: it makes this site a bit less interesting. I am sorry for having caused this, i had the best of intentions. Now i need to find another toy to tinker with. (My wife says that all of my projects are essentially useless, as they don't yield much in terms of money. Well, she has a point... :-( ) Here is a list of my projects: https://github.com/MoserMichael https://github.com/MoserMichael
- MichaelMoser123 5y ago
  Nah, they had that already. Just got it after scanning all of the entry id's by range.

Here is the crawler project: https://github.com/MoserMichael/flagged-hn/ https://github.com/MoserMichael/flagged-hn/ . It crawls HN and creates a database record for each submission. The page is generated by formatting the result of the sql query. A bit boring, all written in python3. (I don't have the nerve to run the crawler and page generator on a daily basis; the page has instructions on how to install and run the script)

WalterGR 5y ago
Is the Hacker News API not useful for your project?
- MichaelMoser123 5y ago
 didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?print=pretty https://hacker-news.firebaseio.com/v0/item/29103056.json?pri... On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.
- MichaelMoser123 5y ago
 for this task I need to rescan the pages often, in order to check if the post got flagged. There is no way to know, if or when they rescan the submissions with the HN api.
 - WalterGR 5y ago
 Thanks for your responses. So you have to check the HTML (substring search) to see if a story is flagged? Edit: Answered my own question: yes: https://github.com/MoserMichael/flagged-hn/blob/master/crawl.py#L257 https://github.com/MoserMichael/flagged-hn/blob/master/crawl...
 - MichaelMoser123 5y ago
 Thanks for your feedback, it is very much appreciated! yes, it's part of the table row that includes the title, but if you take the text within the link tag only, then you don't get that indication - I think that's what the HN api might be doing. Example: <td class="title"> [flagged] <a href="https://www.dailywire.com/news/mcauliffe-education-plan-could-lead-to-1970s-style-busing" class="titlelink">McAuliffe Plan: Giving Parents a ‘Choice’ Leads to ‘Segregation’</a> (<a href="from?site=dailywire.com">dailywire.com</a>)</td> Actually if you ask me, I don't know why they are using this 'flagging' thing over here, they might have just moved a submission some pages away from the front pages, if they don't like it.
 - mattowen_uk 5y ago
 Users with enough Karma can 'vouch' for a flagged entry, and effectively de-flag it if they think the flagging is erroneous. Flagging isn't a one-way street.

MichaelMoser123 5y ago

Wow! I did cause them to change: now they flag stories, but they also delete the title from the story, like here: https://news.ycombinator.com/item?id=29134228 https://news.ycombinator.com/item?id=29134228 . Now you only see [flagged] in the title. That makes my little exercise here much less useful. That's also a lesson: crawlers can be made ineffective by slight changes in the input. At least i had some fun at making this little exercise... On the other side: it makes this site a bit less interesting. I am sorry for having caused this, i had the best of intentions. Now i need to find another toy to tinker with. (My wife says that all of my projects are essentially useless, as they don't yield much in terms of money. Well, she has a point... :-( ) Here is a list of my projects: https://github.com/MoserMichael https://github.com/MoserMichael

MichaelMoser123 5y ago
Nah, they had that already. Just got it after scanning all of the entry id's by range.