7 ms·

Show HN: Hacker News bot that automated archive finding and commenting

52 points by ss_ 4y ago

Hi everyone, I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription blocks The idea was to have this run on a server and then I wouldn’t have this daily annoyance, the bot was banned, so its not currently active, but I still wanted to show it off and get some feedback Disclaimer : HN doesn't allow bots, so please don’t use this to run your own HN bot.

cloudking 4y ago
Cool idea. It's kind of silly that "Hacker" News doesn't allow hacking the comments section with useful bots. Why not let the community decide through comment voting?
- krapp 4y ago
- matttb 4y ago
  I'm very happy they don't allow them. It would probably be flooded with annoying and useless bots like reddit. Sure you can downvote them, but it's still extra effort * a lot of people's attention
moonshotideas 4y ago
Love the idea. Too bad it’s not allowed, it would save me some annoyance.
unknownaccount 4y ago
I wish there was a bot that automatically archived [flagged] posts. the fact users here can use botnets to flag/remove any submission or post they dislike makes Hacker News positively useless for whistleblowing. I learned from experience anything even tangentially controversial is almost guaranteed to be false-flagged and taken down.
- evandwight 4y ago
  Can't you get flagged/dead posts on the new listing or firebase if you really wanted?
  - krapp 4y ago
    You can get metadata in order to preserve the comment tree, but it won't show the text for quality control reasons.
    - pbhjpbhj 4y ago
      You can see dead/flagged comments via a setting, perhaps only exposed to users if their own score is high enough(?).
      - metadat 4y ago
        I've thought about making a site that publicly indexes and exposes all flagged and dead HN comments and stories. Occasionally there are gems. hnflagflyer.com
    - hoppyhoppy2 4y ago
      You should be able to turn on the "showdead" option in your user settings to see the text. But (like the other user) I'm not sure if there's a minimum karma amount for that toggle to show up.
      - krapp 4y ago
        I was referring to the API.
- pbhjpbhj 4y ago
  I've often thought I should make a reddit bot that posts all the r/askhistorians comments to r/dontaskhistorians so I could see all the comments that are removed. I've chanced a few comments myself and the mods would rather have no information than some information (no matter how well referenced) that isn't entirely academically rigorous; their choice, it's not awful. I think that a lot of proverbial babies get thrown out with the bath water and a lot of interesting comments probably get removed. Someone said "nothing was known about X until $date" or words to that effect and I replied with two sources, available online, that spoke about X prior to the date and that it was from my own recollection that I knew the sources existed. The incorrect comment stands and I got the usual "your comment was not complete enough for us". Every now and again I'll post something, especially when there are no responses and I have a source to share -- but I've taken to messaging the OP now instead. Maybe it's all chaff, but I'm curious to find out. I like the Slashdot way of having scoring and a slider so you can see it all if you want, and the "boosts" they do so you can see humorous stuff even if others want to see serious stuff. User-level filtering makes much more sense to me.
arjvik 4y ago
@dang, would you consider whitelisting this bot on HN? I think the community would agree that this provides meaningful value to us.
- pvg 4y ago
  https://news.ycombinator.com/item?id=32571890 https://news.ycombinator.com/item?id=32571890 Also, if you want to tell mods something, email it because @dang doesn't automatically summon a dang.
- tptacek 4y ago
  The community does not agree about this.
matham 4y ago
I misread it as "Hacker News bot that automated archive finding and tormenting"...
metadat 4y ago
It's a nice idea, but I like how HN is devoid of blatantly automated posts. Reddit is full of them and it's annoying. Dang has actually written a response to OP a few days ago, saying essentially that he understands the intention is good, and that this sort of automation is at odds with the core philosophy of HN existing to promote curiosity and interesting conversation between human beings. When a post is popular enough, the community tends to organically post the un-paywall links, making the bot unnecessary. Imagine having machine-automated archive link posts on every NYT and WaPo submission, it would be annoying and the early comment count would become meaningless noise, not even indicating that some person found it interesting enough to dig up the archive links for.
- hamandcheese 4y ago
  > and the early comment count would become meaningless noise, not even indicating that some person found it interesting enough to dig up the archive links for. I was on the fence, but this is the compelling argument against, IMO.
- ss_ 4y ago
  > but I like how HN is devoid of blatantly automated posts. Reddit is full of them and it's annoying. This argument I understand, because as @dang said - while this bot would enhance the HN experience, once the Floodgates are allowed to open, it will have unintended consequences and lead to a worse HN experience > this sort of automation is at odds with the core philosophy of HN existing to promote curiosity and interesting conversation between human beings. I don't agree with that statement. Enabling easier access to the articles that are being discussed, would allow more people to participate and enhance the conversation > it would be annoying and the early comment count would become meaningless noise Maybe, but (a) Whats the impact of a single comment? (b) you could always prevent the bot from commenting the archive link on posts with less than N points or N comments. That being said, I think the first point @dang made is a strong argument against the bot
  - pvg 4y ago
    would allow more people to participate and enhance the conversation Whenever there's an access issue, someone posts a workaround. The argument for making an exception for some bot has to be stronger than 'takes something that isn't broken and fixes it so it's clunkier and noisier'. The entire premise that it's 'enabling easier access' just not well-supported by what one can easily see by reading HN threads.
forgotmypw17 4y ago
Thank you very much for sharing this. I was hoping to see the code as soon as you posted it for the first time.
- ss_ 4y ago
  :) your welcome - I hope you enjoy!
1vuio0pswjnm7 4y ago
"I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription blocks" It is easy to automate this using a forward proxy. I use one on a loopback address. The way I have it set up, I add a given domain to a text file, reload,^1 and from then on, the proxy will rewrite the URLs using that domain to point to archive.org or whatever. I have never encountered an archive.ph URL on HN where the site proxied, purportedly to avoid a "subscription block",^2 was not accessible without using the archive.ph URL. For example, sometimes the issue is simply that Javascript needs to be disabled, or one needs to use an AMP URL without Javascript. (The later looks great in a text-only browsersuch as links.) Whereas submissions for articles on sites that are password-protected, e.g., ft.com, rarely or never have archive.ph URLs. This notion of "bots" is a never-ending source of amusement. Everyone thinks they know what a "bot" is, but IMO that term alone tells us next to nothing. For example, one can write scripts, here using Python, that are disrespectful of server resources or whatever "rules" are in place and, as a result, get "banned". One can also write scripts that are respectful of server resources and any rules and never get banned. Automation is not per se "bots" and "bots" are not per se "bad". Computers and automation are everywhere. We do not label all automation using the internet as "bots". Whether something is "good" or "bad" depends on what a script is doing and, sometimes, who is using it.^3 I have been using automation for reading HN for over 10 years. Yes, it is possible to get banned but it depends on what a script does. The HN software does have limitations and one has to learn to work within those constraints. 1. Something like echo reload |socat stdio /path/to/control/socket 2. IMO a true "subscription" would require a password. That is, non-subscribers would not have access to the content. Why do sites make content avaiable to to non-subscribers. Is it only to try to attract new subscribers. The site is gathering data about computer users and serving advertising. The site operator profits from the traffic from non-subscribers. If a site operator wants to make content only accesible to sucbscribers it can make that content password-protected. If the operator chooses not to use password protection then she may have a difficult time convincing a court the content published was not public. Annoying people with Javascript is not access control. 3. Corporations performing telemetry and remotely installing and running "updates", automatically, without explicit consent on computers that do not belong to them generally do not get labeled as "bots". Whereas using automation to "subscribe" to "tech" company websites that aim to collect data about users in order to target advertising might be labeled as "bots" in a perjorative sense, if one is in the business of online advertising.
octoberfranklin 4y ago
How are you able to automate anything involving archive.ph? Everything on that site that triggers a page-scrape is behind Google's ReCAPTCHA. And they've got the threshhold set at "hyper-aggressive" -- way above any other non-email-sending site I know of. It's the one site that I still get hit with a captcha from every time (excluding direct links to already-archived pages -- those don't seem to trigger the captcha).
bArray 4y ago
Perhaps this would be better served as a plugin? For the wayback machine plugin I can right click and open that link in the archive, but this is of course for only one website.

52 points by ss_ 4y ago

cloudking 4y ago

Cool idea. It's kind of silly that "Hacker" News doesn't allow hacking the comments section with useful bots. Why not let the community decide through comment voting?

krapp 4y ago
matttb 4y ago
I'm very happy they don't allow them. It would probably be flooded with annoying and useless bots like reddit. Sure you can downvote them, but it's still extra effort * a lot of people's attention

moonshotideas 4y ago

Love the idea. Too bad it’s not allowed, it would save me some annoyance.

unknownaccount 4y ago

I wish there was a bot that automatically archived [flagged] posts. the fact users here can use botnets to flag/remove any submission or post they dislike makes Hacker News positively useless for whistleblowing. I learned from experience anything even tangentially controversial is almost guaranteed to be false-flagged and taken down.

evandwight 4y ago
Can't you get flagged/dead posts on the new listing or firebase if you really wanted?
- krapp 4y ago
  You can get metadata in order to preserve the comment tree, but it won't show the text for quality control reasons.
  - pbhjpbhj 4y ago
    You can see dead/flagged comments via a setting, perhaps only exposed to users if their own score is high enough(?).
    - metadat 4y ago
      I've thought about making a site that publicly indexes and exposes all flagged and dead HN comments and stories. Occasionally there are gems. hnflagflyer.com
  - hoppyhoppy2 4y ago
    You should be able to turn on the "showdead" option in your user settings to see the text. But (like the other user) I'm not sure if there's a minimum karma amount for that toggle to show up.
    - krapp 4y ago
      I was referring to the API.
pbhjpbhj 4y ago
I've often thought I should make a reddit bot that posts all the r/askhistorians comments to r/dontaskhistorians so I could see all the comments that are removed. I've chanced a few comments myself and the mods would rather have no information than some information (no matter how well referenced) that isn't entirely academically rigorous; their choice, it's not awful. I think that a lot of proverbial babies get thrown out with the bath water and a lot of interesting comments probably get removed. Someone said "nothing was known about X until $date" or words to that effect and I replied with two sources, available online, that spoke about X prior to the date and that it was from my own recollection that I knew the sources existed. The incorrect comment stands and I got the usual "your comment was not complete enough for us". Every now and again I'll post something, especially when there are no responses and I have a source to share -- but I've taken to messaging the OP now instead. Maybe it's all chaff, but I'm curious to find out. I like the Slashdot way of having scoring and a slider so you can see it all if you want, and the "boosts" they do so you can see humorous stuff even if others want to see serious stuff. User-level filtering makes much more sense to me.

arjvik 4y ago

@dang, would you consider whitelisting this bot on HN? I think the community would agree that this provides meaningful value to us.

pvg 4y ago
https://news.ycombinator.com/item?id=32571890 https://news.ycombinator.com/item?id=32571890 Also, if you want to tell mods something, email it because @dang doesn't automatically summon a dang.
tptacek 4y ago
The community does not agree about this.

matham 4y ago

I misread it as "Hacker News bot that automated archive finding and tormenting"...

metadat 4y ago

It's a nice idea, but I like how HN is devoid of blatantly automated posts. Reddit is full of them and it's annoying. Dang has actually written a response to OP a few days ago, saying essentially that he understands the intention is good, and that this sort of automation is at odds with the core philosophy of HN existing to promote curiosity and interesting conversation between human beings. When a post is popular enough, the community tends to organically post the un-paywall links, making the bot unnecessary. Imagine having machine-automated archive link posts on every NYT and WaPo submission, it would be annoying and the early comment count would become meaningless noise, not even indicating that some person found it interesting enough to dig up the archive links for.

hamandcheese 4y ago
> and the early comment count would become meaningless noise, not even indicating that some person found it interesting enough to dig up the archive links for. I was on the fence, but this is the compelling argument against, IMO.
ss_ 4y ago
> but I like how HN is devoid of blatantly automated posts. Reddit is full of them and it's annoying. This argument I understand, because as @dang said - while this bot would enhance the HN experience, once the Floodgates are allowed to open, it will have unintended consequences and lead to a worse HN experience > this sort of automation is at odds with the core philosophy of HN existing to promote curiosity and interesting conversation between human beings. I don't agree with that statement. Enabling easier access to the articles that are being discussed, would allow more people to participate and enhance the conversation > it would be annoying and the early comment count would become meaningless noise Maybe, but (a) Whats the impact of a single comment? (b) you could always prevent the bot from commenting the archive link on posts with less than N points or N comments. That being said, I think the first point @dang made is a strong argument against the bot
- pvg 4y ago
  would allow more people to participate and enhance the conversation Whenever there's an access issue, someone posts a workaround. The argument for making an exception for some bot has to be stronger than 'takes something that isn't broken and fixes it so it's clunkier and noisier'. The entire premise that it's 'enabling easier access' just not well-supported by what one can easily see by reading HN threads.

forgotmypw17 4y ago

Thank you very much for sharing this. I was hoping to see the code as soon as you posted it for the first time.

ss_ 4y ago
:) your welcome - I hope you enjoy!

1vuio0pswjnm7 4y ago

"I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription blocks" It is easy to automate this using a forward proxy. I use one on a loopback address. The way I have it set up, I add a given domain to a text file, reload,^1 and from then on, the proxy will rewrite the URLs using that domain to point to archive.org or whatever. I have never encountered an archive.ph URL on HN where the site proxied, purportedly to avoid a "subscription block",^2 was not accessible without using the archive.ph URL. For example, sometimes the issue is simply that Javascript needs to be disabled, or one needs to use an AMP URL without Javascript. (The later looks great in a text-only browsersuch as links.) Whereas submissions for articles on sites that are password-protected, e.g., ft.com, rarely or never have archive.ph URLs. This notion of "bots" is a never-ending source of amusement. Everyone thinks they know what a "bot" is, but IMO that term alone tells us next to nothing. For example, one can write scripts, here using Python, that are disrespectful of server resources or whatever "rules" are in place and, as a result, get "banned". One can also write scripts that are respectful of server resources and any rules and never get banned. Automation is not per se "bots" and "bots" are not per se "bad". Computers and automation are everywhere. We do not label all automation using the internet as "bots". Whether something is "good" or "bad" depends on what a script is doing and, sometimes, who is using it.^3 I have been using automation for reading HN for over 10 years. Yes, it is possible to get banned but it depends on what a script does. The HN software does have limitations and one has to learn to work within those constraints. 1. Something like echo reload |socat stdio /path/to/control/socket 2. IMO a true "subscription" would require a password. That is, non-subscribers would not have access to the content. Why do sites make content avaiable to to non-subscribers. Is it only to try to attract new subscribers. The site is gathering data about computer users and serving advertising. The site operator profits from the traffic from non-subscribers. If a site operator wants to make content only accesible to sucbscribers it can make that content password-protected. If the operator chooses not to use password protection then she may have a difficult time convincing a court the content published was not public. Annoying people with Javascript is not access control. 3. Corporations performing telemetry and remotely installing and running "updates", automatically, without explicit consent on computers that do not belong to them generally do not get labeled as "bots". Whereas using automation to "subscribe" to "tech" company websites that aim to collect data about users in order to target advertising might be labeled as "bots" in a perjorative sense, if one is in the business of online advertising.

octoberfranklin 4y ago

How are you able to automate anything involving archive.ph? Everything on that site that triggers a page-scrape is behind Google's ReCAPTCHA. And they've got the threshhold set at "hyper-aggressive" -- way above any other non-email-sending site I know of. It's the one site that I still get hit with a captcha from every time (excluding direct links to already-archived pages -- those don't seem to trigger the captcha).

bArray 4y ago

Perhaps this would be better served as a plugin? For the wayback machine plugin I can right click and open that link in the archive, but this is of course for only one website.