5 ms·

Show HN: Archivy-HN – automatically save and download your upvoted HN posts

144 points by etherio 6y ago

amelius 6y ago
Is there a way to download all your HN comments?
- etherio 6y ago
  I'm thinking about doing something like that - I'll implement it because it'd also be quite useful. Do you think it would be worthwhile to be able to save upvoted / favorited comments also? I also have to think about what context (ie parent / child comments) it would be useful to gather.
  - amelius 6y ago
    Actually, the thing I would be interested in most is a better way to search my comments, my upvoted comments, etc. HN has a search box, but afaik there's no way to filter by user, or to restrict the search to my upvoted comments, etc. Downloading would be a good second-best option, because at least it would allow me to "grep" through the archives.
    - etherio 6y ago
      Hmm I see that could be interesting. The software this script is built around [0] comes with a developing [1] search functionality. The idea is that using these plugins and the main software one can group / save / search their digital identity - and that's exactly what saving comments would work towards. [0]: https://archivy.github.io https://archivy.github.io [1]: https://github.com/archivy/archivy/pulls/159 https://github.com/archivy/archivy/pulls/159
    - smichel17 6y ago
      You can restrict "by:user".
  - zxcvbn4038 6y ago
    I do not see any rate limiting, this could lock people out if they have more then a couple pages of posts.
    - etherio 6y ago
      The API (https://hn.algolia.com/api https://hn.algolia.com/api) being used limits to 10000 requests per hour so I doubt there would be a problem.
  - dataflow 6y ago
    Upvoted comments is the one thing I've always wanted (I don't care for the rest). I often upvote things that are useful to refer back to, and without this there's no easy way to find them again.
    - colejohnson66 6y ago
      Why not favorite them?
      - mehrdadn 6y ago
        They're public? I don't necessarily feel a desire to make a public announcement about every comment I think might be interesting or worth referring back to.
      - akkartik 6y ago
        Are favorited comments really any more discoverable? (I use them a lot: https://news.ycombinator.com/favorites?id=akkartik&comments=t https://news.ycombinator.com/favorites?id=akkartik&comments=...)
- krapp 6y ago
  The username endpoint in the HN API contains ids for all of that user's comments - you could take that and iterate the ids and download them from the API. It could probably be done with a simple script and you would end up with a folder full of JSON files. Your ISP might hate you, though.
- taphangum 6y ago
  This would be very useful.
- jaredsohn 6y ago
  I built a node library that does this 6 years ago and I just verified that it still works. It uses the algolia API. https://github.com/jaredsohn/hnuserdownload https://github.com/jaredsohn/hnuserdownload There was also a website which is not working. Going to turn that off now.
- reilly3000 6y ago
  All HN comments are available with Google's BigQuery public datasets: https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news?pli=1 https://console.cloud.google.com/marketplace/details/y-combi...
- pabs3 6y ago
  I was going to write a script using haxor to convert my HN submissions and comments to Maildir, didn't get around to it though. https://github.com/avinassh/haxor https://github.com/avinassh/haxor
- jwilk 6y ago
  https://github.com/jwilk/hackerlates https://github.com/jwilk/hackerlates
braydo25 6y ago
oooo meta
longnguyen 6y ago
Interesting. But for me upvoted links are already stored on Hacker News. What I wanted to download / archive is others' comments. Some comments are so valuable that I saved in a personal note (HN Wisdoms)
ashish01 6y ago
Cool. If you want an smaller alternative, just getting upvoted ids is easy enough in python import requests from bs4 import BeautifulSoup with requests.Session() as session: page = 1 while True: resp = session.get( f"https://news.ycombinator.com/upvoted?id=ashish01&p={page}", cookies={"user": "get this from your browser"}, ) tree = BeautifulSoup(resp.text) links = [x["href"] for x in tree.select(".subtext .age a")] if len(links) == 0: break for i, link in enumerate(links): print(page, i, link.split("=")[1]) page = page + 1
- pabs3 6y ago
  You could also use the haxor Python library that wraps the HN API: https://github.com/avinassh/haxor https://github.com/avinassh/haxor

144 points by etherio 6y ago

amelius 6y ago

Is there a way to download all your HN comments?

etherio 6y ago
I'm thinking about doing something like that - I'll implement it because it'd also be quite useful. Do you think it would be worthwhile to be able to save upvoted / favorited comments also? I also have to think about what context (ie parent / child comments) it would be useful to gather.
- amelius 6y ago
  Actually, the thing I would be interested in most is a better way to search my comments, my upvoted comments, etc. HN has a search box, but afaik there's no way to filter by user, or to restrict the search to my upvoted comments, etc. Downloading would be a good second-best option, because at least it would allow me to "grep" through the archives.
  - etherio 6y ago
    Hmm I see that could be interesting. The software this script is built around [0] comes with a developing [1] search functionality. The idea is that using these plugins and the main software one can group / save / search their digital identity - and that's exactly what saving comments would work towards. [0]: https://archivy.github.io https://archivy.github.io [1]: https://github.com/archivy/archivy/pulls/159 https://github.com/archivy/archivy/pulls/159
  - smichel17 6y ago
    You can restrict "by:user".
- zxcvbn4038 6y ago
  I do not see any rate limiting, this could lock people out if they have more then a couple pages of posts.
  - etherio 6y ago
    The API (https://hn.algolia.com/api https://hn.algolia.com/api) being used limits to 10000 requests per hour so I doubt there would be a problem.
- dataflow 6y ago
  Upvoted comments is the one thing I've always wanted (I don't care for the rest). I often upvote things that are useful to refer back to, and without this there's no easy way to find them again.
  - colejohnson66 6y ago
    Why not favorite them?
    - mehrdadn 6y ago
      They're public? I don't necessarily feel a desire to make a public announcement about every comment I think might be interesting or worth referring back to.
    - akkartik 6y ago
      Are favorited comments really any more discoverable? (I use them a lot: https://news.ycombinator.com/favorites?id=akkartik&comments=t https://news.ycombinator.com/favorites?id=akkartik&comments=...)
krapp 6y ago
The username endpoint in the HN API contains ids for all of that user's comments - you could take that and iterate the ids and download them from the API. It could probably be done with a simple script and you would end up with a folder full of JSON files. Your ISP might hate you, though.
taphangum 6y ago
This would be very useful.
jaredsohn 6y ago
I built a node library that does this 6 years ago and I just verified that it still works. It uses the algolia API. https://github.com/jaredsohn/hnuserdownload https://github.com/jaredsohn/hnuserdownload There was also a website which is not working. Going to turn that off now.
reilly3000 6y ago
All HN comments are available with Google's BigQuery public datasets: https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news?pli=1 https://console.cloud.google.com/marketplace/details/y-combi...
pabs3 6y ago
I was going to write a script using haxor to convert my HN submissions and comments to Maildir, didn't get around to it though. https://github.com/avinassh/haxor https://github.com/avinassh/haxor
jwilk 6y ago
https://github.com/jwilk/hackerlates https://github.com/jwilk/hackerlates

braydo25 6y ago

oooo meta

longnguyen 6y ago

Interesting. But for me upvoted links are already stored on Hacker News. What I wanted to download / archive is others' comments. Some comments are so valuable that I saved in a personal note (HN Wisdoms)

ashish01 6y ago

Cool. If you want an smaller alternative, just getting upvoted ids is easy enough in python import requests from bs4 import BeautifulSoup with requests.Session() as session: page = 1 while True: resp = session.get( f"https://news.ycombinator.com/upvoted?id=ashish01&p={page}", cookies={"user": "get this from your browser"}, ) tree = BeautifulSoup(resp.text) links = [x["href"] for x in tree.select(".subtext .age a")] if len(links) == 0: break for i, link in enumerate(links): print(page, i, link.split("=")[1]) page = page + 1

pabs3 6y ago
You could also use the haxor Python library that wraps the HN API: https://github.com/avinassh/haxor https://github.com/avinassh/haxor