4 ms·

Could someone convert it to Python-script?

gabrielsroka 7y ago
Part of the advantage of running JavaScript in your browser is that you might already be authenticated and it can use your session. But, fetching your HN favorites doesn't require authentication. #!/usr/bin/env python3 import requests from bs4 import BeautifulSoup for p in range(1, 17): r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}') s = BeautifulSoup(r.text, 'html.parser') print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')])
- death-by-ppt 7y ago
  Can someone convert it to Bel?
- app4soft 7y ago
  Thanks! One more question: what is the best way stop it when it will reaches last page? > for p in range(1, 17): Actually p=17[0] is empty (as p=16 is maximum as for now). Maybe, script should scrap pages from `1` to `infinity` UNTIL it detect next message on page[0]: > app4soft hasn't added any favorite submissions yet. [0] https://news.ycombinator.com/favorites?id=app4soft&p=17 https://news.ycombinator.com/favorites?id=app4soft&p=17
  - gabrielsroka 7y ago
    In Python, `range(1, 17)` produces the numbers 1-16. I hard-coded it just for your favorites. A better way to solve it is to look at the `len()` of the results, and stop when it gets to 0: p = 1 while True: r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}') s = BeautifulSoup(r.text, 'html.parser') faves = [{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')] print(faves) p += 1 if len(faves) == 0: break
    - app4soft 7y ago
      Great!
      - gabrielsroka 7y ago
        I think this one is a little cleaner. I used some of the ideas in sbr464's code. path = 'favorites?id=app4soft' while path: r = requests.get('https://news.ycombinator.com/' + path) s = BeautifulSoup(r.text, 'html.parser') print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')]) more = s.select_one('a.morelink') path = more['href'] if more else None
        app4soft 7y ago
        Go on! ;)
amjd 7y ago
I had written a Python script to get saved (upvoted) links as JSON / CSV a few years ago: https://github.com/amjd/HN-Saved-Links-Export https://github.com/amjd/HN-Saved-Links-Export I'm not sure if it still works as it too relied on HTML scraping. Perhaps I should update it to support favorites too. Edit: Whoa, it's been 4 years already. I believe HN didn't have favorite feature at the time. That's why I used upvoting as my bookmarks system and created a script to export that data.
- gabrielsroka 7y ago
  @amjd, thanks for sharing. I upgraded it from Python 2 to Python 3, but when I ran it, I got a 404 error on the `saved` endpoint. Does it work for you? Edit: I see from the other PR it's called `upvoted`. Edit 2: I changed it to `upvoted` and now I get a 200 OK, but the code crashed right afterwards on `tree.cssselect()`.
  - amjd 7y ago
    It's four years old so the page HTML structure must have likely changed. I'll take a look at it when I'm free and update the script. Thanks for your PR, I'll review and merge the two open PRs as well. Btw, good work on your JS solution. It's great that it just works without requiring any download or installation. :)

Part of the advantage of running JavaScript in your browser is that you might already be authenticated and it can use your session. But, fetching your HN favorites doesn't require authentication. #!/usr/bin/env python3 import requests from bs4 import BeautifulSoup for p in range(1, 17): r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}') s = BeautifulSoup(r.text, 'html.parser') print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')])

death-by-ppt 7y ago
Can someone convert it to Bel?
app4soft 7y ago
Thanks! One more question: what is the best way stop it when it will reaches last page? > for p in range(1, 17): Actually p=17[0] is empty (as p=16 is maximum as for now). Maybe, script should scrap pages from `1` to `infinity` UNTIL it detect next message on page[0]: > app4soft hasn't added any favorite submissions yet. [0] https://news.ycombinator.com/favorites?id=app4soft&p=17 https://news.ycombinator.com/favorites?id=app4soft&p=17
- gabrielsroka 7y ago
  In Python, `range(1, 17)` produces the numbers 1-16. I hard-coded it just for your favorites. A better way to solve it is to look at the `len()` of the results, and stop when it gets to 0: p = 1 while True: r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}') s = BeautifulSoup(r.text, 'html.parser') faves = [{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')] print(faves) p += 1 if len(faves) == 0: break
  - app4soft 7y ago
    Great!
    - gabrielsroka 7y ago
      I think this one is a little cleaner. I used some of the ideas in sbr464's code. path = 'favorites?id=app4soft' while path: r = requests.get('https://news.ycombinator.com/' + path) s = BeautifulSoup(r.text, 'html.parser') print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')]) more = s.select_one('a.morelink') path = more['href'] if more else None
      - app4soft 7y ago
        Go on! ;)

amjd 7y ago

I had written a Python script to get saved (upvoted) links as JSON / CSV a few years ago: https://github.com/amjd/HN-Saved-Links-Export https://github.com/amjd/HN-Saved-Links-Export I'm not sure if it still works as it too relied on HTML scraping. Perhaps I should update it to support favorites too. Edit: Whoa, it's been 4 years already. I believe HN didn't have favorite feature at the time. That's why I used upvoting as my bookmarks system and created a script to export that data.

gabrielsroka 7y ago
@amjd, thanks for sharing. I upgraded it from Python 2 to Python 3, but when I ran it, I got a 404 error on the `saved` endpoint. Does it work for you? Edit: I see from the other PR it's called `upvoted`. Edit 2: I changed it to `upvoted` and now I get a 200 OK, but the code crashed right afterwards on `tree.cssselect()`.
- amjd 7y ago
  It's four years old so the page HTML structure must have likely changed. I'll take a look at it when I'm free and update the script. Thanks for your PR, I'll review and merge the two open PRs as well. Btw, good work on your JS solution. It's great that it just works without requiring any download or installation. :)