4 ms·
Part of the advantage of running JavaScript in your browser is that you might already be authenticated and it can use your session. But, fetching your HN favori
by gabrielsroka 7y ago
Part of the advantage of running JavaScript in your browser is that you might already be authenticated and it can use your session. But, fetching your HN favorites doesn't require authentication.
#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup
for p in range(1, 17):
r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}')
s = BeautifulSoup(r.text, 'html.parser')
print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')])
- death-by-ppt 7y agoCan someone convert it to Bel?
- app4soft 7y agoThanks! One more question: what is the best way stop it when it will reaches last page? > for p in range(1, 17): Actually p=17[0] is empty (as p=16 is maximum as for now). Maybe, script should scrap pages from `1` to `infinity` UNTIL it detect next message on page[0]: > app4soft hasn't added any favorite submissions yet. [0] https://news.ycombinator.com/favorites?id=app4soft&p=17 https://news.ycombinator.com/favorites?id=app4soft&p=17
- gabrielsroka 7y agoIn Python, `range(1, 17)` produces the numbers 1-16. I hard-coded it just for your favorites. A better way to solve it is to look at the `len()` of the results, and stop when it gets to 0: p = 1 while True: r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}') s = BeautifulSoup(r.text, 'html.parser') faves = [{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')] print(faves) p += 1 if len(faves) == 0: break
- app4soft 7y agoGreat!
- gabrielsroka 7y agoI think this one is a little cleaner. I used some of the ideas in sbr464's code. path = 'favorites?id=app4soft' while path: r = requests.get('https://news.ycombinator.com/' + path) s = BeautifulSoup(r.text, 'html.parser') print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')]) more = s.select_one('a.morelink') path = more['href'] if more else None
- app4soft 7y agoGo on! ;)