4 ms·
The article mentions Python, Requests and BeautifulSoup. Here's a short example that scrapes HN Favorites. [0] #!/usr/bin/env python3 import requests
by gabrielsroka 6y ago
The article mentions Python, Requests and BeautifulSoup.
Here's a short example that scrapes HN Favorites. [0]
#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup
username = input('username: ')
# session uses connection pooling, often resulting in faster execution.
session = requests.Session()
base = 'https://news.ycombinator.com/'
path = f'favorites?id={username}'
while path:
r = session.get(base + path)
s = BeautifulSoup(r.text, 'html.parser')
for a in s.select('a.storylink'):
print(a.text, a['href'])
more = s.select_one('a.morelink')
path = more['href'] if more else None
[0] Python: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.py https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...
[1] JavaScript, runs in browser with a UI: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/master/getHNFavorites.js https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...
- lixtra 6y agoUse this an example, but for real stuff prefer using the official API https://github.com/HackerNews/API https://github.com/HackerNews/API
- Dangeranger 6y agoTo be fair, the code above is for scraping favorites which I do not believe is included in the official API. There was a very popular story[0] a few days ago that used scraped data to compile a list of the most favorite posts, which many users didn't even know was a feature until reading the story. Dang even joined in. [0] https://news.ycombinator.com/item?id=24351073 https://news.ycombinator.com/item?id=24351073
- gabrielsroka 6y agoCorrect, the official API doesn't support favorites.
- pj_mukh 6y agoPeople love BeautifulSoup, but I've had better luck with fully emulating a browser with Selenium. While more complicated it seems to run into less issues. Is there a rule of thumb as to when to use one over the other?
- 131012 6y agoWhen a website uses javascript to dynamically render information, Selenium (or any other way to use a headless browser) is usually required.
- d0mine 6y agoThey are orthogonal: 1. you could use selenium to get javascript-generated html 2. then use beautifulsoup to extract the desired info from the html.
- gabrielsroka 6y agoI looked into using Selenium, but it seems like you have to download an 80 MB executable. If you noticed above, I also posted JavaScript that you could run in the browser that you already have. It seemed like a better way to go. I admit Selenium does have some useful features, too.
- derivagral 6y agoNowadays you could just use headless Chrome[1] or Firefox[2], though it looks like Firefox is lagging behind a bit and still wants to require Selenium. Personally, I loved using BS for hobbies until the SPA era started, and then I had to either use headless (selenium back then was great) and/or monitor the network for their API. [1] https://developers.google.com/web/updates/2017/04/headless-chrome https://developers.google.com/web/updates/2017/04/headless-c... [2] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Hea...
- pkukkapalli 6y agoActually surprised they didn't use something more robust like Scrapy.