4 ms·

I've used Scrapy extensively for writing crawlers. There's a lot of good things like not having to worry about storage backends, request throttling (random sec

I've used Scrapy extensively for writing crawlers. There's a lot of good things like not having to worry about storage backends, request throttling (random seconds between requests), the ability to run parallel parsers easily. There is also a lot of open source middleware to help with things like retrying requests with proxies and rotating user agents. However, like any battery included framework it has downsides in terms of flexibility. In most cases requests and lxml should be enough to crawl the web.

ducktective 5y ago
> In most cases requests and lxml should be enough to crawl the web. Don't mind my `curl | pup xmlstarlet grep(!!)`s... Nothing to see here...
- amozoss 5y ago
  My brother-in-law had just finished his pilot training and was trying to apply for a job as teacher to continue his training. However, the jobs were first come, first serve so he was waking up at 4 am and constantly refreshing for hours trying to be the first one. When I heard about it, I quickly whipped up a `curl | grep && send_notif` (used pushback.io for notifs) and it helped him not have to worry so much. When a new job posting finally came along he was the first in line and got the job :)
  - getcrunk 5y ago
    Was rate limiting not a concern? How often did you run it? I'm sure you had to keep some state too?
    - amozoss 5y ago
      Their website was pretty trivial, but I beefed up the script to check for failure status code and also notify me if it wasn't 200. I ran it every minute. I had to include a unique uuid to cache bust, but other than that there wasn't any state. I wrote about it here https://plainice.com/webscrape-notifications https://plainice.com/webscrape-notifications
aynyc 5y ago
If you are just doing one or two pages, say you want to get weather for your location, then requests is sufficient. But if you want to do many pages where you might want to scan and follow, requests gets tedious very quickly.
- r_singh 5y ago
  If you’re a web developer not really, rather than worrying about storage backendes, spiders, yielding and managing loops and items, you could just host a DRF or Flask API with your scrapers (written in Requests+lxml) initiated with an API request. I guess it’s a matter of preference

ducktective 5y ago

> In most cases requests and lxml should be enough to crawl the web. Don't mind my `curl | pup xmlstarlet grep(!!)`s... Nothing to see here...

amozoss 5y ago
My brother-in-law had just finished his pilot training and was trying to apply for a job as teacher to continue his training. However, the jobs were first come, first serve so he was waking up at 4 am and constantly refreshing for hours trying to be the first one. When I heard about it, I quickly whipped up a `curl | grep && send_notif` (used pushback.io for notifs) and it helped him not have to worry so much. When a new job posting finally came along he was the first in line and got the job :)
- getcrunk 5y ago
  Was rate limiting not a concern? How often did you run it? I'm sure you had to keep some state too?
  - amozoss 5y ago
    Their website was pretty trivial, but I beefed up the script to check for failure status code and also notify me if it wasn't 200. I ran it every minute. I had to include a unique uuid to cache bust, but other than that there wasn't any state. I wrote about it here https://plainice.com/webscrape-notifications https://plainice.com/webscrape-notifications

aynyc 5y ago

If you are just doing one or two pages, say you want to get weather for your location, then requests is sufficient. But if you want to do many pages where you might want to scan and follow, requests gets tedious very quickly.

r_singh 5y ago
If you’re a web developer not really, rather than worrying about storage backendes, spiders, yielding and managing loops and items, you could just host a DRF or Flask API with your scrapers (written in Requests+lxml) initiated with an API request. I guess it’s a matter of preference