3 ms·

> In most cases requests and lxml should be enough to crawl the web. Don't mind my `curl | pup xmlstarlet grep(!!)`s... Nothing to see here...

• amozoss 5y ago
  My brother-in-law had just finished his pilot training and was trying to apply for a job as teacher to continue his training. However, the jobs were first come, first serve so he was waking up at 4 am and constantly refreshing for hours trying to be the first one. When I heard about it, I quickly whipped up a `curl | grep && send_notif` (used pushback.io for notifs) and it helped him not have to worry so much. When a new job posting finally came along he was the first in line and got the job :)
  • getcrunk 5y ago
    Was rate limiting not a concern? How often did you run it? I'm sure you had to keep some state too?
    • amozoss 5y ago
      Their website was pretty trivial, but I beefed up the script to check for failure status code and also notify me if it wasn't 200. I ran it every minute. I had to include a unique uuid to cache bust, but other than that there wasn't any state. I wrote about it here https://plainice.com/webscrape-notifications https://plainice.com/webscrape-notifications