5 ms·
These days I just let chagpt generate a script that scrapes a site and spits out an rss file. Then I run it with cron.
by dagurp 3y ago
These days I just let chagpt generate a script that scrapes a site and spits out an rss file. Then I run it with cron.
- dopidopHN 3y agoAre you not limited by the cut off date of the content the model is trained off ?
- pinkcan 3y ago1. the script is generated by the llm 2. the user runs the script that does the scraping these are temporally separate actions
- dopidopHN 3y agoFine, but it’s subject to html selectors brittleness no? Oh, you subject the raw html when you need it maybe?
- dagurp 3y agoHere's how I do it. 1. Tell chatGPT to create a python script that scrapes example.com and generate an rss file. 2. Paste a snippet of the html and tell it to modify the script to use that. 3. I do some minor tweaks myself to fix the date format.
- notadev 3y agoI’m guessing they paste a portion of the website’s source then tell ChatGPT to generate a script that can generate an RSS feed from that site.
- dagurp 3y agoYeah I just copy the html that's relevant. There's some manual work involved but it doesn't take a lot of time.