4 ms·
I have some weekly-updated government datasets I have to pull in. I'd love to automate it, but government tends not to run their websites with automation in min
by oxguy3 11y ago
I have some weekly-updated government datasets I have to pull in. I'd love to automate it, but government tends not to run their websites with automation in mind, so the file names tend to be different every week, and the day it updates varies, and it's just a general pain in the butt.
- borplk 11y agoDo you mind posting a link? I'm good with this stuff I could potentially help you out with a script.
- tmaly 11y agoI have a similar task, but I was able to automate it with a perl script. I use Web::Scraper module along with ORLite. I check the site on a frequency for a file or change to text on the page using a simple MD5 hash of the content, or the name of the file in a link. I keep track of it in a simple sqlite database. When there is a change in some SEC spec or a new file on Edgar, I grab it, mark it as processed in sqlite and then mail it to the people that need it.