3 ms·
How do you scrape the data in real time? Do government agencies have API or do you guys use something like scrapy to scrape their websites?
by cs25 10y ago
How do you scrape the data in real time? Do government agencies have API or do you guys use something like scrapy to scrape their websites?
- pudo 10y agoThe latter. I'm not part of the OpenOil team but with the install I work on (https://data.occrp.org https://data.occrp.org) we run crawlers every day, week or month depending on how often the source changes.