2 ms·
Against unnecessary databases: Parse, don't normalize
- vivegi 4y agoThis makes a lot of sense. In the past when I built scrapers/aggregators, it made sense to separate out the scrape part and the aggregate part. 1. It decoupled the steps and allowed for concurrency of the scraping and aggregation tasks. 2. It helped with debugging the tiny inconsistencies across sites and iterating on the aggregation. 3. As a side-benefit, if I needed to aggregate more data elements from the same sources, I could just re-run the aggregate step.