3 ms·
Scraping, while legal now, is still not free. A scrape does not give you the delta between two points in time without a completely new scrape. You cannot econom
by godzillabrennus 7y ago
Scraping, while legal now, is still not free. A scrape does not give you the delta between two points in time without a completely new scrape. You cannot economically sustainably scrape the entirety of a large site like Twitter.
If I’m wrong I’d love to learn from someone smarter than me.
Even google doesn’t scrape the entire web every day.
- CaptArmchair 7y agoYou're not wrong. Library of Congress started archiving every public tweet in 2010 but abandoned the effort in 2017 mainly because of the sheer volume. https://www.npr.org/sections/thetwo-way/2017/12/26/573609499/library-of-congress-will-no-longer-archive-every-tweet https://www.npr.org/sections/thetwo-way/2017/12/26/573609499... I think it might be feasible to scrape a subset: a single account or a few accounts at most. But even then, you're still fighting problems ranging from wonky HTML generated through JavaScript towards sudden, unannounced changes to the DOM structure. It's not a managed API after all.