5 ms·
Now that scraping has been deemed legal by the highest curts, how can anybody still get burned by API fees?
by TekMol 7y ago
Now that scraping has been deemed legal by the highest curts, how can anybody still get burned by API fees?
- ceejayoz 7y ago"Legal" and "won't be interfered with" are not the same thing. That it's legal to scrape doesn't mean they can't put captchas, IP blocks, etc. in your way. Scraping doesn't really help when publishing tweets, either.
- colejohnson66 7y agoI thought the point of the LinkedIn lawsuit was that they were trying to stop them from scraping?
- ceejayoz 7y agoLinkedIn was trying to do so via legal measures (specifically, alleging violations of the Computer Fraud and Abuse Act in a C&D letter).
- godzillabrennus 7y agoScraping, while legal now, is still not free. A scrape does not give you the delta between two points in time without a completely new scrape. You cannot economically sustainably scrape the entirety of a large site like Twitter. If I’m wrong I’d love to learn from someone smarter than me. Even google doesn’t scrape the entire web every day.
- CaptArmchair 7y agoYou're not wrong. Library of Congress started archiving every public tweet in 2010 but abandoned the effort in 2017 mainly because of the sheer volume. https://www.npr.org/sections/thetwo-way/2017/12/26/573609499/library-of-congress-will-no-longer-archive-every-tweet https://www.npr.org/sections/thetwo-way/2017/12/26/573609499... I think it might be feasible to scrape a subset: a single account or a few accounts at most. But even then, you're still fighting problems ranging from wonky HTML generated through JavaScript towards sudden, unannounced changes to the DOM structure. It's not a managed API after all.