3 ms·
I dunno, maintaining a scraper across neverending site redesigns is a friggin' lot of work. Way more work than updating my code to use a new version of an API.
by octoberfranklin 6y ago
I dunno, maintaining a scraper across neverending site redesigns is a friggin' lot of work. Way more work than updating my code to use a new version of an API.
- eznzt 6y agoWhen was the last time goodreads was redesigned?
- genidoi 6y agoIf all that’s changing on a site is the presentation/ordering of elements, then you just need to update some XPath selectors every now and then and have a notification system for knowing when those selectors aren’t working. You can even automate this process by having a known input (eg, URL to a book on goodreads + its known [and hopefully unchanging] book title) and have a script that periodically checks that the xpath string matches the known pages text / generates a new one to point to the title. This is harder for values that do change but there are always workarounds
- octoberfranklin 6y agoI know. But it's like pager duty. When it breaks, I have to manually intervene to fix it, and in the meantime stuff is not working. And anything I build on top of that needs to take that into account. It turns fun projects into something like a job.