3 ms·
Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to th
by costco 11mo ago
Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways.
Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too expensive to serve. I guess if there is demand for it I could do it. I think most social book sites would probably like to own their recommendation system though.
- goatsi 11mo agoSpeaking of sustained scraping for AI services, I found a strange file on your site: https://book.sv/robots.txt https://book.sv/robots.txt. Would you be able to explain the intent behind it?
- costco 11mo agoI didn't want an agent to get stuck on an infinite loop invoking endpoints that cost GPU resources. Those fears are probably unfounded, so if people really cared I could remove those. /similar is blocked by default because I don't want 500000 "similar books for" pages to pollute the search results for my website but I do not mind if people scrape those pages.
- dbl000 11mo agoI would love an API or the dataset if you could share it somehow! Just to play around with my own book lists.