13 ms·

The recommendations are pretty good; even though I only input six books, it was enough for it to recommend books I have on my wish list. Definitely going to pla

The recommendations are pretty good; even though I only input six books, it was enough for it to recommend books I have on my wish list. Definitely going to play around some more. Plus, the website is super fast, very impressive. Any chance we could get an API going at some point? Are you planning to open source the work? I'm interested in the scrapping of Goodreads too. I'm building a book metadata aggregation API and plan on building a scrapper for Goodreads, but I imagine using a data center IP address will be a problem very fast. Were you scrapping from your home network?

costco 11mo ago
Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways. Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too expensive to serve. I guess if there is demand for it I could do it. I think most social book sites would probably like to own their recommendation system though.
- goatsi 11mo ago
  Speaking of sustained scraping for AI services, I found a strange file on your site: https://book.sv/robots.txt https://book.sv/robots.txt. Would you be able to explain the intent behind it?
  - costco 11mo ago
    I didn't want an agent to get stuck on an infinite loop invoking endpoints that cost GPU resources. Those fears are probably unfounded, so if people really cared I could remove those. /similar is blocked by default because I don't want 500000 "similar books for" pages to pollute the search results for my website but I do not mind if people scrape those pages.
- dbl000 11mo ago
  I would love an API or the dataset if you could share it somehow! Just to play around with my own book lists.

costco 11mo ago

Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways. Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too expensive to serve. I guess if there is demand for it I could do it. I think most social book sites would probably like to own their recommendation system though.

goatsi 11mo ago
Speaking of sustained scraping for AI services, I found a strange file on your site: https://book.sv/robots.txt https://book.sv/robots.txt. Would you be able to explain the intent behind it?
- costco 11mo ago
  I didn't want an agent to get stuck on an infinite loop invoking endpoints that cost GPU resources. Those fears are probably unfounded, so if people really cared I could remove those. /similar is blocked by default because I don't want 500000 "similar books for" pages to pollute the search results for my website but I do not mind if people scrape those pages.
dbl000 11mo ago
I would love an API or the dataset if you could share it somehow! Just to play around with my own book lists.