4 ms·

Are you still scraping or did you migrate to their API?

• nevster 10y ago
  Still scraping. If you use their API, you need to submit your app for evaluation and there are a bunch of restrictions and other things you need to comply with. I'd need to do a whole lot of work and remove some of the features. The level of effort has never felt worth it. It means there are occasional breakages every 6 months or so but it's usually the work of half an hour to fix it and push out an update. My users very quickly tell me if it's broken. ;)
  • laaph 10y ago
    Can I ask you what you are using to scrape with? I am finding that Amazon's anti-bot strategies change every so often, so unless I use phantom.js or Selenium, I have to change odd things like headers and various things every so often. I finally reworked my code to just download the html files using phantom.js and parse it with perl so that I can run this on a headless server, and this method hasn't broken lately. I find trying to use phantom.js is kind of crazy (mainly I have a hard time with the documentation) and javascript is not my native language. I was using curl, LWP, and wget in the past.
    • nevster 10y ago
      I'm not using any libraries. All hand coded to just get the bits it needs in the html using lots of indexOf. With regards to the anti-bot problem, this isn't running on any server, it's running on the user's computer. The number of requests they make per second doesn't seem to trigger any anti-bot measures. I'm just using plain Java URLConnections.