4 ms·

A year. I was spending way too much time on eBay, so I wrote something for myself that would scrape the web pages and use a kill list to filter out the junk. I

by nevster 10y ago

A year. I was spending way too much time on eBay, so I wrote something for myself that would scrape the web pages and use a kill list to filter out the junk. I wanted a better UI to add words to the kill list and realised I could make money via the eBay affiliate program. So I took two months off between contracts and wrote the first version of AuctionSieve - http://auctionsieve.com http://auctionsieve.com There was a D&D forum, the Acaeum where a bunch of people started using it and giving feedback. It was making me money from day 1 but it probably took about a year to repay that 2 months of time investment. It's now been 13 years(!) and it still makes me money - not enough to live off (the payout calculations from eBay have changed several times) but a nice chunk of change. And I only have to occasionally prod it. And add the occasional new feature.

nevster 10y ago
For those curious, it makes about $500 or so per month. At one point it was making $2000 per month but eBay changed the payout strategy - one thing to be aware of when dealing with affiliate programs.
shaunpud 10y ago
Are you still scraping or did you migrate to their API?
- nevster 10y ago
  Still scraping. If you use their API, you need to submit your app for evaluation and there are a bunch of restrictions and other things you need to comply with. I'd need to do a whole lot of work and remove some of the features. The level of effort has never felt worth it. It means there are occasional breakages every 6 months or so but it's usually the work of half an hour to fix it and push out an update. My users very quickly tell me if it's broken. ;)
  - laaph 10y ago
    Can I ask you what you are using to scrape with? I am finding that Amazon's anti-bot strategies change every so often, so unless I use phantom.js or Selenium, I have to change odd things like headers and various things every so often. I finally reworked my code to just download the html files using phantom.js and parse it with perl so that I can run this on a headless server, and this method hasn't broken lately. I find trying to use phantom.js is kind of crazy (mainly I have a hard time with the documentation) and javascript is not my native language. I was using curl, LWP, and wget in the past.
    - nevster 10y ago
      I'm not using any libraries. All hand coded to just get the bits it needs in the html using lots of indexOf. With regards to the anti-bot problem, this isn't running on any server, it's running on the user's computer. The number of requests they make per second doesn't seem to trigger any anti-bot measures. I'm just using plain Java URLConnections.

by nevster 10y ago

nevster 10y ago

For those curious, it makes about $500 or so per month. At one point it was making $2000 per month but eBay changed the payout strategy - one thing to be aware of when dealing with affiliate programs.

shaunpud 10y ago

Are you still scraping or did you migrate to their API?

nevster 10y ago
Still scraping. If you use their API, you need to submit your app for evaluation and there are a bunch of restrictions and other things you need to comply with. I'd need to do a whole lot of work and remove some of the features. The level of effort has never felt worth it. It means there are occasional breakages every 6 months or so but it's usually the work of half an hour to fix it and push out an update. My users very quickly tell me if it's broken. ;)
- laaph 10y ago
  Can I ask you what you are using to scrape with? I am finding that Amazon's anti-bot strategies change every so often, so unless I use phantom.js or Selenium, I have to change odd things like headers and various things every so often. I finally reworked my code to just download the html files using phantom.js and parse it with perl so that I can run this on a headless server, and this method hasn't broken lately. I find trying to use phantom.js is kind of crazy (mainly I have a hard time with the documentation) and javascript is not my native language. I was using curl, LWP, and wget in the past.
  - nevster 10y ago
    I'm not using any libraries. All hand coded to just get the bits it needs in the html using lots of indexOf. With regards to the anti-bot problem, this isn't running on any server, it's running on the user's computer. The number of requests they make per second doesn't seem to trigger any anti-bot measures. I'm just using plain Java URLConnections.