3 ms·

Scraping cloud service

1 points by devsri 14y ago

I developed a scraping service that allows developers to extract data from any web page, set data collection frequency, send push notifications etc. The data extraction logic can be written in JavaScript. So, if you know how to write a browser extension to extract data, you know how to use my service. I wanted to see if there is any interest in such a service. Please email me at cloudserviceforapps@gmail.com if there is any interest in using the alpha version of the service.

pedalpete 14y ago
Can you explain how your service is different to 80legs? I don't think they've had much success, and I recall they had a few competitors at some point, but not sure who's left. I think the interesting thing with scaping services is that we likely need to write so much stuff to extract the data, that the actual scraping portion isn't a significant time drain. Just my two cents, as 80legs tried to get me to use their service when I was doing a ton of scraping, but I was more comfortable doing it myself.
- devsri 14y ago
  Currently, the scraping code to extract data is in JavaScript, but what I am building is a mechanism to "record and replay" solution (like Selenium), that can do the scraping with no coding. Other features include: - Multi-page scraping (example retrieving AA.com frequent flyer miles) - Mechanism to execute transactions (such as buying a ticket)
  - malandrew 14y ago
    If that's your product idea, then you may be asking the wrong people. I would imagine that the audience of people who want scraping but can't code resides elsewhere. Plus, even if this tool does fill a need would you expect the audience to set up their own data store as well? Generally people who are going to scrape data eventually plan on doing something with that data and I can't think of a scenario where a non-coder would would to scrape a lot of data and consume it in a way that doesn't require technical chops. FWIW, I'm working on a scraping project that I will probably open source and the feature that is most valuable to me is being informed when it breaks. In my case it's a real time scraper pulling data on demand, so immediate alert when a scraping formula fails is super important. If you want more feedback, I'd be happy to try it out. Email in profile.

1 points by devsri 14y ago

pedalpete 14y ago

Can you explain how your service is different to 80legs? I don't think they've had much success, and I recall they had a few competitors at some point, but not sure who's left. I think the interesting thing with scaping services is that we likely need to write so much stuff to extract the data, that the actual scraping portion isn't a significant time drain. Just my two cents, as 80legs tried to get me to use their service when I was doing a ton of scraping, but I was more comfortable doing it myself.

devsri 14y ago
Currently, the scraping code to extract data is in JavaScript, but what I am building is a mechanism to "record and replay" solution (like Selenium), that can do the scraping with no coding. Other features include: - Multi-page scraping (example retrieving AA.com frequent flyer miles) - Mechanism to execute transactions (such as buying a ticket)
- malandrew 14y ago
  If that's your product idea, then you may be asking the wrong people. I would imagine that the audience of people who want scraping but can't code resides elsewhere. Plus, even if this tool does fill a need would you expect the audience to set up their own data store as well? Generally people who are going to scrape data eventually plan on doing something with that data and I can't think of a scenario where a non-coder would would to scrape a lot of data and consume it in a way that doesn't require technical chops. FWIW, I'm working on a scraping project that I will probably open source and the feature that is most valuable to me is being informed when it breaks. In my case it's a real time scraper pulling data on demand, so immediate alert when a scraping formula fails is super important. If you want more feedback, I'd be happy to try it out. Email in profile.