4 ms·

Do you need to scrape data from 100s to 1000s of websites?

We've been working on AI agents and in the process found what we built could be good at dynamically scraping data across a variety of websites (10s to 100s of different sites at a time) Existing web scraping tools are bad at this because you need to write custom configurations per site. Not only that, but when a site changes styling, it might completely break your automation We’re curious how useful this might be for people. If you’ve experienced issues that this might solve or have already explored the space, we’d love to talk to you!

airstrike 3y ago
you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link* https://news.ycombinator.com/shownew https://news.ycombinator.com/shownew https://hn.algolia.com/?q=Launch+HN https://hn.algolia.com/?q=Launch+HN having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai https://hn.algolia.com/?q=reworkd.ai
- asim-shrestha 3y ago
  Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem
  - airstrike 3y ago
    Got it. Maybe an Ask HN then? Otherwise the question may not be noticed in the hundreds of daily submissions. It may be worth resubmitting as an Ask HN. The question mark makes it sound like the start of an infomercial, but it may just be me ;-)
    - asim-shrestha 3y ago
      Gotcha, will do Appreciate the feedback, very new to posting here
  - verdverm 3y ago
    use AskHN then