I got frustrated with the time and effort required to code and maintain custom web scrapers, so me and my friends built a generic LLM-based solution for data ex

I got frustrated with the time and effort required to code and maintain custom web scrapers, so me and my friends built a generic LLM-based solution for data extraction from websites. AI should automate tedious and un-creative work, and web scraping definitely fits this description. We're leveraging LLMs to semantically understand websites and generate the DOM selectors for it. Using GPT for every data extraction, as most comparable tools do, would be way too expensive and very slow, but using LLMs to generate the scraper code and subsequently adapt it to website modifications is highly efficient. Try it out for free on our playground https://kadoa.com/playground https://kadoa.com/playground and let me know what you think! And please don't bankrupt me :) Here are a few examples: - Product Listings (Specialized Bikes) - Real Estate (Realtor) - Financial Data (Yahoo Finance) There is still a lot of work ahead of us. Extracting a few data records from a single page with GPT is quite easy. Reliably extracting 100k records from 10 different websites on a daily basis is a whole different beast: Ensuring data accuracy (verifying that the data is on the website, adapting to website changes, etc.) - Handling large data volumes - Managing proxy infrastructure - Elements of RPA to automate scraping tasks like pagination, login, and form-filling We are spending a lot of effort solving each of these points with custom engineering and fine-tuned LLM steps.