Show HN: Generate Web Scrapers for Any Website with GPT

We got frustrated with the time and effort required to code and maintain custom web scrapers, so we built a more generic LLM-based solution for data extraction from websites. AI should automate tedious and un-creative work, and web scraping definitely fits this description. We're leveraging LLMs (GPT and fine-tuned Flan-T5) to semantically understand websites and generate the DOM selectors for it. Using GPT for every data extraction, as most comparable tools do, would be way too expensive and very slow, but using LLMs to generate the scraper code and subsequently adapt it to website modifications is highly efficient. Extracting and transforming data is one of the underestimated superpowers of LLMs. Try it out on our playground https://kadoa.com/playground https://kadoa.com/playground (OpenAI key required) and let me know what you think! Here is a rather impressive example of extracting gaming stats: https://www.kadoa.com/playground?session=e1226fa8-9fa2-4eba-9e7f-00f681a59dc6 https://www.kadoa.com/playground?session=e1226fa8-9fa2-4eba-... There is still a lot of work ahead of us. Extracting a few data records from a single page with GPT is quite easy. Reliably extracting 100k records from 10 different websites on a daily basis is a whole different beast: * Ensuring data accuracy (veryifying that the data is on the website, adapting to website changes, etc.) * Handling large data volumes * Managing proxy infrastructure * Elements of RPA to automate scraping tasks like pagination, login, and form-filling We are spending a lot of effort solving each of these points with custom engineering and fine-tuned LLM steps.