8 ms·
ScrapeGraphAI: Web scraping using LLM and direct graph logic
- simonw 2y agoTypo on your homepage: "You just have to implment just some lines of code and the work is done"
- syntaxgoonoo 2y ago[dead]
- sethx 2y agoAt jobstash.xyz we have similar tech as part of our generalized scraping infra, and it’s been live for half a year performing optimally.
- senordevnyc 2y ago[dead]
- nextworddev 2y agoWould be nice if docs had a comparison between traditional scraping (e.g. using headless browsers, beautifulsoup, etc) versus this approach. Exactly how is AI used?
- geuis 2y agoA lot of larger LLM's have been trained on millions of pages of html. They have the ability to understand raw html structure and extract content from them. I've been having some success with this using Mixtral 8x7B.
- ushakov 2y agoThere’s also llm-scraper in TypeScript https://github.com/mishushakov/llm-scraper https://github.com/mishushakov/llm-scraper
- lucgagan 2y agoSomething similar I worked on in the past https://github.com/lucgagan/auto-playwright/ https://github.com/lucgagan/auto-playwright/
- worldsayshi 2y agoDoes it use ChatGPT every time you run the test or only when a test fails (to check if the selector has changed)?
- nodoodles 2y agoWhat I'd love to see is scraper builder that uses LLMs/'magic' to generate optimised scraping rules for any page, ie css selectors and processing rules mapped to output keys. So you can run scraping itself at low cost and high performance..
- cpobuda 2y agoI have been working on this. Feel free to DM me.
- geuis 2y agoThat's an interesting take. I've been experimenting with reducing the overall rendered html size to just structure and content and using the LLM to extract content from that. It works quite well. But I think your approach might be more efficient and faster.
- nodoodles 2y agoOne fun mechanism I've been using for reducing html size is diffing (with some leniency) pages from same domain to exclude common parts (ie headers/footers). That preprocessing can be useful for any parsing mechanism..
- greggsy 2y agoIt seems also obvious that one would want to simply drag a box around the content you want, and the tool would just provide some examples to help you refine the rule set. Ad blockers have had something very close to this for some time, without any sparkly AI buttons. I’m sure someone would be working on a subscription based model using corporate models in the backend, but it’s something that could easily be implemented with a very small model.
- uptown 2y agoMozenda does something like that. I haven't used it in many years, so I'm not up to date on what it currently offers.
- 2y ago
- mariopt 2y agoWhat is the point of using LLMs for the scrapping itself instead of using them to generate the boring code for mimicking HTTP requests, css/xpath selectors, etc? I get it may be interesting for small tasks combined with a browser extension but for real scrapping just seems to be overkill and expensive.
- geuis 2y agoIt is potentially expensive, but here's a different take. Instead of writing a bunch of selectors that break often, imagine just being able to write a paragraph telling the LLM to fetch the top 10 headlines and their links on a news site. Or to fetch the images, titles, and prices off a store front? It abstracts away a lot of manual fragile work.
- mariopt 2y agoI get that and LLMs are expected to get better. Today, would you build a scraper with current LLMs that randomly hallucinate? I wouldn't. The idea of a LLM powered scraper adapting the selectors every time the website owner updates it, it's pretty cool.
- ewild 2y agoAt my job we are scraping using LLMs. For a 10M sector of the company. GPT4 turbo has never not once out of 1.5 million API requests hallucinated. We however use it to parse data and interpret it from webpages, this is something you wouldn't be able to do with a regular scraper. Not well atleast.
- what 2y agoBold claim, did you review all 1.5 million requests?
- krainboltgreene 2y agoHow do you know that's true?
- spaniard89277 2y agoThis is completely unrealistic unless you want to burn money.
- msp26 2y agoIn practice it's actually very good for lower volume tasks with non-fixed sources. I haven't tried this library but I do use an LLM based scraper in addition to more traditional ones.
- infecto 2y agoI have not used this specific library but its far from unrealistic and hardly a money pit. A LLM can fit in nicely with scraping libraries. Sure if you are crawling the web like google, it makes no sense, but if you have a hit list, this can be a cost effective way to not have engineering hours spent maintaining the crawler.
- spaniard89277 2y agoWhich LLM do you use? Because I can't see an scraper running daily without being very expensive.
- a_wild_dandan 2y agoLlama-3 70B on my local MacBook works wonderfully for these tasks.
- spaniard89277 2y agoHow's the Pipeline? Do you pass all the html to the LLM? Isn't the context window a problem?
- a_wild_dandan 2y agoThere are phenomenal web scraping tools to crudely "preprocess" the document a bit, slashing outer HTML fluff while preserving the small subset of actual data. From there, 8k tokens (or whatever) goes really far.
- nurettin 2y agoA lot of websites (like online shopping ) won't let you scrape for long unless you are logged in. Some (like real estate sites) won't tolerate you for long even if you are logged in. Some (like newspapers) won't accept a simple request, they will try to detect browser and user behavior. Some will even detect data center ip blocks to get rid of you. I don't believe scraping is such a solved problem that you can slap AI and some cute vector spiders on it and claim that everything works.
- RamblingCTO 2y agoI don't see the benefit. I don't wanna say what I built 2markdown.com with (converts websites to markdown for LLMs), but it has a pretty decent performance without any high cost (and also sometimes erroneous) LLMs thrown on top of the scraping.
- dev213 2y agoThe "Integrates Easily" links to Langchain and Pipedream are flipped on your website.
- RamblingCTO 2y agoOuch, thanks! Will fix asap
- holoduke 2y agoWriting a few selector queries is probably easier than letting a llm output the selector queries by instructing the llm with the webpage and output request. I do scraping for a living. Casperjs/phantomjs is still my best friend for scraping dynamic websites. Selector Queries are the least of a problem.
- barrenko 2y agotell me more about your hobby?
- deleted 2y ago[deleted]
- maxrmk 2y agoI'd love to try the demo but there's no way I'm putting my openai key into that site.