3 ms·
I have not used this specific library but its far from unrealistic and hardly a money pit. A LLM can fit in nicely with scraping libraries. Sure if you are craw
by infecto 3y ago
I have not used this specific library but its far from unrealistic and hardly a money pit. A LLM can fit in nicely with scraping libraries. Sure if you are crawling the web like google, it makes no sense, but if you have a hit list, this can be a cost effective way to not have engineering hours spent maintaining the crawler.
- spaniard89277 3y agoWhich LLM do you use? Because I can't see an scraper running daily without being very expensive.
- a_wild_dandan 3y agoLlama-3 70B on my local MacBook works wonderfully for these tasks.
- spaniard89277 3y agoHow's the Pipeline? Do you pass all the html to the LLM? Isn't the context window a problem?
- a_wild_dandan 3y agoThere are phenomenal web scraping tools to crudely "preprocess" the document a bit, slashing outer HTML fluff while preserving the small subset of actual data. From there, 8k tokens (or whatever) goes really far.
- LunaSea 3y agoAt a very generous 50 tokens per second doesn't that still leave you with more than two and a half minutes (160s) processing time per document?
- mrbungie 3y agoGPT-3.5/GPT-4 ain't the only LLMs available. A Flan-T5/T5 or Llama2/3 8B models may be finetuning for this use case and used for much cheaper.
- spaniard89277 3y agoHow do you handle the context window limit? If you push the entire Dom to the LLM it will exceed the context window by far in most cases, isn't it?
- aleksiy123 3y agoMy guess is you do some preprocessing on the DOM to get it down to text but still retains some structure. Something like https://github.com/Alir3z4/html2text https://github.com/Alir3z4/html2text. I'm sure there are other (better?) options as well.
- tarasglek 3y agoI wrote https://markdown.download https://markdown.download as a general helper for this
- msp26 3y agoTrim unwanted html elements + convert to markdown. Significantly reduces token counts while retaining structure.
- infecto 3y agoAgain, depends on the volume of the scraping and the value of the data within it. Even 3.5 can be cost effective for certain workflows and data value.