4 ms·
This is completely unrealistic unless you want to burn money.
by spaniard89277 2y ago
This is completely unrealistic unless you want to burn money.
- msp26 2y agoIn practice it's actually very good for lower volume tasks with non-fixed sources. I haven't tried this library but I do use an LLM based scraper in addition to more traditional ones.
- infecto 2y agoI have not used this specific library but its far from unrealistic and hardly a money pit. A LLM can fit in nicely with scraping libraries. Sure if you are crawling the web like google, it makes no sense, but if you have a hit list, this can be a cost effective way to not have engineering hours spent maintaining the crawler.
- spaniard89277 2y agoWhich LLM do you use? Because I can't see an scraper running daily without being very expensive.
- a_wild_dandan 2y agoLlama-3 70B on my local MacBook works wonderfully for these tasks.
- spaniard89277 2y agoHow's the Pipeline? Do you pass all the html to the LLM? Isn't the context window a problem?
- a_wild_dandan 2y agoThere are phenomenal web scraping tools to crudely "preprocess" the document a bit, slashing outer HTML fluff while preserving the small subset of actual data. From there, 8k tokens (or whatever) goes really far.
- LunaSea 2y agoAt a very generous 50 tokens per second doesn't that still leave you with more than two and a half minutes (160s) processing time per document?
- mrbungie 2y agoGPT-3.5/GPT-4 ain't the only LLMs available. A Flan-T5/T5 or Llama2/3 8B models may be finetuning for this use case and used for much cheaper.
- spaniard89277 2y agoHow do you handle the context window limit? If you push the entire Dom to the LLM it will exceed the context window by far in most cases, isn't it?
- aleksiy123 2y agoMy guess is you do some preprocessing on the DOM to get it down to text but still retains some structure. Something like https://github.com/Alir3z4/html2text https://github.com/Alir3z4/html2text. I'm sure there are other (better?) options as well.
- tarasglek 2y agoI wrote https://markdown.download https://markdown.download as a general helper for this
- msp26 2y agoTrim unwanted html elements + convert to markdown. Significantly reduces token counts while retaining structure.
- infecto 2y agoAgain, depends on the volume of the scraping and the value of the data within it. Even 3.5 can be cost effective for certain workflows and data value.