3 ms·
Which LLM do you use? Because I can't see an scraper running daily without being very expensive.
by spaniard89277 2y ago
Which LLM do you use? Because I can't see an scraper running daily without being very expensive.
- a_wild_dandan 2y agoLlama-3 70B on my local MacBook works wonderfully for these tasks.
- spaniard89277 2y agoHow's the Pipeline? Do you pass all the html to the LLM? Isn't the context window a problem?
- a_wild_dandan 2y agoThere are phenomenal web scraping tools to crudely "preprocess" the document a bit, slashing outer HTML fluff while preserving the small subset of actual data. From there, 8k tokens (or whatever) goes really far.
- LunaSea 2y agoAt a very generous 50 tokens per second doesn't that still leave you with more than two and a half minutes (160s) processing time per document?
- mrbungie 2y agoGPT-3.5/GPT-4 ain't the only LLMs available. A Flan-T5/T5 or Llama2/3 8B models may be finetuning for this use case and used for much cheaper.
- spaniard89277 2y agoHow do you handle the context window limit? If you push the entire Dom to the LLM it will exceed the context window by far in most cases, isn't it?
- aleksiy123 2y agoMy guess is you do some preprocessing on the DOM to get it down to text but still retains some structure. Something like https://github.com/Alir3z4/html2text https://github.com/Alir3z4/html2text. I'm sure there are other (better?) options as well.
- tarasglek 2y agoI wrote https://markdown.download https://markdown.download as a general helper for this
- msp26 2y agoTrim unwanted html elements + convert to markdown. Significantly reduces token counts while retaining structure.
- infecto 2y agoAgain, depends on the volume of the scraping and the value of the data within it. Even 3.5 can be cost effective for certain workflows and data value.