3 ms·

Anyone know of any research on generating HTML differentials against updated webpages and with automatic healing of wrappers / selectors or research on using LL

Anyone know of any research on generating HTML differentials against updated webpages and with automatic healing of wrappers / selectors or research on using LLMs with webscraping and how to reduce token usage while retaining context?

simonw 3y ago
I've been doing some pretty dumb tricks for reducing token usage and piping to LLMs which have worked really well. I have a strip-tags CLI tool which I can pipe HTML through on its way to an LLM, described here: https://simonwillison.net/2023/May/18/cli-tools-for-llms/ https://simonwillison.net/2023/May/18/cli-tools-for-llms/ I also do things like this: shot-scraper javascript news.ycombinator.com 'document.body.innerText' -r \ | llm -s 'General themes, illustrated by emoji' Output here: https://gist.github.com/simonw/3fbfa44f83e12f9451b58b595451414a https://gist.github.com/simonw/3fbfa44f83e12f9451b58b5954514... That's using https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ to get just the document.body.innerText as a raw string, then piping that to gpt-3.5-turbo with a system prompt. In terms of retaining context, I added a feature to my strip-tags tool where you can ask it to NOT strip specific tags - e.g.: curl -s https://www.theguardian.com/us | \ strip-tags -m -t h1 -t h2 -t h3 That strips all HTML tags except for h1, h2 and h3 - output here: https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd8871f https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd88... Full documentation here: https://github.com/simonw/strip-tags/blob/main/README.md https://github.com/simonw/strip-tags/blob/main/README.md
- _boffin_ 3y ago
  First, love your work. Sadly, I don't think this path would work for me as for what I do, I need the selectors as part of the workflow. Roughly, my end goal is to do a single or multi-shot with the following information HTML differential (could be selectors, xpaths, data regions, differentials of any of the above, etc...), code stacktrace, related code, and prompt. For this example, let's consider that the flow involves the bot to login to a website. I have selectors for the `.username` and `.password` inputs and then a selector for the login button as `.login-btn`. 1. The site updates their page and changes up all their IDs, but keeps the same structure. 2. The site updates their page and changes up all their IDs, but changes the structure and the form is named something different and is somewhere else in the DOM. 3. many... many other examples. Trying to figure out how to minimize the tokens, but keep the needed context to regenerate the selectors that are needed to maintain the workflow.
  - simonw 3y ago
    Yeah, I've been thinking a bit about that kind of problem too. My hunch is you could do it with a much more complex setup involving OpenAI functions - by trying different things (like "list just input elements with their names and associated labels") in a loop with the LLM where it gets to keep asking follow-up questions of the DOM until it finds the right combination.
    - _boffin_ 3y ago
      Would love to connect with you and discuss this further.
  - zenyc 3y ago
    I’m also working on this issue. Open to brainstorming some approaches together
    - _boffin_ 3y ago
      Lets! Would love to hear your thoughts.
      - zenyc 3y ago
        z (at) 6bow.com
        ryanwaggoner 3y ago
        I’d like to join the discussion if I can, I’m actively working in this area.

simonw 3y ago

I've been doing some pretty dumb tricks for reducing token usage and piping to LLMs which have worked really well. I have a strip-tags CLI tool which I can pipe HTML through on its way to an LLM, described here: https://simonwillison.net/2023/May/18/cli-tools-for-llms/ https://simonwillison.net/2023/May/18/cli-tools-for-llms/ I also do things like this: shot-scraper javascript news.ycombinator.com 'document.body.innerText' -r \ | llm -s 'General themes, illustrated by emoji' Output here: https://gist.github.com/simonw/3fbfa44f83e12f9451b58b595451414a https://gist.github.com/simonw/3fbfa44f83e12f9451b58b5954514... That's using https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ to get just the document.body.innerText as a raw string, then piping that to gpt-3.5-turbo with a system prompt. In terms of retaining context, I added a feature to my strip-tags tool where you can ask it to NOT strip specific tags - e.g.: curl -s https://www.theguardian.com/us | \ strip-tags -m -t h1 -t h2 -t h3 That strips all HTML tags except for h1, h2 and h3 - output here: https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd8871f https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd88... Full documentation here: https://github.com/simonw/strip-tags/blob/main/README.md https://github.com/simonw/strip-tags/blob/main/README.md

_boffin_ 3y ago
First, love your work. Sadly, I don't think this path would work for me as for what I do, I need the selectors as part of the workflow. Roughly, my end goal is to do a single or multi-shot with the following information HTML differential (could be selectors, xpaths, data regions, differentials of any of the above, etc...), code stacktrace, related code, and prompt. For this example, let's consider that the flow involves the bot to login to a website. I have selectors for the `.username` and `.password` inputs and then a selector for the login button as `.login-btn`. 1. The site updates their page and changes up all their IDs, but keeps the same structure. 2. The site updates their page and changes up all their IDs, but changes the structure and the form is named something different and is somewhere else in the DOM. 3. many... many other examples. Trying to figure out how to minimize the tokens, but keep the needed context to regenerate the selectors that are needed to maintain the workflow.
- simonw 3y ago
  Yeah, I've been thinking a bit about that kind of problem too. My hunch is you could do it with a much more complex setup involving OpenAI functions - by trying different things (like "list just input elements with their names and associated labels") in a loop with the LLM where it gets to keep asking follow-up questions of the DOM until it finds the right combination.
  - _boffin_ 3y ago
    Would love to connect with you and discuss this further.
- zenyc 3y ago
  I’m also working on this issue. Open to brainstorming some approaches together
  - _boffin_ 3y ago
    Lets! Would love to hear your thoughts.
    - zenyc 3y ago
      z (at) 6bow.com
      - ryanwaggoner 3y ago
        I’d like to join the discussion if I can, I’m actively working in this area.