3 ms·
> Web scraping is usually pretty tedious, but I found that I could send the minimised HTML to GPT-3 and get (almost) perfect JSON back: the prompt includes the
by nextaccountic 4y ago
> Web scraping is usually pretty tedious, but I found that I could send the minimised HTML to GPT-3 and get (almost) perfect JSON back: the prompt includes the Typescript definition.
Could you share the prompt? Or, if OP can't share, does anyone have ideas for a prompt to do something like this?
- kwerk 4y ago+1, and the OP mentions a wrapper to handle invalid JSON from GPT-3. I’d be interested in that too OP’s other write up: https://interconnected.org/home/2023/02/07/braggoscope https://interconnected.org/home/2023/02/07/braggoscope
- malborodog 4y agoThe prompt is probably simple, but the bigger challenge is that even a minified html of a typical web page would be more than the 4k gpt token limit
- genmon 4y agoI extract the main content div, which includes various other divs and assorted HTML cruft from 25 years of content management systems. Then convert that to Markdown, which GPT groks happily, and it preserves the right balance of discarding meaningless structure but preserving some semantics (italics, headings, etc). The best tool I've found for that process is aaronsw's html2text, amazing that it's still so valuable after all these years.
- malborodog 4y agoThanks for explaining — very helpful!
- genmon 4y agoShared in this comment https://news.ycombinator.com/item?id=35073824 https://news.ycombinator.com/item?id=35073824