3 ms·
The ChatGPT part is pretty easy actually. You can just dump text and HTML and ask it a question, and it usually answers correctly. The trickier part is “everyt
by marcell 2y ago
The ChatGPT part is pretty easy actually. You can just dump text and HTML and ask it a question, and it usually answers correctly.
The trickier part is “everything else” to make the extension work.
- smcin 2y agoEven the parsing of obfuscated HTML + CSS + dynamic JSON content?
- marcell 2y agoSurprisingly yes, most of the time. I’ve put in a few optimizations: 1. Remove all <style> and <svg > tags. These rarely add value, and can dramatically increase token counts. 2. For the “crawl” step, I exclusively pull out <a> tags and only look at those. The “extract” step looks at full HTML 3. For now, it only looks at the first 50k text characters, and the first 120k HTML characters. This is to stay within token limits. The last part will be what I focus on improving in the next version.
- dillondoyle 2y agoCould go the google way, capture an image screenshot of state, ocr, then parse it. They keep throwing it in my url bar. I refuse to click (big warning it sends to google's servers)
- FrenchDevRemote 2y agohow do you deal with the fact that some basic pages can have tens of thousand of tokens?
- marcell 2y agoRight now, not much. The extension is fairly basic in that is just looks at the raw text + HTML and sends it to the LLM. The benefit of this approach is it's very simple and easy, but the downside is it sends a lot of unnecessary tokens to the LLM. That drives up the cost, slows things down, and hurts accuracy. I'm working on a few improvements now to improve this.
- superasn 2y agoI remember there was something called readability for chrome which is just what browsers have incorporated as reader view. And mozilla even had a stand-alone version of it (1). Might be of interest to you. [1] https://github.com/mozilla/readability https://github.com/mozilla/readability