3 ms·
> As well as an algorithm for efficiently converting/compressing large html pages into a semantic format. For the love of humanity please open source this. Thi
by stormfather 2y ago
> As well as an algorithm for efficiently converting/compressing large html pages into a semantic format.
For the love of humanity please open source this. This seems tremendously useful by itself.
- jawerty 2y agoOh damn I will definitely look into open sourcing it and making it a sdk
- stormfather 2y agoAwesome! I write LLM powered scrapers and stuff all the time and one of the biggest pain points is HTML is full of so much crap that isn't meaningful and overwhelms the context. And being a data science guy idk how to solve this.
- jawerty 2y agoawesome that's the same reason why I use it. It's basically a balance between the full html and having the markdown type scrapers that are better for just text. Do you mind if I reach out to you once I set up the Github?
- stormfather 2y agoYou're very welcome to! Please do. You can reach out to notpricedinyet@gmail.com
- pavelfeldman 2y agoThere is an open source alternative that might be even better: https://playwright.dev/docs/api/class-locator#locator-aria-snapshot https://playwright.dev/docs/api/class-locator#locator-aria-s....