5 ms·
I've been messing about with a toy search engine for a long time, and starting to tie it into my Pinboard bookmarks (6,000 and counting!), and the most irritati
by thaumaturgy 8y ago
I've been messing about with a toy search engine for a long time, and starting to tie it into my Pinboard bookmarks (6,000 and counting!), and the most irritating part of the whole thing for a while has been extracting useful text from modern websites.
I'd considered using a headless browser setup but I never got 'round to it.
If you structured this as a paid service where I could make some number of requests for some amount of money and get back basically what you've got now in monochrome mode, I'd probably sign up pretty quick.
This is sort of relevant to some other recent HN discussions around the trouble with large-scale crawling and the trouble with so much of it being restricted to Google now, so I don't think I'm the only one.
- tombh 8y agoI've thought about this too, because https://text.brow.sh https://text.brow.sh certainly does make scraping websites easy, you just need to `curl` and `grep`. But the trouble is, as I mentioned elsewhere in the thread, Browsh's text renderer is more of an art than a science, it doesn't guarantee consistent results for every render. Whereas of course downloading HTML and using something like Beautiful Soup does give a much better guarantee of consistency.
- bambataa 8y agoI don’t think that lack of consistency should stop you though. Ok maybe it wouldn’t be suitable for testing things but there are other uses for a headless browser
- jacquesm 8y agoThe web 1995-2005 or thereabout will be the best documented period of humanity and it is saddening to see how payload has been replaced by cruft to the point that we might as well have remote PDF viewers. So much for 'the semantic web' I guess.
- tombh 8y agoThis precisely the sort of thing that Browsh can remedy!