4 ms·
I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details
by fareesh 8y ago
I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details
- jacquesm 8y agoI'd go the other way and say that pages that no longer contain relevant information in a normally digestible format should be dropped from search engines and other automated indices. After all, the web was built on accessibility of information, not on purposeful obfuscation. If you go so far as to essentially flatten the webpage to the point where you might as well print it out and then do OCR on it then you've thrown out the baby with the bathwater, you had all that information when you started. Or at least, you should have had it. Otherwise we might as well kiss HTML goodbye and render the web as pdfs, with or without links.
- zzzcpan 8y agoThe biggest search engine doesn't have your best interests at heart and has been trying to make HTML and accessibility of information obsolete for years. Some pages now render only with javascript or require solving javascript challenge to even get to the rendering (hello cloudflare) and essentially kissed HTML goodbye.
- deleted 8y ago[deleted]
- nerdponx 8y agoOCR tech is good but it's not there yet.
- vinceguidry 8y agoI want machine learning, but definitely not to identify information off of a gif. Websites are too dynamic for that to really be feasible. Say I want to scrape plane ticket prices. Getting the info off the page once you're there is the easy part. The hard part is figuring out how to navigate the website. I don't want to do the annoying manual work of programming page traversal. I want to be able to make a ML bot that does it for me. That way no matter how often they change the interface, all I have to do is re-run the ML.
- dredmorbius 8y agoDumping rendered text, aswith lynx or w3m, is another option. There's a lot of very bad HTML out there.
- tegansnyder 8y agoMaybe someone can chime in, but I'm pretty sure Diffbot does something similar.
- dwynings 8y agoHey Tegan, Answered on the parent, but it's somewhat similar.
- dwynings 8y agoFair warning: I work at Diffbot. Essentially that's what Diffbot (https://www.diffbot.com/ https://www.diffbot.com/) does, except we don't the render pages as an image nor do OCR. Diffbot renders the page in a headless browser, and uses computer vision to automatically identify the key page attributes and extract normalized data for specific page types (Articles, Products, Discussions, Profiles, Images, and Videos). This approach enables us to work in any language and on sites that we've never come across before automatically with better than human level accuracy.