3 ms·
A DOM aggregator/filter browser plugin. Let's say I'm looking for a room on AirBNB. The filter they have on their website isn't specific enough: if I want to f
by archagon 13y ago
A DOM aggregator/filter browser plugin.
Let's say I'm looking for a room on AirBNB. The filter they have on their website isn't specific enough: if I want to find a room that's $32 or less and that has at least 20 reviews, I have to use my eyeballs to filter all the < $35 results. What's more, there are many pages of results, making side-by-side comparison difficult. My idea is to create a browser plugin that harvests each element of a particular HTML class on a webpage (".search-result"), filters it based on the contents of some of the child classes (".listing-price < 32", ". listing-review-count > 20"), and puts all the results in a single window, probably using CSS copied from the original page. All I'd have to do in the AirBNB case is plug in the classes and comparators listed above and quickly cycle through each page of the listings with "gather mode" turned on. I'd then have all my preferred results in one place. Perhaps the plugin could even allow you to identify your classes by simply pointing and clicking on the desired nodes, much like the inspector in developer tools!
Ultimately, with this idea as the basis, I think you could make a great Automator-style tool that streamlines your research. Is there some information that's missing in the search results? Make it follow the links and filter the full listing descriptions! Looking for listings with specific kinds of owners? Do some keyword matching on each profile! Want a map of all your filtered locations? Gather the GPS data and display it in a beautiful native map interface! (Maybe you could even interface with external services like Wolfram or Yelp: crime statistics, nearby restaurant reviews, Amazon ratings?) Kind of like a web scraper, but with a very user-friendly UI specifically tailored for filtering and aggregating search results. You wouldn't have to keep 100 tabs open anymore when you're doing research: everything would be saved and synced in a handy Evernote-like interface.
I've run into this problem frequently enough with comparison shopping that I think there's value in this idea. I'm not really all that inclined to make it unless I can make some money, but I made a little Javascript rough draft script that can filter a webpage's DOM if you manually enter all the classes and comparators and copy/paste it into the dev console. A lot more work than the browser plugin I described, but still very useful for my purposes.
https://gist.github.com/archagon/10233251 https://gist.github.com/archagon/10233251
- thom 13y agoimport.io might be close to what you want here. You create a custom web scraper with a nice visual interface, it extracts the data for you.
- archagon 13y agoThat looks really promising! I will have to look into that. The previous discussion on HN[1] has a few more interesting leads. A lot of people there are advocating writing their own scraper, but the point of this sort of idea is to allow you to get going in just a few seconds, without ever having to poke around in the dev tools, debug custom scripts, or even look at the HTML source. If you're going to be doing research for an hour, spending half an hour on writing your own scraper just feels like a waste. [1] https://news.ycombinator.com/item?id=7582858 https://news.ycombinator.com/item?id=7582858