5 ms·
Nice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.
by ricardo81 5y ago
Nice, but it's essentially scraping? Scraping is brittle.
Some design changes and the whole thing breaks. Maintenance nightmare.
- ricardo81 5y agoNot sure for the downvote reason. Like I say scraping is brittle. I've done hundreds of scraping projects in the past. A recent one was taking 50 shopping sites looking for what products they covered. Safe to say you look at it a month later and the XPaths/CSS selectors/whatever scraping method you used has changed. This is why APIs exist.
- Vinnl 5y agoIdeally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…
- ricardo81 5y agoIndeed, but the scraping workaround is brittle. We could make rules for all sites on the web that don't have a feed or API but it's not all that manageable. It also begs the question of why they don't make their content that way, perhaps it was by choice. Especially if you use this kind of tool to syndicate things. Not quite sure of the novelty of this one given that people have scraped things for decades. There's probably a wiki endpoint for their example, maybe, maybe not. A lot of wiki's stuff is free to download and they have extensive endpoints for acquiring data.
- eyedontgetit 5y agoRe-stating the "brittleness" over and over doesn't add any weight to your argument. Regardless of whether or not the solution is brittle - it is still a viable solution for users who don't care or mind maintaining it. It sounds like you're not the target user and that's ok...
- Vinnl 5y agoIf you have a less brittle alternative that allows me to get the latest real estate listings in my area in my feed reader before it hits the aggregator sites, I'm all ears!
- ricardo81 5y agoI can't reply to the other subcomments because of the downvotes. Just to be clear, I'm not against scraping. Done it myself plenty, point was that it's hard to maintain and if I were to make a tool for the masses, it'd be in a UI that highlights elements and lets you select them rather than dig around the DOM. Pretty sure things like this have existed for 10 years +
- Vinnl 5y agoAh that's fine, this is explicitly not a tool for the masses. The problem I had with tools like the ones you describe are that either they just use string matching, or the selectors they generate are particularly brittle (e.g. incorporating randomised class names, like you mentioned elsewhere). Given that there wasn't really an alternative to scraping for me, I wanted to at least be able to pick selectors myself that were less likely to break with minor changes. Then I figured there might be more people who know CSS and have the same desires, hence my sharing it here :)
- ricardo81 5y agoI do have some code lying around somewhere that would revive wordpress blogs via wayback, using a UI where you could select elements in the 'theme' and restore the backend database. It's simply a messy business IMO. But for one off jobs, scraping is the go to.
- bscphil 5y agoI'm not sure I understand the criticism. What exactly is supposed to be the alternative? It's not like this is supposed to be used for high-availability purposes or stock trading or something. It scratches a particular itch for a handful of end users who just want to get all their news in their feed reader. It's expected that they'll maintain their own scraper parameters.
- ricardo81 5y agoPick a number of programming languages and you can scrape a page in 10-20 lines of code in many cases. The barrier to entry is understanding the DOM layout for a particular website, which is subject to change at any moment. Purely IMO, a more friendly way to go about it to abstract from code and CSS knowledge is to run a UI that highlights elements, lets you select them, select the title, description, link etc and there you go. Same thing but without the knowledge of DOM/XPath/selectors.
- cromka 5y ago> highlights elements, lets you select them, And how would that software then remember your chosen elements if not by their CSS ids/classes?
- z3c0 5y agoBy positional indexing, of course. Just kidding, that's horrendous.
- ricardo81 5y agoYou must have missed the point, or I'm missing the point of this thing. Select whatever you like, do I need an external tool to do that? Either I have knowledge of the DOM or I don't, and if I don't then a UI selector would be the next best guess.
- kortilla 5y agoWebsite doesn’t offer API. So we must scrape. This is a tool for that paradigm. Your comment is “scraping is brittle”, which is not helpful. Everyone knows scraping sucks. It’s why these tools are being made to make it less unpleasant.
- ricardo81 5y agoIt's not even that 'scraping sucks', that's just a stigma to the word. The point is that it's not stable and having to have DOM knowledge to select the paths is less inclusive.
- moehm 5y ago> having to have DOM knowledge to select the paths is less inclusive Right click on the element in Firefox, click "inspect element" and it shows you the unique selector for that element.
- lhnz 5y agoIf you scrape using accessible parts of the page (e.g. `aria-` attributes) it is less likely to be brittle, since if it were it would mean their site had stopped being accessible to screen readers, etc.
- KirillPanov 5y agoWhy the downvotes here? This is objectively true and, in my experience, very very useful. Screen readers are scrapers too. "Scraper" does not mean "user agent I don't like".
- DoreenMichele 5y agoIs there someplace I can read more about this idea? Any aspect of it. I'm one of the less technically literate members of HN, but I'm a co-owner of a blind dev group. Please and thank you.
- KirillPanov 5y agohttps://developer.mozilla.org/en-US/docs/Web/Accessibility/An_overview_of_accessible_web_applications_and_widgets https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]