4 ms·
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interf
by ricardobeat 13y ago
HN pages are possibly the worst case, very hard to infer structure from due to its 1998 coding standards. You'll have a better chance with an alternative interface like http://ihackernews.com/ http://ihackernews.com/ or http://hckrnews.com http://hckrnews.com (no comments though).
- roel_v 13y agoOK so you're saying that instead of using a scraper to deal with malformed data out there (the whole reason of its existence), instead we should use a format that is better suited for machine representation? That's like saying 'yeah I've got this car here to take you to places that are very far away, too far to walk; except it doesn't work very well if you want to go far away, so you're betting off just staying at home. Or walk along time.'.
- ricardobeat 13y agoScrapers exist to take data out of HTML, malformed or not, when an API or feed is not available. Yes, ideally HTML is written in a semantic, annotated machine-parseable way; that also enables smarter search engines, better accessibility, interoperability and so on. That's one of the main reasons behind the changes to the standard made in HTML5. A better example would be "hey I have this cool device that will improve your mileage by 100%, but it only works on cars built according to current emission standards...".
- pranade 13y agoKimono can handle many pages with malformed/ old HTML. Of course, it's still beta and there are still pages that break it, so we're improving it as we go with the help of early adopters like everyone on this thread. Of course, our goal is an ideal state it works everywhere perfectly :)
- voltagex_ 13y agoA lot of the time when I'm writing a scraper it's because of bad/old code or incorrect HTML. So if Kimono has issues with that its utility is reduced.
- pranade 13y agoKimono can handle several pages with malformed and old/bad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.
- voltagex_ 13y agoSounds good. I've made a note of it for the next time I need to screen scrape something.
- tlarkworthy 13y agoI developed some hot strategies for the next level ... http://edinburghhacklab.com/2013/09/probabalistic-scraping-of-plain-text-tables/ http://edinburghhacklab.com/2013/09/probabalistic-scraping-o...
- Nilzor 13y agoHave you thought of implementing a system for custom handling of edge cases? Realistically you can't handle all edge cases, so leaving the last 2% up to the user (which is a developer) would be a good idea. It'd still save him 98% of the work, but also give him the comfort of knowing it won't break even if site X is doing something weird with their HTML in the future. I'll leave it to you to figure out how one would implement such a feature ;-)
- wilg 13y agoI really wish that HN wasn't even in the running for a "worst case". For a community that seems to be all about UX and innovation, shouldn't it run on at least a marginally user-friendly piece of software with this-century markup? I get that it has a sort of kitschy or retro appeal, but it's just basically a pain to use and looks terrible. I can't tell you how often I click next page to find that I've taken too long and my session or whatever has expired.
- tinco 13y agoNo. This site isn't about UX or innovation, it's about tech start ups. It's a constant reminder that something can be successful even if it was written in a LISP dialect and has a bunch of UX misses as long as the core idea is valid and the product usable enough. I've used a bunch of HN skins that were supposedly better designed, but none of them stuck. Apparently it's just plain unnecessary for HN to be better.
- theg2 13y agoSo because it's written in LISP it gets a free pass in other areas? That seems like a dubious claim.
- tinco 13y agoIndubitably.
- Flimm 13y agoThat's not what he said.
- SkyMarshal 13y agoNo, despite it being written in an obscure language and sparse UI it has massive traction anyway. There's a lesson in there somewhere.
- bmelton 13y ago> There's a lesson in there somewhere The website that lets technologists hang out with people who might give them large sums of money to see their ideas succeed doesn't have to be good, or pretty. If the website were better, or prettier, that would not add any additional value to the previously mentioned large sums of money. (Oh, and advice, and being able to chat with industry leaders and experts on diverse arrays of topics, etc.)
- easy_rider 13y agoI'd love to argue that PG probably thinks this is a good thing. (no scraping). And for good reasons. Who is paying the bandwidth bills in the end, and for whom, you might ask. And then if those users will end up contributing to HN in any intellectual way in the end, or are just scraping content to spin in their auto-blogs filled with ads.
- riffraff 13y agohn pages are not too hard, each story is a <tr>, title is in 'tr .title' and link is in 'tr .title a'. There are some irregularities (i.e. YC announcements without score, more button, self posts) but it's not really structurally complicated (compare reddit, which has 3 "score" fields per link)
- ronaldx 13y agoThis is awkward, but I disagree. HN pages are very well structured and are probably an ideal case for an automatic scraper. If your automatic scraper doesn't work on HN, it is unlikely to work generally.