5 ms·
Python API for Hacker News
- Sharma 13y agoI think screen scrapping is not allowed by HN. Few tries with these APIs might get your IP banned!
- sprizzle 13y agoThe robots.txt file doesn't seem to disallow scraping. https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt
- karangoeluw 13y agoScraping the listing pages seems allowed though.
- scott_s 13y agoI don't think there's a prohibition to screen scraping, but if you make too many requests to the server in a certain amount of time, your IP will be banned to prevent the server from melting.
- EugeneOZ 13y agoAgree. Also HN have RSS.
- mapleoin 13y agoI tried building a REST API once for a challenge if anyone is interested: https://github.com/mapleoin/newhackers https://github.com/mapleoin/newhackers
- thejosh 13y agoDoes it use https://www.hnsearch.com/api https://www.hnsearch.com/api ?
- jonesetc 13y agoNope, it's scraping.
- bnejad 13y agoNot OP, but from a quick glance at the source it doesn't appear to. It downloads the pages and uses beautiful soup (python html parsing library).
- Xeoncross 13y agoIs that an official API? How long has it been around?
- karangoeluw 13y agoCompletely unofficial. I started creating it a month ago.
- dholowiski 13y agoWow, that's great. I use another one and it's quite unreliable. Thanks!
- karangoeluw 13y agoYou can use mine and compare the two, and based on your feedback either I or any other dev can improve it.
- karangoeluw 13y agoIt scraped, slices.
- sprizzle 13y agoIt's silly to use BeautifulSoup to parse the page when you could use a simple RegEx: <td class=\"title\"><a href=\"(.?)\"(.?)>(.?)</a>(.?)</td>
- sprizzle 13y agoArg, there should be asterisks after every period.
- joshbaptiste 13y agoSome people, when confronted with a problem... bah you know the rest.
- kaeawc 13y ago"HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo͟ur eye͢s̸ ̛l̕ik͏e liquid pain" http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags http://stackoverflow.com/questions/1732348/regex-match-open-...
- michaelmcmillan 13y agoI am willing to sacrifice my soul and everything that is holy.
- karangoeluw 13y agoRegex to parse HTMl is probably the single worst thing you can do.
- lloeki 13y agoCrafting a wide purpose regex to parse whatever HTML comes in is bad. Building a regex to extract relevant data from simple, fixed-form page data, bypassing tags irrelevant to the problem at hand is not.
- untothebreach 13y ago...until the HTML changes. I haven't look at their parsing code, so I have no idea if it is any better than using a regex, but if the regex assumes too much, simply reordering the attributes in a tag (or something similar) could break a regex-based solution.
- addflip 13y agoI don't get why you're using a try except block for the num_comments variable. You shouldn't be casting to an int if it doesn't have the attribute.
- karangoeluw 13y agoThe meta text on any page can be this: > 21 points by johns 15 minutes ago | discuss or > 152 points by ar7hur 3 hours ago | 58 comments If the rgex matches (case 2), then I cast it to an int. Otherwise (case 1, 0 comments).
- eeadc 13y agoThat library is in many ways deprecated and broken: At first, it uses only old-style classes because it doesn't inherits object explicitly. Furthermore, it uses print in a method; it would be more "Pythonic" to return a str object, which was formatted using str.format. I think the future is Python 3, and new implementations in Python 2 syntax are simply unneccessary. I would suggest the usage of Python-3-style syntax, which is also valid in Python 2.7 (which isn't hard).
- karangoeluw 13y ago> At first, it uses only old-style classes because it doesn't inherits object explicitly. Please explain this further. > usage of Python-3-style syntax, which is also valid in Python 2.7 Will do this
- mapleoin 13y agoSee http://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes http://docs.python.org/2/reference/datamodel.html#new-style-... for the distinction
- karangoeluw 13y agoAlright. Fixed.
- gpsarakis 13y agoNice effort. Just a few remarks: - You should certainly use Requests http://docs.python-requests.org/en/latest/ http://docs.python-requests.org/en/latest/ - The Story class seems somewhat redundant. You could possibly use collections.namedtuple as a container for properties or simply a dictionary. The print_story method could just be the __str__ special method. - JSON output would be useful.
- karangoeluw 13y agoI will try and implement these. Thanks for the suggestions.
- karangoeluw 13y agoFixed.