16 ms·

Hacker News API

• chrismccoy 12y ago
  http://jsfiddle.net/chrismccoy/cuz7ugL8/ http://jsfiddle.net/chrismccoy/cuz7ugL8/ quick fiddle to dump the json
• izietto 12y ago
  Yes!!! Now I can make my HN app! I don't like the ones that are on the market now (on Android platform)
• yefim 12y ago
  The GitHub repo is inaccessible because it is private.
  • kogir 12y ago
    Oops. Thanks for letting me know!
    • tokenizerrr 12y ago
      The issue tracker on https://github.com/HackerNews/API https://github.com/HackerNews/API appears to be disabled, though the summary says "Documentation, Samples, and Issue Tracking for the Official HN API."
      • kogir 12y ago
        We decided email would be better until we got a feel for things. Updated. Examples are coming soon, but we didn't want to make people wait.
        • wvl 12y ago
          Please consider opening up the issue tracker. Other people might benefit from seeing these issues, such as the issue about the comment count that I just raised as a comment here.
• deleted 12y ago
  [deleted]
• zuck9 12y ago
  What's the Arc rendering engine? Google didn't help
  • Goopplesoft 12y ago
    http://www.paulgraham.com/arc.html http://www.paulgraham.com/arc.html
  • deleted 12y ago
    [deleted]
• minimaxir 12y ago
  How does this differ from the Algolia HN API in terms of data access? (https://hn.algolia.com/api https://hn.algolia.com/api) I was able to download all HN data recently with ease using that endpoint. Authentication? EDIT: After looking at the documentation there are two new aspects of the Firebase API not in the Algolia API: 1) Ability to see deleted/dead stories. 2) Endpoint for user data. Question to kogir/dang: Has the "delay" field (Delay in minutes between a comment's creation and its visibility to other users) always been there?
  • zuck9 12y ago
    Algolia scrapes HN HTML and provides their own API from what it looks like.
    • jaredsohn 12y ago
      They don't scrape it. From the hnsearch about page: "Every 2 minutes, the last 1000 HN items (stories, comments, polls) are sent to Algolia's indexing API. Items from the last 48 hours are refreshed every hour."
      • redox_ 12y ago
        Correct, and we're planning to move to the new official API ASAP (instead of the legacy one -> it was not web crawler but far from perfect) for the indexing. Regarding the REST APIs, let's keep both for now :)
  • deleted 12y ago
    [deleted]
  • dang 12y ago
    The 'delay' field has been there for years. I vaguely remember pg announcing it here when he added it, so we could probably dig up the exact date.
  • gwern 12y ago
    I'm also curious whether it removes some of the limitations of the Algolia version; I wanted to download my content for some statistical analysis (notes at http://www.gwern.net/HN http://www.gwern.net/HN ) but I discovered that it seems there's some hard limits to how much of my data I can reach: https://github.com/algolia/hn-search/pull/36 https://github.com/algolia/hn-search/pull/36
• bambax 12y ago
  Any kind of rate limitations we should be made aware of?
  • kevin 12y ago
    Rate limit is up to Firebase. We'll see if we can get them to answer.
    • 650REDHAIR 12y ago
      That doesn't worry you at all? Were any alternatives considered?
      • kogir 12y ago
        Trust me, their rate limit will be much higher than anything we'd easily be able to provide.
  • sararob 12y ago
    [Firebase Dev Advocate here] We don't currently have rate limits. I'd recommend using our SDKs, they handle connections more efficiently than dealing with the REST API. You can also run your own server process using our Node (https://www.npmjs.org/package/firebase https://www.npmjs.org/package/firebase) or Java libraries (https://www.firebase.com/docs/android/quickstart.html https://www.firebase.com/docs/android/quickstart.html).
    • dmunoz 12y ago
      Any kind of rate limitations with the REST API we should be made aware of?
      • kevin 12y ago
        You really want to use the Firebase SDKs. https://news.ycombinator.com/item?id=8423055 https://news.ycombinator.com/item?id=8423055
      • 12y ago
• jxm262 12y ago
  Yay! I've been wanting something like this to come out. I've been playing around with some new tech stacks and built a css replacer of hacker news, but always wanted an actual api to make it easier. http://jmaat.me/hn http://jmaat.me/hn There's a bunch of css pages that come out for hacker news, but I couldn't find anything that aggregates them. This will be alot easier to extend and customize the site. I'm not seeing any api's for the jobs or show sections though? Hopefully this might come in the future?
  • minimaxir 12y ago
    Why do you need an API to replace only CSS? You can do that with an extension like Stylish/Greasemonkey.
    • jxm262 12y ago
      Well personally, I didn't want to install any browser add-ons. I also had some other ideas, like aggregating reddit and hackernews posts, but would need to scrape for that (unless there's an external api I'm missing).
• josephwegner 12y ago
  I really appreciate giving a 3 week heads up before moving to a new frontend structure. It's a nice gesture, but I have this horrible feeling that there's only about a 10% chance that my Hacker News app gets updated in time. I know you can't not iterate because people are scraping, but it does stink. At least this will make everything more future-proof going forward. However, it may be nice to give a bit more heads up than 3 weeks. I know a lot of apps can take ~2 weeks to get through the review process for iOS.
  • mahyarm 12y ago
    Maybe they can put the new renderer on a different sub domain for a few weeks after their 3 week deadline. One to beta test the interface and two to give devs a bit more time to convert.
    • kevin 12y ago
      We'll probably start with logged out home to a percentage of traffic and work our way to updating the rest of the site and handling the full traffic over a few weeks. If people need help converting their apps, I believe the Firebase guys have offered to help. Contact us at api@ycombinator.com and we'll connect you.
      • josephwegner 12y ago
        Love the staged roll out idea. You could probably also use user agents to recognize scrapers from real users (could also be helpful for slowly rolling out to older browsers).
  • jack-r-abbit 12y ago
    Some sites actively punish scrapers by constantly, purposely changing their markup. So giving them an API and 3 week head start is leaps and bounds above and beyond what can be expected. When you operate a scraper, you are always on the defensive when it comes to site updates suddenly breaking your app. They should be so lucky to get this 3 week notice. It is all on them if they can't turn it around in time.
• tonysuper 12y ago
  HN is getting a mobile-optimized site? Best news I've heard all day!
• burntcookie90 12y ago
  Very cool, I'm glad I prepped my android app for this :D One question: why the choice of returning everything as an ID, on mobile, this will require a lot of very small network requests.
  • sararob 12y ago
    If you use the Firebase SDKs the request is sent over a reused full duplex socket rather than creating new/individual connections.
    • burntcookie90 12y ago
      Yeah, unfortunately, I'm trying to not use any closed source stuff in the app. Going to write some service layer stuff for it
• ChuckMcM 12y ago
  Oh this will be cool, and I really look forward to being able to read HN on a phone or tablet in the future without all the zooming and scrolling! It will be interesting to see if it has an impact on site traffic, how much of that traffic is scrapers today?
• angersock 12y ago
  Why do you have all the times in Unix time instead of ISO-8601 UTC? EDIT: Also, the "about" value in the users api appears to be truncated. Compare... https://hacker-news.firebaseio.com/v0/user/angersock# https://hacker-news.firebaseio.com/v0/user/angersock# ...with... https://news.ycombinator.com/user?id=angersock https://news.ycombinator.com/user?id=angersock EDIT2: Note that the JSON is correct, but the preview in the firebase API seems to be broken. EDIT3: No issue tracker on the Github? Laaaame.
  • Someone1234 12y ago
    UNIX time is actually much easier to parse and more accurate. Almost all platforms (even Windows[0]) have a way to convert a UNIX time into a culture specific local time. What you're asking for is a string you have to parse. That's a lot more work and there is a lot more that can go wrong. [0]https://stackoverflow.com/questions/249760/how-to-convert-unix-timestamp-to-datetime-and-vice-versa https://stackoverflow.com/questions/249760/how-to-convert-un...
    • angersock 12y ago
      Um, how about, no? var time = new Date("<ISO-8601 string>") // in Javascript time = DateTime.iso8601("<ISO-8601 string>") # in Ruby Aaaand it's human readable! Aaaaand it works before the Unix epoch! EDIT: Changed DateTime.parse to DateTime.iso8601, to be even more retentive. Look, parent claimed that parsing ISO strings is hard (it's not, especially if you're consuming a web api on a modern web language) and that it was more readable (which is so clearly wrong I have no words). As for being more accurate, again no. The range is worse (lol wraparound if you're using a 32-bit int), there isn't explicit support for fractional seconds, doesn't map onto UTC cleanly, doesn't handle leap seconds, and so on. It's only "easier" if you don't actually care about a human-readable timestamp that is robust and if you desire to do date parsing yourself instead of using any of the well-established libraries out there. Ugh.
      • teach 12y ago
        That's two programming languages. Not all languages have a nice ISO-8601 parser built-in. Human-readable is a slight advantage, agreed.
• kasperset 12y ago
  This is a very good news. Made my day. Thank you.
• jamest 12y ago
  [Firebase founder here] This is pretty exciting for us, we're glad kogir, dang, kevin and sctb chose to expose HN's data through Firebase. We're seen quite a few startups (and big companies like Nest) do this, since building, maintaining, and documenting a public API often isn't a easy task.
  • nacs 12y ago
    How does this API work with Firebase? Is HN data already in Firebase (as its primary data store) or is content from HN's DB getting 'mirrored/cloned' on-demand to Firebase for the API?
• dpcan 12y ago
  As soon as someone uses this API to create a replica of the site where everything else is the same, but the design is responsive, I want to know about it :)
• christiangenco 12y ago
  Oh man you guys, patio11 has generated massive amounts of content: https://hacker-news.firebaseio.com/v0/user/patio11.json?print=pretty https://hacker-news.firebaseio.com/v0/user/patio11.json?prin... I count 8,483 submissions. I'm sure there's something interesting to be done with all of this data. A word frequency chart? --- Edit: So apparently there's a ruby gem that lets you feed it a body of text and generates pseudo-random phrases based on that text. I present to you the patio11 impersonator: https://gist.github.com/christiangenco/e8d085e47479be0131e1 https://gist.github.com/christiangenco/e8d085e47479be0131e1 One of my favorites: A nice set of challenges -- kitty at a school with tens of thousands of bucks a year or less immediately. --- Also, a word count on patio11's submissions: 1,052,351. For comparison, all 7 Harry Potter books total 1,084,170 words. patio11 has written the entire Harry Potter series worth of content on HN. Just... wow.
  • minimaxir 12y ago
    In February, I did an analysis on all Hacker News stories: http://minimaxir.com/2014/02/hacking-hacker-news/ http://minimaxir.com/2014/02/hacking-hacker-news/ Yesterday, I published an analysis on all Hacker News comments: http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ There's a lot of interesting trends in the data. Let me know if you want to know anything in particular and I'll get back to you. :)
    • reduce 12y ago
      Thanks for that. Great stuff.
    • DanBC 12y ago
      It'd be interesting to see if including an URL in the body of a post has any effect on votes received.
    • danieltillett 12y ago
      I would be interested to know the average hourly cost of HN in lost productivity :)
• pbnjay 12y ago
  Great news! Can we add endpoints for "Show HN" and "Ask HN" too though? I don't see an easy way to get them.
• airlocksoftware 12y ago
  This... is cool, but also kinda sucks for me. I've invested dozens of hours into writing an extremely complicated scraper for my Android version of HN. https://play.google.com/store/apps/details?id=com.airlocksoftware.hackernews https://play.google.com/store/apps/details?id=com.airlocksof... The newest version (still under development, probably a month or two from release) adds support for displaying polls, linking to subthreads, and full write support (voting, commenting, submitting, etc). I'm fine with switching to a new API (Square's Retrofit will make it super easy to switch), but without submitting, commenting, and upvote support I have to disable a bunch of features I worked really hard on. Also it would've been cool to know this was coming about 3 months ago so I didn't waste my time. Anyways, quick question on how it works -- when I query for the list of top stories https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty https://hacker-news.firebaseio.com/v0/topstories.json?print=... it just returns a list of ids. Do I have to make a separate request for each story https://hacker-news.firebaseio.com/v0/item/8863.json?print=pretty https://hacker-news.firebaseio.com/v0/item/8863.json?print=p...) to assemble them into a list for the front page, or am I missing something?
  • dimillian 12y ago
    This is a big question for me too. It sounds like you need to fetch every id from the REST API. I need to test the iOS (and you Android) SDK.
  • thegeomaster 12y ago
    I'm also currently writing a scraper[1] for the HN frontpage (for my WIP Hacker News redesign), and while there's a limited Algolia API available, it doesn't do much good if users can't post comments, upvote etc. Same goes for the official one now. So, @anyone involved with the API project, can you give us an estimate on when will the OAuth-based user-specific API be rolled out? I'm fining with pausing my efforts until then, if it's going to be soon, in order to go a less complex and error-prone path. [1]: https://github.com/geomaster/hnop/blob/master/backend/src/hnparser.lisp https://github.com/geomaster/hnop/blob/master/backend/src/hn...
    • kogir 12y ago
• tlo 12y ago
  For my twitter bot https://twitter.com/hn_bot_top1 https://twitter.com/hn_bot_top1 I use http://api.ihackernews.com/ http://api.ihackernews.com/ at the moment. This works but the site (and the API) is quite often not available. So I'll probably switch to the new API as soon as I have some spare time...
• dabit3 12y ago
  I love firebase!
• dimillian 12y ago
  So, does it mean I can get the top stories, and then get a top story item with all the comment expanded ? I mean, at first it look like it just send me the id and I need to fetch the detail for each of them. Again, this is just looking at the REST API, not the iOS SDK for example. I'll need to "convert" SwiftHN (https://github.com/Dimillian/SwiftHN https://github.com/Dimillian/SwiftHN) either to this new API or adapt my scrapping engine to the new site layout.
  • matznerd 12y ago
    Please do convert it, I am working on a project with some of your code as a base!
    • dimillian 12y ago
      You're using the app, or the scrapping engine (Hacker Swifter)? The nice thing is that HackerSwifter public API is already in a quite finished state for the available functions, even if I switch to the API, the method calls will be the same.
      • matznerd 12y ago
        I am actually using the app because I am doing a mashup of HackerNews and another piece of software. If I decide to end up changing the UI a lot (which looks like it is trending towards), I was thinking of switching to the scraping engine. It's awesome that I won't have to change a lot if you do switch to the API. I'm following on GitHub, keep up the great work!
• BenSS 12y ago
  Great news, I've avoided touching my iPad reader because of the whole scraping issue to get at some of the data. Now I can justify updating for iOS8 + the new API. (just hope it gets approved in time)
• deleted 12y ago
  [deleted]
• Achshar 12y ago
  I always thought HN was more of a one man job. I didn't realize there is an entire team working on it (full time?). Honestly there doesn't seem to be much to work on here. It's a simple site that rarely changes (at least on the front end).
  • tdicola 12y ago
    'Simple site' that gets probably tens of thousand if not hundreds of throusand unique visitors a day.
    • deleted 12y ago
      [deleted]
      • deleted 12y ago
        [deleted]
    • Igglyboo 12y ago
      HN uses 12 lines of javascript total, it's pretty damn simple.
      • tdicola 12y ago
        And what about the backend? Last I heard it's all stored as files on a handful of machines, so it's not like it's some off the shelf database you can just keep throwing machines at to scale. Also it's written in lisp originally by one person, so have fun figuring out how it all works and ramping up to it quickly. There's also certainly a big support overhead dealing with the moderation of stories and comments. I'm not saying it should take hundreds of dedicated engineers to run the site, but I think it's silly to look at it and think you could run it in your spare time because it's just a list of stories.
  • petercooper 12y ago
    I believe the whole idea of having multiple people working on it permanently was only introduced six months to a year ago(?) And it seems like big changes are happening underneath us, mostly in terms of story quality: https://twitter.com/sama/status/519240112907894784 https://twitter.com/sama/status/519240112907894784
• Livven 12y ago
  I've been working on a Hacker News client for Windows Phone over the past several weeks and am very close to an initial release, so I feel somewhat ambivalent about this. On the one hand, of course it's great that HN is finally getting a proper API and also modernizing its markup (which is a mess even if you ignore all the tables – for example, the first paragraph in a comment usually isn't wrapped in <p> tags), but on the other hand this current v0 version is very lacking and impractical for a regular client application. Since the top stories (limited to 100) and child comments are only available as a list of IDs a client app would have to make a separate HTTP request for every single item, which is obviously not something you'd want to do especially in a mobile environment. Other lists apart from the top stories (new, show, ask, best, active etc.) don't seem to be available at all right now. Of course this is just the first version, and the documentation promises improvements over time – which I don't doubt at all – but there's no clear indication that the API will be at feature-parity with the current website, even excluding anything that requires authentication, by October 28. So this means that I – and other developers of client apps or unofficial APIs – will probably have to write new scraping code once the new rendering engine (which I assume refers to the website) arrives instead of being able to switch to the new API immediately. Now I guess I might just be needlessly worried, especially since the blog post explicitly says that the new API "should hopefully making switching your apps fairly painless", but then why not wait until it's actually ready for that before making the announcement? Putting a half-baked API out there a few days/weeks (?) in advance before it's fully fleshed out doesn't seem all that helpful, at least to me.
  • minimaxir 12y ago
    > (which is a mess even if you ignore all the tables – for example, the first paragraph in a comment usually isn't wrapped in <p> tags) Oh, that's in the actual API. https://hacker-news.firebaseio.com/v0/item/8422922.json?print=pretty https://hacker-news.firebaseio.com/v0/item/8422922.json?prin... I strongly hope they add a plain-text field for returned comments.
    • dang 12y ago
      What you're getting there is how the comment text happens to be stored (and presumably always has been). We've talked about changing that, because it would allow us to do some implementation improvements like... well, I forget just now. Might have had to do with caching. Anyhow, if enough people want it, we'll bump up the priority.
• bambax 12y ago
  Can we query new stories?
  • bambax 12y ago
    I meant the list of newest stories as on this page https://news.ycombinator.com/newest https://news.ycombinator.com/newest Any idea?
• projectileboy 12y ago
  Listening to programmers complaining about the markup used to implement HN is a perfect example of tech folks not focusing on the users.
  • NewHermes 12y ago
    Could you elaborate? I assume you are criticizing HN or the complaining programmers but I can't tell which.
  • levlandau 12y ago
    slight nitpick: i agree with you (i.e i dont think people should be complaining about the markup) but this issue is not that obvious when the "users" are the "programmers" complaining....
• ossama 12y ago
  Here's a simple example that displays the top story and votes using the Firebase JS SDK (and updates in realtime): http://jsfiddle.net/firebase/cm8ne9nh/ http://jsfiddle.net/firebase/cm8ne9nh/
  • ossama 12y ago
    and one showing all top stories updating in realtime. Obviously, in JS, but the other Firebase SDKs are similar: http://jsfiddle.net/firebase/96voj1xh/ http://jsfiddle.net/firebase/96voj1xh/
• eevilspock 12y ago
  If up/down vote data were included in the API, much needed experimentation on collaborative filtering would be made possible! This is Hacker News after all. Right now one team, Ycombinator, is trying to fix important issues in the ranking and moderation of posts and comments. Many of us are frustrated by the increasing domination of popularity (and hatred) over quality and relevance. A lot of good submissions and comments are simply buried, never to be found. There is too much muck to have to wade through. The timing of posts and comments plays a much larger role than quality. I could go on and on. Imagine a Netflix Prize-like flowering of experiments and collaboration, leveraging the hacker community's collective smarts and enthusiasm. Many of us have ideas, but right now are unable to test them. What a shame if a great idea dies on a notepad. There are two possible issues with opening up voting data: gaming and privacy. If having vote data allows someone to game the front page, then only include it with some delay (2 days?) so that it could't be used to game the front page. This will still allow experimentation with collaborative filtering algorithms and the like. My take on the privacy issue is that anonymity isn’t that important for a site like Hacker News: 1. Startup culture is about straight talk, putting your money where your mouth is, and open critical feedback, both in the giving and receiving. There are precedents for exposing voting data (e.g. Quora, Facebook, Stack Exchange). 2. HN is not aimed at political discussions or other topics where anonymity can be paramount. 3. Pseudonymity is sufficient for those who don’t want their votes and comments tied back to their actual identity. Thoughts? I would love to hear from others who yearn to experiment with alternate algorithms and strategies for improving Hacker News.
  • dang 12y ago
    There are many legitimate views on this, but FWIW mine differs from yours. I believe that anonymity actually is important for a site like Hacker News, and the odds of us ever publishing the vote data—even pseudo-anonymized—are small. Sorry to disappoint.
    • eevilspock 12y ago
      Daniel, I understand. Do you or any others at Y-Combinator have any thoughts on how the hacker community could experiment in the areas I mention above, or whether you guys even think such experimentation would be valuable?
• BigChiefSmokem 12y ago
  Does anyone have a good solution on how to get around making a separate request for each Item? Is there somewhere we can pass an array of Item Ids? Is this planned for the future? Thanks HN! EDIT: I just read the post about using the Firebase SDK to do this efficiently.
• dang 12y ago
  To everyone asking about logged-in access and write access: this is just a first release! Where it goes from here will depend, in good iterative fashion, on what people want.
  • jffry 12y ago
    I think part of the angst centers around the bit where the markup is going to change. Lots of apps for HN that scrape the UI rely upon the current site to enable things like voting and other signed-in-only actions, for which there is (currently) no first-class API way to do these things. Even if the endpoints for voting change, getting other information (i.e. which items you voted on, so the vote arrows hide) is still markup-dependent.
• kevin 12y ago
  Oh and if you guys build anything cool with the API, please let us know at api@ycombinator.com.
• me_bx 12y ago
  I hope they'll add an article search (by keyword) feature soon.
  • eevilspock 12y ago
    The API will allow others to come up with competing designs for HN search.
    • me_bx 12y ago
      Yes, but someone willing to get the HN posts about a certain topic from the API will need to keep polling the API to get all the new submissions, and filter out the content irrelevant to her purpose. Search directly from the API would be so much more convenient.
• comeonnow 12y ago
  I built a scraper around 3 years ago (been through a few usernames since then), and I've had to change it once 3 months ago because the HTML output added quotes around HTML attributes. Even though it's read only, I'll continue to use my scraper rather than the API simple because it's one request, rather than the API would require one request for the top IDs and then one call per story, so it would be 31 calls instead of just 1. Unless I'm missing something, it seems fairly poorly designed for top stories, and non existent for new stories. ------ EDIT: Looks like I missed the text about updating to a new rendering system in 3 weeks time, and to iterate designs faster to allow mobile friendly theming. Looks Like I WILL be updating to use the API
  • tomw1808 12y ago
    yeah, I just have the same problem here... and then I have basically the same question as someone mentioned below... new stories through the api? do we have to get the max-id and then get everything below the max-id and check if its a story? and other ideas?
• markrickert 12y ago
  Nice! I'll be changing my Mac system tray app HackerBar over to use this instead of some Objective-C scraping magic that it uses now: http://hackerbarapp.com/ http://hackerbarapp.com/
  • Igglyboo 12y ago
    Wow I've never heard of this but that is a really cool app, I'll definitely check it out. Do you know of anything similar for reddit?
    • markrickert 12y ago
      There's way too many reddit things already out there... I was trying to tap a market that doesn't have lots of enthusiastic software developers. oh.... wait...
• jcampbell1 12y ago
  This makes it really easy to add average karma to the comment section for every user. For instance, you can paste the below into the console, and should add average karma data for each user. Array.prototype.forEach.call(document.querySelectorAll('a[href^=user]'), function(v,k) { var s = document.createElement("script"); s.src = '//hacker-news.firebaseio.com/v0/user/' + v.innerHTML + '.json?callback=ud_' + k; document.head.appendChild(s); window['ud_' + k] = function(user_data){ var avg_karma = user_data.karma / user_data.submitted.length; v.innerHTML += ' (' + avg_karma.toFixed(1) + ')'; } } );
• kogir 12y ago
  Just pushed the ability to subscribe to all changes and new items: https://github.com/HackerNews/API/blob/master/README.md#live-data https://github.com/HackerNews/API/blob/master/README.md#live...
• IgorPartola 12y ago
  I am excited about this because now I can finally build a way to query my own posts and comments. I often times come across products/services/cool hacks on here that I vaguely remember, but cannot always locate. Using built-in search is kludgy, and I'd like to be able to do something more complex. Thanks for doing this!
  • DanBC 12y ago
    It's interesting that you mention the phenomenom of "I remember seeing it (or maybe even saying it) but I cannot find it". This has happened to me a lot. It is intensely frustrating to me because it's blocking - I don't get much done until I re-find whatever it was. I wonder how many other people have this? And what their techniques are?
    • terhechte 12y ago
      I have the same problem, and I think the best solution would be to index my web browser search history and be able to search through that index. However that would probably generate a lot of junk. My current solution is to bookmark anything that's vaguely interesting. And when I remember something, I search through the bookmarks. I never look at them, they're just for keeping the stack of stuff to search through.
      • raldu 12y ago
        I am using the same approach too. However, most of the old stories I had liked at HN were not bookmarked, so I had to struggle writing a scaper for them. It would be very cool to be able to authenticate and grab all my saved stories and the comments for archiving and fun!
• davexunit 12y ago
  Glad to see that HN will still be written in a lisp after these major upgrades.
  • dang 12y ago
    I regard that as part of the site's DNA. It permeates so many things about its design, both internal and external.
• lnanek2 12y ago
  Disappointing they are still going to use Arc even with the update to modern HTML. I despise how with Arc if I leave the page open during the day and try to click a link later the link has expired. They should really update to a modern web framework at the same time. Big modern frameworks like Rails are making ridiculously awesome improvements like replacing page loads with XHR (quicker loading since JS/styles/etc is all loaded already, no screen flash, etc.) in a progressive enhancements manner. So Arc can't even generate fully functional links, let alone keep up with modern web advancements.
  • danielhughes 12y ago
    What is it about Arc that causes the unexpired link issue? I know very little about Arc. But I understand it to be a language whereas the issue you describe appears to be the function of some implementation decision. Can you please elaborate?
    • dang 12y ago
      (I'll take a crack at explaining this momentarily...) Good grief, that took a long time. Here you go: http://pastebin.com/bSW5dfRQ http://pastebin.com/bSW5dfRQ [1]. I'd better stop neglecting my duties now! Edit: One thing I forgot to put in there: one reason the closure technique is powerful is that you're leveraging the programming language and runtime to do most of the book-keeping for you. Whatever data is handy, you just reference. The system will remember all the references. That's why using things like query strings and hidden form fields is more complicated: you have to handle all those details yourself (not to mention serialize and deserialize them if you're passing through any other format than what your program keeps in memory). That is tedious, and when your app has many kinds of request, the complexity quickly piles up. Of course there are other abstractions you can build over this, but closures are an elegant one—especially in cases where programming simplicity is more important than scalability, which is most cases. Edit 2: A few people thought this should be its own post, so I made https://news.ycombinator.com/item?id=8425011 https://news.ycombinator.com/item?id=8425011. [1] Originally http://pastebin.com/dETyYtpX http://pastebin.com/dETyYtpX, but I added the above bit etc.
  • 12y ago
• awwstn 12y ago
  This is so awesome – albeit a bit overdue ;) I'm going to start diving into the API to build a simple, powerful "Google Alerts for HN" app on Assembly, and I'd love help from anyone who's interested: https://assembly.com/hn-monitor https://assembly.com/hn-monitor There are some products like this out there, but they had to rely on scrapers and the HNSearch API, so I've always found them to be spotty. I think we can make something better.
• oftenwrong 12y ago
  I have been working on a toy project that uses the Algolia API. Maybe I'll switch it to this now.
• EGreg 12y ago
  This is excellent news, and allows me to make an app for YCombinator using our social app platform! This should really energize the HN community.
• danieldisu 12y ago
  Can't wait to see all the apps that will be built with this, now that you don't have to scrape the horrible HN HTML!
• hardwaresofton 12y ago
  "Let there be apps". Anyone with a Firefox phone that wants to work together on a HN client for it? FFOS has been lacking a good HN app.
• calebm 12y ago
  Why could you not have released it 1 day sooner? :) I just wrote a scraper: https://github.com/calebmadrigal/hn-tracker https://github.com/calebmadrigal/hn-tracker
• calebm 12y ago
  This story via the API: https://hacker-news.firebaseio.com/v0/item/8422599.json?print=pretty https://hacker-news.firebaseio.com/v0/item/8422599.json?prin...
• calebm 12y ago
  How can I find the current rank of a story via the API?
  • calebm 12y ago
    Nvm, I see: https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty https://hacker-news.firebaseio.com/v0/topstories.json?print=...
• andrethegiant 12y ago
  Any plan on having another field on the items route that can be queried besides id, e.g. url?
• kolev 12y ago
  At last! A hacker-friendly Hacker News! Yay!
• jaredsohn 12y ago
  Here is something I built with the Algolia API awhile back and just haven't gotten around to cleaning it up to post here. It lets you download all comments/stories for a user as a JSON or CSV file, breaks down karma between comments and stories, and plots comment/story counts, karma, etc. over time on a line chart (clicking will show you the details via an hnsearch). Also I built some npm modules so you can get this information via the commandline. http://hnuser.herokuapp.com/ http://hnuser.herokuapp.com/. Example: http://hnuser.herokuapp.com/user/tptacek/ http://hnuser.herokuapp.com/user/tptacek/ The Chrome extension hasn't been updated for awhile (it just superimposes a small amount of this information on the user page).
• deleted 12y ago
  [deleted]
• bennyg 12y ago
  I've been waiting forever for an API from HN, but unfortunately I will not be using it for my app (https://github.com/bennyguitar/News-YC---iPhone https://github.com/bennyguitar/News-YC---iPhone). I've built a library for iOS (https://github.com/bennyguitar/libHN https://github.com/bennyguitar/libHN) that handles scraping, commenting, submitting, voting, etc pretty well and allows me to make as few web calls as necessary to use HN. It looks like I'd have to drop functionality and completely change the networking scheme to match this API - something I'm not willing to do yet. Correct me if I'm wrong here, but to get every comment on a post, I'd have to recursively get each item for each child. Instead, right now, I can make one network request and get all comments for a story. Granted, I have to parse the HTML (which I hate), but it's a much cleaner solution than going through every item, checking the children and then getting those items ad infinitum. Again, I just glanced over the documentation, but that seems untenable to me.
  • kogir 12y ago
    If you use the Firebase iOS library they handle it all and use an efficient bidirectional streaming protocol.
    • bennyg 12y ago
      That's good to know - wasn't super clear on my first pass of the documentation.
  • dailen 12y ago
    Check the article btw "Most importantly, the reason we released an API is so that we can start modernizing the markup on Hacker News. Because there are a lot of apps and projects out there that rely on scraping the site to access the data inside it, we decided it would be best to release a proper API and give everyone time to convert their code before we launch any new HTML."
    • bennyg 12y ago
      Yeah, the only problem is that I don't want to cut major functionality just to use their API. Things I do, that it doesn't look like the API handles: - More than 100 stories - Best, Top, Ask, ShowHN, Jobs, User Submission Posts - User Management (logging in/out) - Commenting - Submitting - Voting My app doesn't just function as a reader, which is what this API seems geared towards with the v0 release, it functions as about as full-fledged of an HN client as you could get. There's a couple things that I haven't built in yet like changing your about me text, but those were on the roadmap. I'm actually thinking about storing the configuration of how my app scrapes online such that if the HTML markup changes, I won't have to push huge sweeping changes to the App Store to get my app online again. I just deploy to Heroku and the app will handle that configuration and scrape correctly sans pushing to Apple.
• dstaley 12y ago
  Decided to recreate the Hacker News homepage using Ember and the new API. I was really pleased with how easy it was! https://realtimehackernews.firebaseapp.com/ https://realtimehackernews.firebaseapp.com/
  • tlarkworthy 12y ago
    that is really cool seeing the numbers change (occasionally)
  • rkrzr 12y ago
    Very cool, but why not update the design a bit, while you're at it?
    • SiVal 12y ago
      The API was just announced today. Maybe give him until tomorrow to reinvent Hacker News. ;-)
  • ryanseys 12y ago
    Very cool. Would be neat if changes were more pronounced when they happen. Will the posts change order in this demo if they change on the homepage?
    • dstaley 12y ago
      Sadly no, but I'm looking into how to make that happen! It's a result of my particular implementation, not a fault of the API.
  • deleted 12y ago
    [deleted]
  • dang 12y ago
    Holy crap, that was fast. Impressive!
  • d0ugie 12y ago
    dstaley nice work. Would be nicer still if firebase had SPDY and QUIC lit up. I don't think it would be a problem with Apache already in front of Varnish. But more importantly, why no favicon? ;)
• wvl 12y ago
  Would it be possible to cache the number of comments a story has? Or am I wrong in my understanding that the only way to find the number of comments a story has is to walk the tree of child items and maintain my own count?
  • calebm 12y ago
    +1
• deleted 12y ago
  [deleted]
• ryanseys 12y ago
  Smart move using Firebase. This instantly gives developers traction with the API and as a big API client guy, I love making clients but also very happy we'll have all the tools we need to get started using the API right off the bat. Considering Firebase doesn't have rate-limiting too, this means that the things people can build with this API are limitless.
• wslh 12y ago
  I would love a personal API too where I can download my own data (i.e: saved articles).
• bch 12y ago
  Does this encoding: ' have a name ? I can't actually find anything useful on the 'net.
  • tlb 12y ago
    It's part of HTML. https://en.wikipedia.org/wiki/Character_encodings_in_HTML https://en.wikipedia.org/wiki/Character_encodings_in_HTML
    • bch 12y ago
      Thanks. I'm trying to get cURL to decode, but it doesn't seem to natively handle. Now I'm digging into .../escape.c I feel like I must be missing something... :/
      • tlb 12y ago
        You can't simply decode each character without losing information. For example, &#x3c; means a literal < character to be shown on the page, as opposed to a < in the stream which starts an HTML tag. If you're just planning on displaying the text in a browser, no decoding is needed. If you want to parse the text to do some sort of textual analysis, an HTML parser library might be best.
        • bch 12y ago
          I understand what you're talking about re: &#x3c; and '<' -- the json -looks- page (terminal in my case) displayable, barring the &#xhhhh; encoding. cURL has facilities for decoding %20 (for example), but not what we're getting back w/ this json. You've given me an idea though, so back to vi for me. Thx.
          • mh- 12y ago
            not sure if you figured something out already, but just saw your comment and remembered that this exists in PHP: http://us1.php.net/manual/en/function.get-html-translation-table.php http://us1.php.net/manual/en/function.get-html-translation-t... absent another source, you could dump it out for your usage elsewhere. % php -r 'print_r(get_html_translation_table(HTML_ENTITIES, ENT_QUOTES|ENT_HTML5));' or % php -r 'print json_encode(get_html_translation_table(HTML_ENTITIES, ENT_QUOTES|ENT_HTML5));' | jq . edit: just found http://dev.w3.org/html5/html-author/charref http://dev.w3.org/html5/html-author/charref (but might be harder to parse..)
• stevekemp 12y ago
  Until now I've been using the RSS feed to write a Maildir archive of stories. It will be nice to write that directly via the API :)
• jgill 12y ago
  A very welcome addition.
• deutronium 12y ago
  I'm just wondering if you'd consider releasing a regular data dump, possibly via bittorrent. As I'd like to do some analysis, but don't really want to thrash the API, downloading all data.
• godgod 12y ago
  Bad idea. We don't need another process to slow hacker news down anymore. It's already operating at crawl speed. Why the hell is this site so slow?
• yawniek 12y ago
  is there a simple way to stream all the updates or at least the new submissions/deletes by just subscribing to a Websocket?
• shill 12y ago
  I wrote a stupid simple wrapper and pushed it to PyPI. My excuse is that I needed to learn how to use setuptools today. pip install hackernews-python Usage: >>> from hackernews import HackerNews >>> hn = HackerNews() >>> hn.top_stories() [8422599, 8422087, 8422928, 8422581, 8423825... >>> hn.user('pg') {'delay': 2, 'id': 'pg', 'submitted': [7494555, 7494520, 749411... >>> hn.item(7494555)['title']) Hacker News API >>> hn.max_item() 8424314 >>> hn.updates() {'items': [8423690, 8424315, 8424299...], 'profiles': ['exampleuser',...]} https://github.com/abrinsmead/hackernews-python https://github.com/abrinsmead/hackernews-python
  • BaryonBundle 12y ago
    Can't get it to work, e.g., for `hn.top_stories()` I get: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "hackernews.py", line 23, in top_stories return r.json() TypeError: 'list' object is not callable Tried in both python 2.7.3 and python 3.2.3 EDIT: You need a relatively new version of requests for this to work. The version packaged with Debian Wheezy is too old. Use pip.
    • shill 12y ago
      Thanks for the feedback! Prior to version 1.0.0 response.json() was a property, and was not callable. That's probably what caused your error. I'll add a minimum version to requirements.txt for the folks that install this package manually.
• thekingshorses 12y ago
  I added upvoting stories and comments feature to my ios/android app few weeks ago. And thinking about adding commenting. http://hn.premii.com/about http://hn.premii.com/about Now I will have to remove that. Please keep current version of the YC site on a different subdomain.
  • Livven 12y ago
    I use your app, it's awesome! Also agree with your request, as I said elsewhere in this thread... By the way, animations (the slide between list and comments) have been jerky since a recent update. I'm using the Android app on an HTC One M7 with the latest stock ROM (4.4.3 and Sense 6.0). A few other people have mentioned this in Play Store reviews. Is this a known issue?
• quesera 12y ago
  Great news! Will require a fair bit of retooling, but, ya know -- omelets, eggs, etc! I see the topstories query to replace scraping /news: https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty https://hacker-news.firebaseio.com/v0/topstories.json?print=... And there must be a query to get the newest items instead of the current /newest. Are there also new equivalents for /active, /best, /classic, /show, and /shownew? I'll be happy to replace dodgy XPATH parsing with a proper API. Hope we won't lose these other views though!
• elwell 12y ago
  > the reason we released an API is so that we can start modernizing the markup on Hacker News This is bigger news. No more tables!
• fosk 12y ago
  If you guys are already using Mashape, it has been released as a community API at https://www.mashape.com/community/hacker-news https://www.mashape.com/community/hacker-news
• na85 12y ago
  Please please please no javascript in the next HN iteration.
  • hmsimha 12y ago
    Do you want to have to reload the page every time you upvote someone?
  • krapp 12y ago
    I hope they redo it all in AngularJS just to hear people's heads explode.
• xtrumanx 12y ago
  Perhaps keep the old site still running on a different URL so scrapers who can't get their act together in 3 weeks can just change the URL they're scraping from.
• oldboyFX 12y ago
  Ummm I have a question. Why is so much preparation necessary to redesign like 3 simple templates. Shouldn't it take like ~10 hours for one person to do this? I'm talking about the front end. Why wasn't hacker news optimized for mobile a long time ago? :|
• astevens 12y ago
  I whipped up a quick test in Ruby https://gist.github.com/astevens/627756eafbc6ca07e3dd https://gist.github.com/astevens/627756eafbc6ca07e3dd
• induscreep 12y ago
  Finally, an AlienBlue caliber app for HN is now possible. Soon.
  • twodayslate 12y ago
    news:yc is the best I have found. Just wish it has collapsable comments.
• hrjet 12y ago
  On a related tangent, since you mention modernizing the markup, can we please have markdown support in comments?
• paulsutter 12y ago
  Please do allow other sites to use HN logins. Then the community could develop useful sister services. For example, a site where HN members can upvote and rate different development tools, libraries, IDEs, management tools, etc. All with backlinks to HN discussions. It's a great community and there are many ways we could share knowledge and experience.
• jtwebman 12y ago
  Nice time to makes some awesome data charts like most common subjects, etc...
• stevenspasbo 12y ago
  I spent about 20 minutes throwing this together, so it's VERY rough, but maybe it will turn into something useful. I'm primarily a java programmer but I've been wanting to teach myself more ruby, so here it goes. https://github.com/stevenspasbo/Hackernews https://github.com/stevenspasbo/Hackernews EDIT: I should say, I started on this to create word clouds, but if anyone has any ideas, contribute!
• sinwave 12y ago
  It would be fun to whoop up an LDA analysis of this data.
• ssorallen 12y ago
  The Firebase JavaScript library makes make this impressively straightforward to use. I built a clone using React.js and Firebase's library. Because v0 of the API requires a request for each news story, it's not possible to use Firebase's React mixin yet. https://github.com/ssorallen/hackernews-react https://github.com/ssorallen/hackernews-react
  • insin 12y ago
    Here's another React version which also does comments (and allows you to fold them - I needed to write a userscript for that before!). It's using react-router to switch between top stories, comments, individual comment and user profile pages. http://insin.github.io/react-hn http://insin.github.io/react-hn I've just gone for it with Firebase's React mixin, binding everything as an object, since their devs in this thread don't seem concerned about rate limiting. The mixin seems to throw an error every time I try to unbind, which I'm just catching and logging for now. Edit: I just watched this comment pop up live in my version - pretty neat :)
    • ssorallen 12y ago
      Nicely done! Binding every HN story as an object would have been the simplest approach, that makes sense. Is the source available somewhere?
      • insin 12y ago
        Original source is here until I set up a proper repo (after modularising and setting up a build process): https://github.com/insin/insin.github.com/blob/master/react-hn/app.jsx https://github.com/insin/insin.github.com/blob/master/react-...
  • sararob 12y ago
    [Firebase Dev Advocate] This is great, thank you for sharing!
• andreash 12y ago
  Give me a HN trend chart for the words "pivot" and "mvp" please:)
• x201s 12y ago
  Glad to see there is an API. What about the ability to delete accounts? Why do I have to beg to delete an account?
• aetherspawn 12y ago
  I managed to take a picture when this got 1337 upvotes. https://hostr.co/file/cHCoBCgwsjEc/2.PNG https://hostr.co/file/cHCoBCgwsjEc/2.PNG
• r0nin 12y ago
  I started a small package for golang here: https://github.com/cryptix/gohn https://github.com/cryptix/gohn
• s9w 12y ago
  I welcome the idea, but this barely qualifies as an API. The most useful part is the "current top stories" - but what timeframe exactly? Seems to be over 3 days at least and can't be customized. And even my test parsing of the 100 top stories took a good minute. And that returns only the ids, nothing else. To get basic information like the score, title or url you have to lookup the ids individually. And even the story items do not contain such basic information as the number of comments. And you can't calculate it yourself since only the top comments are even returned (as ids of course). So you'll have to recursively dig through the comments to get the number. This is even more curious as there is a very solid Algolia API where you can filter for submission time, story score, number of comments and even return a greater number of results + access page numbers to get even more. To get the information of a single algolia api call you will need hundreds or thousands (in case of nested comments) "official" API calls. Hoping for updates
• bajsejohannes 12y ago
  Awesome! On the top of my wishlist: Look up HN story by URL. (From the comments, I learned about a different API that will suffice for now: https://hn.algolia.com/api/v1/search?query=google.com&restrictSearchableAttributes=url https://hn.algolia.com/api/v1/search?query=google.com&restri... )
• jingwen 12y ago
  Wrote a simple ruby wrapper for this. https://github.com/infinitus/hnrb https://github.com/infinitus/hnrb
• so_ 12y ago
  I think quality of the v0 API is not so good for the following reasons. 1. Major functions are dropped (best, new, job, ask,...) 2. Useless response schema of the top stories API (should learn from Netflix's internal/public API design) 3. A short transition period It's very hard to provide same responses to apps.
• andrewstuart2 12y ago
  I'm definitely excited about the API and the future possibilities with it. Looks like a great start. I do have a few questions and suggestions, though. Is there any chance of getting more than just the top 100 stories returned? I think it will be a lot more useful for api consumers if you can use a query parameter to set the limit (within reason, usually 1,000) and a number of results to skip. For now, scraping is still more desirable to me since I can retrieve any number of results in their current order. Better yet, but more complex: a number to skip and a certain timestamp so I don't see the same article on two pages due to natural upvoting, downvoting, or rank decay. Also, if there's any flexibility still with property names, I'd suggest these changes for clearer semantics: "deleted" -> "hidden" (since they're obviously not deleted) "by" -> "author" (for more clarity) "kids" -> "children" (the common convention)
• Lekoaf 12y ago
  Chiming in here and presenting my try at a Hacker News website using Angular JS and Bootstrap. https://github.com/lekoaf/HackerNews https://github.com/lekoaf/HackerNews My only problem with the API is that you need to do an awful lot of Ajax calls just to get something out of it. The topstories endpoint just gives you an array of IDs and then you need to do one Ajax call for each ID to get the story. Oh well, the site is a work in progress. Not done yet.
• bonzoq 12y ago
  If someone would be interested in contributing to an OSS project to build an iOS HN client, please have a look at https://github.com/bonzoq/hniosreader https://github.com/bonzoq/hniosreader.
  • LazyGhost 12y ago
    OSS under what license? I don't see anything in a quick look over the repo.
    • bonzoq 12y ago
      MIT. Added it to readme now. Thanks.
• tsenkov 12y ago
  Are there API methods for New, Show, Ask, Jobs and Comments page items? Or should I be picking them up from "/v0/topstories"?
• bibonix 12y ago
  What about search? Would be great to have an ability to search by URL or title fragments
• zx33 12y ago
  Is there a way we can get all the submissions for a month or a date range?
• benhamner 12y ago
  Suggestions for improving the API, to make it more valuable for data mining and analytics. This assumes more historical data is available. 1. Provide a way to bulk download the data (that's a click, instead of scraping the API) 2. Add a field for the maximum position a story reached on the front page 3. Add the numerical score for the comment (at least on comments that are N days old, which won't interfere with the reason to hide the scores on the main comments) Some other changes that would be awesome (but are less realistic) include: 4. A historical event log of votes (even better would be relating those votes back to users, but I imagine that's not going to happen for privacy reasons. An intermediate possibility would be a vote log connected back to anonymized user ids, assuming the anonymous id -> real user id mapping is difficult) 5. A historical event log of display position changes for stories & comments 6. An event log of pageviews with as much metadata as possible to release without infringing on privacy