4 ms·

TL,DR: + Use content negotiation headers instead of explicit content extensions for resources. + Don't pass auth tokens as part of the URL (you monster). + D

TL,DR: + Use content negotiation headers instead of explicit content extensions for resources. + Don't pass auth tokens as part of the URL (you monster). + Don't have onerous processes for obtaining API keys. + Web scraping is totally a legit way of providing programmatic access to data. ~ Sadly, the author is kind of wrong in these cases. First, as I've run into on some of my own projects, specifying desired content type (.html, .csv, .json) in the URL is actually pretty handy. In Rails, for example, you just you a respond-to-format block. This lets clients using dumb web browsers (and you'd be surprised how many of those there are) download easily the type of content they want. Accept headers are useful, but they don't solve everything. Second, I do agree that auth tokens should go in the header--that's just reasonable. If I'm doing something that needs an auth token, I probably am curl'ing, and so I can easily set headers. Third, keys are a necessary evil. They are the least annoying way to track access and handle authorization. That said, it shouldn't be awful to get a hold of one--in our previous startup, api keys were similar to auth tokens, and that worked out fine. Fourth, web-scraping is not a good solution. "Herf derf just have your dev scrape the thing" is cool and all, but if the document is not marked-up in a friendly way, that information can be very brittle. Moreover, you run the risk of having cosmetic changes break scrapers silently. It's far better just to expose a machine-friendly API (which is handy for testing and monitoring anyways) and let your frontend devs do whatever wacky stuff they want in the name of UX. EDIT: I am all for rate-limiting as a basic step where keys do not suffice. As for scraping, the article is a bit weird on this point. The author's insistence on "DONT USE APIS EVER RAWR" and then on "hey, let's use application/json to provide documents under the same paths for machines" is goofy. It's like they don't want you to use an API, except when they do. The wording and phrasing just really gets in the way of the article--had the tone been a bit less hyperbolic, it would've been a decent "This is why I find web APIs frustrating to work with" with examples. EDIT EDIT: The author is a Semantic Web wonk. That explains it.

koide 13y ago
Counterpoint to First: What is comfortable to you as a Rails dev is immaterial to what's the better solution, so that's a moot argument. My take on that particular issue is that you accept headers, offering them as the supported way of interaction, but extensions or other content-type declaring suffixes as convenience Counterpoint to Third and Fourth: Scraping was mentioned as to why keys are not necessary, not as a real solution. If you can scrape the content without a key, why should you need a key to get the JSON version? Nonsense. If you are worried about misuse and abuse, you should put in place throttling and other countermeasures regardless of it's standard HTML version or JSON.
- angersock 13y ago
  To your first point, was just offering an example from Rails--I'd do something different in Node or Laravel. I think that you need to handle both, ultimately, and you need to document which takes precedence (your request was for '/cats.html', but content-type was 'application/json'...what do?). That said, the author seemed to dismiss suffixes out of hand. The author fairly explicitly seemed to suggest that writing a RESTy API was a fool's errand, and that you could simply emulate it by scraping content. The key stuff was a red-herring, honestly--and again, I do agree that you should be doing rate-limiting unless, perhaps, a valid auth token is supplied with the requests. The author's screed here was poorly titled, and the sound arguments one could make against REST APIs are not present.
  - koide 13y ago
    The author is indeed suggesting writing RESTy APIs are fool's errands, where the operative word is "RESTy" (ie, misinterpreted REST, which is the most common REST implementation, sadly) I mostly agree with that author's sentiment. I disagree with your interpretation that the author is suggesting scraping as a proper alternative to an actual REST (not RESTy) API. Barring that, I tend to think you, the author and myself have more agreements than disagreements on the subject at hand.
    - angersock 13y ago
      Barring that, I tend to think you, the author and myself have more agreements than disagreements on the subject at hand. Indeed, sir or madam. Good luck with your projects!
  - smizell 13y ago
    Regarding content negotiation, the idea is to put all of the media types in the Accept header that the client can understand (instead of saying I want such and such representation). You then request a URI and the server decides which representation of that resource to return in the response based on those headers. I think you may have mentioned most of this, but I reiterate to say that if a client cannot do this, it's to the fault of the client. The web was built to work with content negotiation and it does this beautifully. With that said, there is nothing wrong with linking to the HTML file with .html, but the developer (or user) consuming the API (or website) should not be adding suffixes to get the representation they want... the API should be returning those URLs in the response. Additionally, regarding screen scraping, HTML coupled with something like RDF, Schema.org, or other microformats/microdata allows you to parse the HTML looking for that data without breaking clients on HTML changes. It really is a good media type for REST, which does seem odd at first and does feels like screen scraping, but you're just parsing the HTML like you would XML. The difference is HTML is a hypermedia type and XML knows nothing about hypertext.
sktrdie 13y ago
You're missing the whole point. He's not proposing "Web scraping" for data at all. He's proposing to use a single unique identifier for a resource, its URL. This identifier should provide both the presentation, and the data. So you wouldn't "scrape" this URL for its data, you would simply content negotiate a machine-friendly version and use that instead. He's essentially saying that instead of having both http://foo/myresource http://foo/myresource and http://foo/api/myresource.json http://foo/api/myresource.json, it's much more beneficial to have a single identifier for that resource, not two. I'd even go further and say that a resource should also be linked to other data. This is the idea behind Linked Data which is essentially building a distributed database of datasets using this exact technique. This is why unique identifiers are important, because then people can reuse them and link to them. I'd suggest to look at Tim Berners-Lee presentation on this subject: http://www.youtube.com/watch?v=OM6XIICm_qo http://www.youtube.com/watch?v=OM6XIICm_qo And also perhaps a small presentation I did to explain this very specific subject: http://www.genebanks.org/tutorials/the-power-of-linked-data http://www.genebanks.org/tutorials/the-power-of-linked-data
martindale 13y ago
How do you reconcile the idealism of the web as we all _want_ it to be with the pragmatism of the web as we know it to _be_?

koide 13y ago

Counterpoint to First: What is comfortable to you as a Rails dev is immaterial to what's the better solution, so that's a moot argument. My take on that particular issue is that you accept headers, offering them as the supported way of interaction, but extensions or other content-type declaring suffixes as convenience Counterpoint to Third and Fourth: Scraping was mentioned as to why keys are not necessary, not as a real solution. If you can scrape the content without a key, why should you need a key to get the JSON version? Nonsense. If you are worried about misuse and abuse, you should put in place throttling and other countermeasures regardless of it's standard HTML version or JSON.

angersock 13y ago
To your first point, was just offering an example from Rails--I'd do something different in Node or Laravel. I think that you need to handle both, ultimately, and you need to document which takes precedence (your request was for '/cats.html', but content-type was 'application/json'...what do?). That said, the author seemed to dismiss suffixes out of hand. The author fairly explicitly seemed to suggest that writing a RESTy API was a fool's errand, and that you could simply emulate it by scraping content. The key stuff was a red-herring, honestly--and again, I do agree that you should be doing rate-limiting unless, perhaps, a valid auth token is supplied with the requests. The author's screed here was poorly titled, and the sound arguments one could make against REST APIs are not present.
- koide 13y ago
  The author is indeed suggesting writing RESTy APIs are fool's errands, where the operative word is "RESTy" (ie, misinterpreted REST, which is the most common REST implementation, sadly) I mostly agree with that author's sentiment. I disagree with your interpretation that the author is suggesting scraping as a proper alternative to an actual REST (not RESTy) API. Barring that, I tend to think you, the author and myself have more agreements than disagreements on the subject at hand.
  - angersock 13y ago
    Barring that, I tend to think you, the author and myself have more agreements than disagreements on the subject at hand. Indeed, sir or madam. Good luck with your projects!
- smizell 13y ago
  Regarding content negotiation, the idea is to put all of the media types in the Accept header that the client can understand (instead of saying I want such and such representation). You then request a URI and the server decides which representation of that resource to return in the response based on those headers. I think you may have mentioned most of this, but I reiterate to say that if a client cannot do this, it's to the fault of the client. The web was built to work with content negotiation and it does this beautifully. With that said, there is nothing wrong with linking to the HTML file with .html, but the developer (or user) consuming the API (or website) should not be adding suffixes to get the representation they want... the API should be returning those URLs in the response. Additionally, regarding screen scraping, HTML coupled with something like RDF, Schema.org, or other microformats/microdata allows you to parse the HTML looking for that data without breaking clients on HTML changes. It really is a good media type for REST, which does seem odd at first and does feels like screen scraping, but you're just parsing the HTML like you would XML. The difference is HTML is a hypermedia type and XML knows nothing about hypertext.

sktrdie 13y ago

You're missing the whole point. He's not proposing "Web scraping" for data at all. He's proposing to use a single unique identifier for a resource, its URL. This identifier should provide both the presentation, and the data. So you wouldn't "scrape" this URL for its data, you would simply content negotiate a machine-friendly version and use that instead. He's essentially saying that instead of having both http://foo/myresource http://foo/myresource and http://foo/api/myresource.json http://foo/api/myresource.json, it's much more beneficial to have a single identifier for that resource, not two. I'd even go further and say that a resource should also be linked to other data. This is the idea behind Linked Data which is essentially building a distributed database of datasets using this exact technique. This is why unique identifiers are important, because then people can reuse them and link to them. I'd suggest to look at Tim Berners-Lee presentation on this subject: http://www.youtube.com/watch?v=OM6XIICm_qo http://www.youtube.com/watch?v=OM6XIICm_qo And also perhaps a small presentation I did to explain this very specific subject: http://www.genebanks.org/tutorials/the-power-of-linked-data http://www.genebanks.org/tutorials/the-power-of-linked-data

martindale 13y ago

How do you reconcile the idealism of the web as we all _want_ it to be with the pragmatism of the web as we know it to _be_?