11 ms·

Why I tend not to use content negotiation

272 points by TheBigRoomXXL 3y ago

esjeon 3y ago
So, this is actually about: Why I Tend Not To Use Content Negotiation (to serve both HTML and JSON data from the same endpoints). The author also suggests: > The alternative is to ... splitting your APIs. This means providing different paths (or sub-domains, or whatever) for your JSON API and your hypermedia (HTML) API. I believe the alternative has been the norm actually. For example, many front-end frameworks encode UI states in URL, and it's not so sustainable to keep the alignment b/w UI states and data APIs in the long term.
whalesalad 3y ago
I think content negotiation is great when your usecase supports it, like asking for XML or JSON. Also the mappings of content types should be well defined for all to see and edit. Rails is kinda a labyrinth in that regard. I do tend to prefer actual file extensions though. Friendlier for humans (curl https://endpoint/item.json https://endpoint/item.json vs curl -H "accept: application/json" https://endpoint/item https://endpoint/item), and it is visible in logging infrastructure, sharable, etc.
- qwertox 3y ago
  For me it has always been a part of the query parameter http://endpoint/item?format=json http://endpoint/item?format=json or ?type=json I never ever had a problem with that. The only reason to use a file extension would be if the request would take no query parameters. It would never occur to me to use header information for this.
  - dgb23 3y ago
    Both file extensions and query parameters are functionally very similar. What I don’t necessarily like about query params for this, is that their usage implies filtering. It’s a cultural assumption but it’s there. Another point: you might actually have a literal file at a similar path that you serve, typically when you cache responses (managed cache or straight from your app server). Using file extensions gives you a bit more natural affordances. It’s just overal less clunky.
- FullstakBlogger 3y ago
  Content negotiation is "friendlier to humans" in the sense that you can serve the right content to different clients using the same URL, transparent to the user. If the URL itself is never meant to be shared by the user, then I don't see the point. Say, an RSS feed being served as a formatted and styled page to a browser, or a client that accepts the usual XML.
  - None4U 3y ago
    That's what XSLT is for though, the display layer should be your browser
- IanCal 3y ago
  Something nice about accept headers is you can have your own formats, and include versioning information. That means I can have a consumer specify it can process V2 and V3 formats of the API, and when the API is upgraded or downgraded around a release everything works smoothly. Your API can serve both but it doesn't have to - it can be easier to have clients support two versions than support long running parallel API versions. I sometimes prefer file extensions, but then consuming code can often get filled with appending or removing or changing formats. Broadly, I like content negotiation as a concept because it describes what you actually want and is much more typed (as far as it can be). Adding file extensions feels very stringly typed processing. Both have problems, you pick your poison and look wistfully at the greener grass on the other side.
- conor- 3y ago
  Your example of http://endpoint/item.json http://endpoint/item.json to me would imply that this is a static file- item.json -being served from the web server; whereas the explicit content-negotiated endpoint implies I'm hitting the endpoint item which will dynamically provide me a computed list of items.
simonw 3y ago
Cloudflare doesn't obey the Accept header on anything other than image content types. This means if you have an endpoint that returns HTML or JSON depending on the requested content type and you try to serve it from behind Cloudflare you risk serving cached JSON to HTML user agents or vice-versa. I dropped the idea of supporting content negotiation from my Datasette project because of this. (And because I personally don't like that kind of endpoint - I want to be able to know if a specific URL is going to return JSON or HTML).
- ForkMeOnTinder 3y ago
  That's even with the proper Vary header, right? Seems like a cloudflare bug if that's true. Maybe even a security bug if you find a service that supports text/plain.
  - zoogeny 3y ago
    I haven't dealt with Cloudflare specifically, but I did deal with a number of big CDNs for large amounts of traffic. They were pretty adamant about NOT supporting arbitrary Vary header values. It broke some logic on a few of our systems and we eventually just decided to work around it instead of pushing our case. Interestingly, one of the big CDN providers did have controls in their UI for explicitly allowing/disallowing Vary header entries but they disabled it for us at some point (e.g. it was still in the UI but greyed out). I assumed once we hit a certain level of traffic it was too computationally expensive? Ever since, I've avoided any kind of fancy header/response variance in APIs just in case I end up in the same situation. It is rarely a necessity. IIRC, the only thing they continued to support variance wise was gzip (e.g. content-encoding). It's also worth noting they were extremely conservative with query parameters too. Also to reiterate, this was very high traffic and high volume with expectations of low latency, so probably not applicable to most people using CDNs for static website assets.
    - derefr 3y ago
      > I assumed once we hit a certain level of traffic it was too computationally expensive? Seems strange; AFAIK in e.g. Varnish, Vary just means you get more "stuff" tacked onto the buffer that gets built from the request and then hashed to create the cache key. And actually, come to think of it, if memory for N concurrent in-flight requests is the concern, then you don't even need an actual (dynamically allocated) buffer, either; presuming you're using a streaming hash, you can feed each constituent field directly into the hasher, with only the hasher's (probably stack-allocated) internal static buffer for blockwise hashing required. (Which you're gonna need regardless of whether you're doing any Vary-ing.) So it's really just a question of how many CPU cycles are being spent hashing. And it's likely just going to be a difference between hashing 300 bytes (base request — hostname, path, headers that are always implicitly Varied upon) and 350 bytes (those things, plus whatever you explicitly Varied) per request. Doesn't seem like too much of a win... (especially when hardware-accelerated hashing ops operate on blocks anyway, such that you only get stepwise cost increases for every e.g. 128 bytes.) I wonder why they bothered?
foul 3y ago
>Not being content with alienating only the general purpose JSON API enthusiasts, let me now proceed to also alienate my erstwhile hypermedia enthusiast allies Nah I think he's right and it's coherent to avoid HTTP subtleties in that web architecture.
quotemstr 3y ago
> Your JSON API should remain stable. You can’t be adding and removing end-points willy-nilly. Yes, you can have some end-points respond with either JSON or HTML and others only respond with HTML, but it gets messy. What if you accidentally copy-and-paste in the wrong code somewhere, for example. Can't we handle this situation by regarding each version of the JSON output as a separate content type (which is arguably the semantically correct thing anyway) and then letting the server pick the more recent output version that the client supports?
- mst 3y ago
  I believe you could, but the one time I saw it done in practice it confused the heck out of most of the client devs and ended up being rewritten (this was an API talked to by multiple teams who were within the same megacorp but far enough away organisationally to be effectively 'just' users). I might consider using it for an API whose primary purpose is to support a specific client app that I also control, so users running older versions of the client app still get the desired results, but I don't think the additional elegance is a suitable trade-off for a general-use-by-others API sadly.
briHass 3y ago
I agree with the underlying point being made here: a separate API (REST/JSON) that serves data should be kept separate from the endpoints that serve an application, in this case, HTML-based. It seems like building your application around a single API that is also used to provide data externally saves you time, but you end up polluting that API with presentation concerns needed to drive the application's reports/grids/views. It's not worth the mental energy to consider how changes you need to make for presentation might affect the 'purity' of your public API. Returning hypermedia from the 'internal' API just forces that separation: there's no expectation that this 'data' is being returned for consumption by anything except the app that uses it.
- twelfthnight 3y ago
  It's a really good point. It's natural, but if an API is primarily used to serve a front end, all other internal consumers (analyst, data science, etc) take a back seat. I think it also leads to situations where most of the engineers on the team don't really understand the "natural" data domains, but always think about it in a front end lens (e.g. pagination, merging of different concepts because they are on the same UI element, very quick endpoints only, JSON instead of less web friendly serializations, etc)
- BiteCode_dev 3y ago
  Also the filtering, grouping, caching, pre-fetching and ordering contraints are unlikely to be the same. Even worse if the json api is public, as it may need to restrict capabilities.
rtpg 3y ago
I get this on one level but wow can you tell the difference between when a team uses the API they offer users and when they don’t, and the “single API” approach gets you there so well. I have basically never seen a nice user-facing API when it’s been split out. Sometimes that’s fine, but at least for enterprise use cases having a “real” API just feels like table stakes in so many domains for getting bigger clients onboard.
- 8n4vidtmkvmk 3y ago
  I thought I knew what the author meant until I got to the end. I could be that they're referring to HTML as one API, and JSON-RPC as the other API. Originally I thought they meant 2 JSON APIs. One that's tightly coupled with the HTML to handle all the "ajax" requests, and the other for 3rd parties to fetch arbitrary bits of data. Otherwise, I know what you mean. My company has our internal RPCs and then our customer-facing API and the customer one hasn't been updated in ages and it's just a thin layer over some old internal RPCs we used to have and now we have to maintain backwards compatibility but keep breaking it anyway.
  - rtpg 3y ago
    Yeah here they’re talking about a JSON API and then just “serving HTML” (which is its own API). I really think it’s so valuable to take what you use internally seriously, exposing and documenting it to end users is a great way to avoid hackiness, and just leads to more regular designs IMO. More work of course but … not that much in the vast majority of cases
    - BlueTemplar 3y ago
      Another issue might be security ? Completely different assumptions about it between internal and external sounds like you would want avoid sharing them anyway ?
      - erhaetherth 3y ago
        1st party APIs that are exposed to the client need to be secure anyway or users will discover and start using it. I found a library on GitHub that does exactly this to our app. They reverse-engineered nearly everything. I'm sure we could break the lib if we encrypt a few tokens but no one seems to care. I don't mind either, since we clearly aren't giving users a proper API.
agentultra 3y ago
Split it out but keep using the same models. Wouldn’t want your entities and semantics to drift apart between your UI and API. Personally I prefer sticking to the standards. At least that way when you move between projects you know what you’re getting into. But everyone has their own conventions these days. It’s all fragmenting.
HdS84 3y ago
Content negotiation is useful when there are multiple feasible return formats. But when is that the case? For data, Json is the absolute king. For content, html is king. There is very little to negotiate. The only case where I needed that feature was when data scientists wanted to download data from my API and needed a bunch of formats (parquet, CSV, TSV). But then they did not really grok content negotiation and asked for a query param. So finally I think this is like a lot of html features: half baked and from a different time. Html would do well to drop it.
3cats-in-a-coat 3y ago
Carson Gross' main thesis is this: "I am advocating tightly coupling your web application to your hypermedia API". In other words "have an API that returns HTML snippets tightly coupled to your website." It's ironic he wrote an article about how the industry uses the term "REST API" incorrectly, because he himself keeps using the term "API" incorrectly. If an "API" is tightly coupled to a single application, it's not an "Application Programming Interface"... it's just a part of your application. An API is supposed to be an interface on top of which multiple applications may rest. Particularly without a specific frontend in mind - so web, desktop app, mobile app, as a component of other services and so on. Obviously if it serves site-specific HTML snippets, that's not the case. The only reason he advocates this whole thing is because without it HTMX won't work, and in this way I find it quite myopic as a position. But if I was pushing HTMX I'd also be compelled to figure out reasons to make it sound good. So from that PoV, talking about "Content Negotiation in HTML APIs" loses meaning, as what he has is not an API in the first place, it's just his HTML website, but with some partial requests in the mix. And of course you wouldn't mix your API and your site. But this does not imply you can't and shouldn't use Content Negotiation either on your site, or in your API. You simply shouldn't use them to mix two things that never made sense to mix. A lot of his blog posts would become completely unnecessary if he just says "don't mix your website and your API, and the HTMX partial requests are part of your website, not your API". Alas he's stuck on this odd formulation of "hypermedia API" and constantly having to clarify himself and making things as clear as mud.
- dgb23 3y ago
  I think your definition of API is very useful, but API is the wrong term. It’s much more general than that and is used in contexts where these constraints don’t even make sense. Specifically a hypermedia API is something browsers (and the implicit backbone of the internet) understand very well. In fact you have to go out of your way to serve an application that your browser doesn’t inherently understand. The clarity that seems to be lacking here is not necessarily a fault of the author. We re-purposed some of these terms (REST, API etc.) to serve specific needs. But then kind of lost the understanding of what we had before. I think that’s not our fault though. Standardization didn’t move fast enough and the quality we needed in the mobile context wasn’t there.
liampulles 3y ago
At a very basic level, isn't it just confusing to have the same URL and method return two different completely different formats? I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to.
- runeks 3y ago
  > At a very basic level, isn't it just confusing to have the same URL and method return two different completely different formats? No, I don't think so. Why do you think so? > I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to. Content negotiation does not change what the API accepts. It changes the format of the response.
runeks 3y ago
I really enjoy these blog posts from htmx.org. Makes me better at creating applications faster, because they challenge my conventional wisdom that slows me down without offering a compelling advantage. Side note: I guess a better title for this blog post would be "Don't share endpoints between machines and humans". If the consumer of an endpoint is a human, it's probably a bad idea to make machines use the same endpoint. Content negotiation is fine, for example if I want to have an API return binary data instead of JSON (if, for example, I'm on an IoT device with sparse resources). This is fine because it's exactly the same API, with just the data format being different. And in both cases it's consumed by a machine/computer. In the case of the frontend/HTML, the consumer is a human (and not a machine), which — as the article mentions — adds a bunch of constraints that are not applicable to machines: a need for pagination (because scrolling through a thousand results can be annoying), a need for ordering (because I want the most relevant results first), displaying related content (as the article explains). The machine API doesn't necessarily need any of these features.
- rmbyrro 3y ago
  I'm new to htmx and hypermedia. Was about to use content-negotiation in a new web app. This post convinced me otherwise. I find this blog very useful and informative indeed.
jillesvangurp 3y ago
To me this just feels wasteful. Most APIs only ever return one type of thing. So the business of asking for application/json with every request and then the API confirming that, yes, it's still sending application/json;charset=utf-8 just seems a pointless waste of bandwidth. Same with API versioning. Most APIs are stuck at v1 forever. It never changes. It never gets verified. It gets hard coded all over the place. All for the option to, maybe introduce a v2 on the same server. Seems like a lot of premature optimization for not a whole lot of gain.
- pjc50 3y ago
  The number of situations where I have had to do miserable things in file formats (admittedly not APIs) to handle extending a format that was rushed together by a previous developer who did not make it extensible or support future versions makes me think that versioning is really important. Little company-specific binary file formats are just the worst.
deleted 3y ago
[deleted]
BlueTemplar 3y ago
Isn't the name "Data Application Programming Interface" redundant ? And "Hypermedia Application Programming Interface" wrong, because generally not for an application at all, but rather a (non-programming, as the author says, "for humans") interface to display multimedia documents ? (I guess you get (inevitable, if not necessarily good) feature creep as soon as you start including something like forms - see also : forms in pdfs ?)
xyzzy_plugh 3y ago
Content negotiation is surely a fire composed of tires but there are sadly many scenarios where it is necessary. Deciding whether to redirect for example is often context sensitive. In other words content negotiation is useful to be able to respond intelligibly. If a client asks you for json but not html, it might not make sense to return html.
mschuster91 3y ago
> Data APIs should be rate limited Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape (like many HN apps do, because HN doesn't offer an API). All a "data" API does is make the scraper's job easier. > Data APIs typically use some sort of token-based authentication / Hypermedia APIs typically use some sort of session-cookie based authentication So what. Any web framework worth its salt can support multiple authentication / credential mechanisms - the only "benefit" I can see from limiting cookie authentication is to make life for bad actors with cookie-stealer malware harder (like GitLab does, IIRC).
- benmanns 3y ago
  Not anymore! https://github.com/HackerNews/API https://github.com/HackerNews/API
- ronsor 3y ago
  > Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape Then proxies or other IPs will be used anyway.
- serial_dev 3y ago
  Hacker News has an API for a while now (I used it years ago around 2018), based on this post since 2014. Announcement https://www.ycombinator.com/blog/hacker-news-api https://www.ycombinator.com/blog/hacker-news-api API https://github.com/HackerNews/API https://github.com/HackerNews/API
- saagarjha 3y ago
  People usually screen scrape because of limitations in the Hacker News API, not because it's nonexistent.
afandian 3y ago
Tangential, but the lack of a "these types are available" as the counterpart to "Accept" has always been so obviously missing. What's the point of Accept if you don't know what to ask for? Maybe in the trivial case saying "I'd prefer a JPG to a PNG" can be an assymetrical choice. But in all the interesting use cases I can think of, e.g. where there are competing representation formats, you'd want the server to be able to respond to a HEAD with the choices. That's the kind of thing you can put in Swagger, but that might lead to hoisting the client's choice into the API, away from Content Negotiation.
- duskwuff 3y ago
  The OPTIONS response already describes which methods are available for an endpoint in the Allow header. Including a list of available types in a new header would be a pretty natural extension. (Content-Type isn't available because it's defined to only contain a single media type.)
- stephenr 3y ago
  Why? The whole point of content negotiation is the client tells the server which types it wants the content in, with weights to determine preference; The server then works out the best match for what's requested. Why do you need to ask about what's available? Just ask for it if you support it, and then handle the response based on its type.
- runeks 3y ago
  > What's the point of Accept if you don't know what to ask for? As a client, you do know what to ask for: everything you support, listed in order of preference. Why would you, as a client, care whether the server supports some format that you don't understand?
- deleted 3y ago
  [deleted]
mberning 3y ago
A technology like htmx seems to demand a “hypermedia” API. Whereas things like angular and react can consume a data API in many cases. However once an application becomes sufficiently complex people end up building data APIs just to suit their frontend framework. And in that case doing htmx and returning html seems nicer.
- simonbarker87 3y ago
  I've been building an app with HTMX over the past 6 months and this is the reason I am loving it. I don't even feel the backend/frontend interface, so rather than feeling like I am making the app twice (once in backend and again in the frontend to consume it) I am just making it once, there is no frontend, it's just all my app.
  - joshmanders 3y ago
    Yep, same here. Now I think my actual API is better and less cumbersome because I'm not trying to fit my own UI needs into a general purpose JSON API meant to be consumed by 3rd party clients who want to do things on my app.
- candiddevmike 3y ago
  Seems like a ton of added complexity to avoid using JavaScript/a fat client
  - infecto 3y ago
    Depends what you are trying to build. If you are building a complicated app that requires lots of states and you want it in the form of a SPA. Sure might be a bad fit. Lots of use cases could benefit from a hypermedia flow.
  - Arch-TK 3y ago
    Doing htmx is just as simple as doing server side rendering. Yes you still have a javascript dependency but that's a lot simpler than a client side javascript framework for pulling data out of a data API and rendering it client side. I don't think anyone's complaint about server side rendering has ever been that it was too complex.
deleted 3y ago
[deleted]

272 points by TheBigRoomXXL 3y ago

So, this is actually about: Why I Tend Not To Use Content Negotiation (to serve both HTML and JSON data from the same endpoints). The author also suggests: > The alternative is to ... splitting your APIs. This means providing different paths (or sub-domains, or whatever) for your JSON API and your hypermedia (HTML) API. I believe the alternative has been the norm actually. For example, many front-end frameworks encode UI states in URL, and it's not so sustainable to keep the alignment b/w UI states and data APIs in the long term.

whalesalad 3y ago

I think content negotiation is great when your usecase supports it, like asking for XML or JSON. Also the mappings of content types should be well defined for all to see and edit. Rails is kinda a labyrinth in that regard. I do tend to prefer actual file extensions though. Friendlier for humans (curl https://endpoint/item.json https://endpoint/item.json vs curl -H "accept: application/json" https://endpoint/item https://endpoint/item), and it is visible in logging infrastructure, sharable, etc.

qwertox 3y ago
For me it has always been a part of the query parameter http://endpoint/item?format=json http://endpoint/item?format=json or ?type=json I never ever had a problem with that. The only reason to use a file extension would be if the request would take no query parameters. It would never occur to me to use header information for this.
- dgb23 3y ago
  Both file extensions and query parameters are functionally very similar. What I don’t necessarily like about query params for this, is that their usage implies filtering. It’s a cultural assumption but it’s there. Another point: you might actually have a literal file at a similar path that you serve, typically when you cache responses (managed cache or straight from your app server). Using file extensions gives you a bit more natural affordances. It’s just overal less clunky.
FullstakBlogger 3y ago
Content negotiation is "friendlier to humans" in the sense that you can serve the right content to different clients using the same URL, transparent to the user. If the URL itself is never meant to be shared by the user, then I don't see the point. Say, an RSS feed being served as a formatted and styled page to a browser, or a client that accepts the usual XML.
- None4U 3y ago
  That's what XSLT is for though, the display layer should be your browser
IanCal 3y ago
Something nice about accept headers is you can have your own formats, and include versioning information. That means I can have a consumer specify it can process V2 and V3 formats of the API, and when the API is upgraded or downgraded around a release everything works smoothly. Your API can serve both but it doesn't have to - it can be easier to have clients support two versions than support long running parallel API versions. I sometimes prefer file extensions, but then consuming code can often get filled with appending or removing or changing formats. Broadly, I like content negotiation as a concept because it describes what you actually want and is much more typed (as far as it can be). Adding file extensions feels very stringly typed processing. Both have problems, you pick your poison and look wistfully at the greener grass on the other side.
conor- 3y ago
Your example of http://endpoint/item.json http://endpoint/item.json to me would imply that this is a static file- item.json -being served from the web server; whereas the explicit content-negotiated endpoint implies I'm hitting the endpoint item which will dynamically provide me a computed list of items.

simonw 3y ago

Cloudflare doesn't obey the Accept header on anything other than image content types. This means if you have an endpoint that returns HTML or JSON depending on the requested content type and you try to serve it from behind Cloudflare you risk serving cached JSON to HTML user agents or vice-versa. I dropped the idea of supporting content negotiation from my Datasette project because of this. (And because I personally don't like that kind of endpoint - I want to be able to know if a specific URL is going to return JSON or HTML).

ForkMeOnTinder 3y ago
That's even with the proper Vary header, right? Seems like a cloudflare bug if that's true. Maybe even a security bug if you find a service that supports text/plain.
- zoogeny 3y ago
  I haven't dealt with Cloudflare specifically, but I did deal with a number of big CDNs for large amounts of traffic. They were pretty adamant about NOT supporting arbitrary Vary header values. It broke some logic on a few of our systems and we eventually just decided to work around it instead of pushing our case. Interestingly, one of the big CDN providers did have controls in their UI for explicitly allowing/disallowing Vary header entries but they disabled it for us at some point (e.g. it was still in the UI but greyed out). I assumed once we hit a certain level of traffic it was too computationally expensive? Ever since, I've avoided any kind of fancy header/response variance in APIs just in case I end up in the same situation. It is rarely a necessity. IIRC, the only thing they continued to support variance wise was gzip (e.g. content-encoding). It's also worth noting they were extremely conservative with query parameters too. Also to reiterate, this was very high traffic and high volume with expectations of low latency, so probably not applicable to most people using CDNs for static website assets.
  - derefr 3y ago
    > I assumed once we hit a certain level of traffic it was too computationally expensive? Seems strange; AFAIK in e.g. Varnish, Vary just means you get more "stuff" tacked onto the buffer that gets built from the request and then hashed to create the cache key. And actually, come to think of it, if memory for N concurrent in-flight requests is the concern, then you don't even need an actual (dynamically allocated) buffer, either; presuming you're using a streaming hash, you can feed each constituent field directly into the hasher, with only the hasher's (probably stack-allocated) internal static buffer for blockwise hashing required. (Which you're gonna need regardless of whether you're doing any Vary-ing.) So it's really just a question of how many CPU cycles are being spent hashing. And it's likely just going to be a difference between hashing 300 bytes (base request — hostname, path, headers that are always implicitly Varied upon) and 350 bytes (those things, plus whatever you explicitly Varied) per request. Doesn't seem like too much of a win... (especially when hardware-accelerated hashing ops operate on blocks anyway, such that you only get stepwise cost increases for every e.g. 128 bytes.) I wonder why they bothered?

foul 3y ago

>Not being content with alienating only the general purpose JSON API enthusiasts, let me now proceed to also alienate my erstwhile hypermedia enthusiast allies Nah I think he's right and it's coherent to avoid HTTP subtleties in that web architecture.

quotemstr 3y ago

> Your JSON API should remain stable. You can’t be adding and removing end-points willy-nilly. Yes, you can have some end-points respond with either JSON or HTML and others only respond with HTML, but it gets messy. What if you accidentally copy-and-paste in the wrong code somewhere, for example. Can't we handle this situation by regarding each version of the JSON output as a separate content type (which is arguably the semantically correct thing anyway) and then letting the server pick the more recent output version that the client supports?

mst 3y ago
I believe you could, but the one time I saw it done in practice it confused the heck out of most of the client devs and ended up being rewritten (this was an API talked to by multiple teams who were within the same megacorp but far enough away organisationally to be effectively 'just' users). I might consider using it for an API whose primary purpose is to support a specific client app that I also control, so users running older versions of the client app still get the desired results, but I don't think the additional elegance is a suitable trade-off for a general-use-by-others API sadly.

briHass 3y ago

I agree with the underlying point being made here: a separate API (REST/JSON) that serves data should be kept separate from the endpoints that serve an application, in this case, HTML-based. It seems like building your application around a single API that is also used to provide data externally saves you time, but you end up polluting that API with presentation concerns needed to drive the application's reports/grids/views. It's not worth the mental energy to consider how changes you need to make for presentation might affect the 'purity' of your public API. Returning hypermedia from the 'internal' API just forces that separation: there's no expectation that this 'data' is being returned for consumption by anything except the app that uses it.

twelfthnight 3y ago
It's a really good point. It's natural, but if an API is primarily used to serve a front end, all other internal consumers (analyst, data science, etc) take a back seat. I think it also leads to situations where most of the engineers on the team don't really understand the "natural" data domains, but always think about it in a front end lens (e.g. pagination, merging of different concepts because they are on the same UI element, very quick endpoints only, JSON instead of less web friendly serializations, etc)
BiteCode_dev 3y ago
Also the filtering, grouping, caching, pre-fetching and ordering contraints are unlikely to be the same. Even worse if the json api is public, as it may need to restrict capabilities.

rtpg 3y ago

I get this on one level but wow can you tell the difference between when a team uses the API they offer users and when they don’t, and the “single API” approach gets you there so well. I have basically never seen a nice user-facing API when it’s been split out. Sometimes that’s fine, but at least for enterprise use cases having a “real” API just feels like table stakes in so many domains for getting bigger clients onboard.

8n4vidtmkvmk 3y ago
I thought I knew what the author meant until I got to the end. I could be that they're referring to HTML as one API, and JSON-RPC as the other API. Originally I thought they meant 2 JSON APIs. One that's tightly coupled with the HTML to handle all the "ajax" requests, and the other for 3rd parties to fetch arbitrary bits of data. Otherwise, I know what you mean. My company has our internal RPCs and then our customer-facing API and the customer one hasn't been updated in ages and it's just a thin layer over some old internal RPCs we used to have and now we have to maintain backwards compatibility but keep breaking it anyway.
- rtpg 3y ago
  Yeah here they’re talking about a JSON API and then just “serving HTML” (which is its own API). I really think it’s so valuable to take what you use internally seriously, exposing and documenting it to end users is a great way to avoid hackiness, and just leads to more regular designs IMO. More work of course but … not that much in the vast majority of cases
  - BlueTemplar 3y ago
    Another issue might be security ? Completely different assumptions about it between internal and external sounds like you would want avoid sharing them anyway ?
    - erhaetherth 3y ago
      1st party APIs that are exposed to the client need to be secure anyway or users will discover and start using it. I found a library on GitHub that does exactly this to our app. They reverse-engineered nearly everything. I'm sure we could break the lib if we encrypt a few tokens but no one seems to care. I don't mind either, since we clearly aren't giving users a proper API.

agentultra 3y ago

Split it out but keep using the same models. Wouldn’t want your entities and semantics to drift apart between your UI and API. Personally I prefer sticking to the standards. At least that way when you move between projects you know what you’re getting into. But everyone has their own conventions these days. It’s all fragmenting.

HdS84 3y ago

Content negotiation is useful when there are multiple feasible return formats. But when is that the case? For data, Json is the absolute king. For content, html is king. There is very little to negotiate. The only case where I needed that feature was when data scientists wanted to download data from my API and needed a bunch of formats (parquet, CSV, TSV). But then they did not really grok content negotiation and asked for a query param. So finally I think this is like a lot of html features: half baked and from a different time. Html would do well to drop it.

3cats-in-a-coat 3y ago

Carson Gross' main thesis is this: "I am advocating tightly coupling your web application to your hypermedia API". In other words "have an API that returns HTML snippets tightly coupled to your website." It's ironic he wrote an article about how the industry uses the term "REST API" incorrectly, because he himself keeps using the term "API" incorrectly. If an "API" is tightly coupled to a single application, it's not an "Application Programming Interface"... it's just a part of your application. An API is supposed to be an interface on top of which multiple applications may rest. Particularly without a specific frontend in mind - so web, desktop app, mobile app, as a component of other services and so on. Obviously if it serves site-specific HTML snippets, that's not the case. The only reason he advocates this whole thing is because without it HTMX won't work, and in this way I find it quite myopic as a position. But if I was pushing HTMX I'd also be compelled to figure out reasons to make it sound good. So from that PoV, talking about "Content Negotiation in HTML APIs" loses meaning, as what he has is not an API in the first place, it's just his HTML website, but with some partial requests in the mix. And of course you wouldn't mix your API and your site. But this does not imply you can't and shouldn't use Content Negotiation either on your site, or in your API. You simply shouldn't use them to mix two things that never made sense to mix. A lot of his blog posts would become completely unnecessary if he just says "don't mix your website and your API, and the HTMX partial requests are part of your website, not your API". Alas he's stuck on this odd formulation of "hypermedia API" and constantly having to clarify himself and making things as clear as mud.

dgb23 3y ago
I think your definition of API is very useful, but API is the wrong term. It’s much more general than that and is used in contexts where these constraints don’t even make sense. Specifically a hypermedia API is something browsers (and the implicit backbone of the internet) understand very well. In fact you have to go out of your way to serve an application that your browser doesn’t inherently understand. The clarity that seems to be lacking here is not necessarily a fault of the author. We re-purposed some of these terms (REST, API etc.) to serve specific needs. But then kind of lost the understanding of what we had before. I think that’s not our fault though. Standardization didn’t move fast enough and the quality we needed in the mobile context wasn’t there.

liampulles 3y ago

At a very basic level, isn't it just confusing to have the same URL and method return two different completely different formats? I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to.

runeks 3y ago
> At a very basic level, isn't it just confusing to have the same URL and method return two different completely different formats? No, I don't think so. Why do you think so? > I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to. Content negotiation does not change what the API accepts. It changes the format of the response.

runeks 3y ago

I really enjoy these blog posts from htmx.org. Makes me better at creating applications faster, because they challenge my conventional wisdom that slows me down without offering a compelling advantage. Side note: I guess a better title for this blog post would be "Don't share endpoints between machines and humans". If the consumer of an endpoint is a human, it's probably a bad idea to make machines use the same endpoint. Content negotiation is fine, for example if I want to have an API return binary data instead of JSON (if, for example, I'm on an IoT device with sparse resources). This is fine because it's exactly the same API, with just the data format being different. And in both cases it's consumed by a machine/computer. In the case of the frontend/HTML, the consumer is a human (and not a machine), which — as the article mentions — adds a bunch of constraints that are not applicable to machines: a need for pagination (because scrolling through a thousand results can be annoying), a need for ordering (because I want the most relevant results first), displaying related content (as the article explains). The machine API doesn't necessarily need any of these features.

rmbyrro 3y ago
I'm new to htmx and hypermedia. Was about to use content-negotiation in a new web app. This post convinced me otherwise. I find this blog very useful and informative indeed.

jillesvangurp 3y ago

To me this just feels wasteful. Most APIs only ever return one type of thing. So the business of asking for application/json with every request and then the API confirming that, yes, it's still sending application/json;charset=utf-8 just seems a pointless waste of bandwidth. Same with API versioning. Most APIs are stuck at v1 forever. It never changes. It never gets verified. It gets hard coded all over the place. All for the option to, maybe introduce a v2 on the same server. Seems like a lot of premature optimization for not a whole lot of gain.

pjc50 3y ago
The number of situations where I have had to do miserable things in file formats (admittedly not APIs) to handle extending a format that was rushed together by a previous developer who did not make it extensible or support future versions makes me think that versioning is really important. Little company-specific binary file formats are just the worst.

deleted 3y ago

[deleted]

BlueTemplar 3y ago

Isn't the name "Data Application Programming Interface" redundant ? And "Hypermedia Application Programming Interface" wrong, because generally not for an application at all, but rather a (non-programming, as the author says, "for humans") interface to display multimedia documents ? (I guess you get (inevitable, if not necessarily good) feature creep as soon as you start including something like forms - see also : forms in pdfs ?)

xyzzy_plugh 3y ago

Content negotiation is surely a fire composed of tires but there are sadly many scenarios where it is necessary. Deciding whether to redirect for example is often context sensitive. In other words content negotiation is useful to be able to respond intelligibly. If a client asks you for json but not html, it might not make sense to return html.

mschuster91 3y ago

> Data APIs should be rate limited Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape (like many HN apps do, because HN doesn't offer an API). All a "data" API does is make the scraper's job easier. > Data APIs typically use some sort of token-based authentication / Hypermedia APIs typically use some sort of session-cookie based authentication So what. Any web framework worth its salt can support multiple authentication / credential mechanisms - the only "benefit" I can see from limiting cookie authentication is to make life for bad actors with cookie-stealer malware harder (like GitLab does, IIRC).

benmanns 3y ago
Not anymore! https://github.com/HackerNews/API https://github.com/HackerNews/API
ronsor 3y ago
> Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape Then proxies or other IPs will be used anyway.
serial_dev 3y ago
Hacker News has an API for a while now (I used it years ago around 2018), based on this post since 2014. Announcement https://www.ycombinator.com/blog/hacker-news-api https://www.ycombinator.com/blog/hacker-news-api API https://github.com/HackerNews/API https://github.com/HackerNews/API
saagarjha 3y ago
People usually screen scrape because of limitations in the Hacker News API, not because it's nonexistent.

afandian 3y ago

Tangential, but the lack of a "these types are available" as the counterpart to "Accept" has always been so obviously missing. What's the point of Accept if you don't know what to ask for? Maybe in the trivial case saying "I'd prefer a JPG to a PNG" can be an assymetrical choice. But in all the interesting use cases I can think of, e.g. where there are competing representation formats, you'd want the server to be able to respond to a HEAD with the choices. That's the kind of thing you can put in Swagger, but that might lead to hoisting the client's choice into the API, away from Content Negotiation.

duskwuff 3y ago
The OPTIONS response already describes which methods are available for an endpoint in the Allow header. Including a list of available types in a new header would be a pretty natural extension. (Content-Type isn't available because it's defined to only contain a single media type.)
stephenr 3y ago
Why? The whole point of content negotiation is the client tells the server which types it wants the content in, with weights to determine preference; The server then works out the best match for what's requested. Why do you need to ask about what's available? Just ask for it if you support it, and then handle the response based on its type.
runeks 3y ago
> What's the point of Accept if you don't know what to ask for? As a client, you do know what to ask for: everything you support, listed in order of preference. Why would you, as a client, care whether the server supports some format that you don't understand?
deleted 3y ago
[deleted]

mberning 3y ago

A technology like htmx seems to demand a “hypermedia” API. Whereas things like angular and react can consume a data API in many cases. However once an application becomes sufficiently complex people end up building data APIs just to suit their frontend framework. And in that case doing htmx and returning html seems nicer.

simonbarker87 3y ago
I've been building an app with HTMX over the past 6 months and this is the reason I am loving it. I don't even feel the backend/frontend interface, so rather than feeling like I am making the app twice (once in backend and again in the frontend to consume it) I am just making it once, there is no frontend, it's just all my app.
- joshmanders 3y ago
  Yep, same here. Now I think my actual API is better and less cumbersome because I'm not trying to fit my own UI needs into a general purpose JSON API meant to be consumed by 3rd party clients who want to do things on my app.
candiddevmike 3y ago
Seems like a ton of added complexity to avoid using JavaScript/a fat client
- infecto 3y ago
  Depends what you are trying to build. If you are building a complicated app that requires lots of states and you want it in the form of a SPA. Sure might be a bad fit. Lots of use cases could benefit from a hypermedia flow.
- Arch-TK 3y ago
  Doing htmx is just as simple as doing server side rendering. Yes you still have a javascript dependency but that's a lot simpler than a client side javascript framework for pulling data out of a data API and rendering it client side. I don't think anyone's complaint about server side rendering has ever been that it was too complex.

deleted 3y ago

[deleted]