8 ms·
Defining a new HTTP method: HTTP Search (2021)
- tkiolp4 3y agoWouldn’t be enough to “extend” GET so that it supports a body payload as well?
- time0ut 3y agoIt sort of does, but support is spotty. I've tried using it only to have things like MITM proxies on my customers' corporate networks drop the body. It'd be a large push to get universal support. It may be more difficult than simply adding a new method.
- tkiolp4 3y agoBut if a new method is needed, servers have to support it (e.g., by updating them to the next version that supports the new method). In the same sense, couldn’t that new version “just” not drop the body of GET requests? In any case we are talking about updating servers.
- Makhini 3y agoIt already does.
- simpsond 3y agoIt’s not exhaustive though. Many servers and proxies support it, but not all. For instance, GCP load balancers do not, but ALB does. At least that was case a few years ago.
- dwb 3y agoSort of, but it’s not that solid: https://datatracker.ietf.org/doc/html/rfc9110#name-get https://datatracker.ietf.org/doc/html/rfc9110#name-get (paragraph starting “Although request message framing is independent of the method used”)
- electroly 3y agoThe RFCs are clear that a server should ignore any body sent by a client after a GET request header. But RFCs don't matter; real life matters, and in real life there are lots of HTTP clients that can't make a GET request with a body. We had to go into our API and add POST support to every endpoint that was GET + body because there was a long tail of clients that couldn't manage to make the GET request. In practice, support is NOT widespread enough to rely on GET + body requests.
- simonw 3y agoThe only software I have seen that does this is Elasticsearch - it works with their client libraries but I'm always worried there may be proxies and suchlike which don't support it. They support POST as a fallback.
- dragonwriter 3y agoNo, because that would make GET not-cachable by URL alone, which breaks systems that rely on the fact that it is. A new method doesn’t break existing systems.
- jillesvangurp 3y agoYou can actually do that already. It's one of those things where the specification is a bit loose on this. Elasticsearch is an example of a product that supports doing a GET with a body to do search. Works fine but not all http clients allow you to do this so they also support POST. Nice discussion on HTTP GET with a body here: https://stackoverflow.com/questions/978061/http-get-with-request-body https://stackoverflow.com/questions/978061/http-get-with-req... TLDR. the http 1.1 spec was a bit vague on this and allowed people to send a body but specified that servers should ignore that if they did. Later updates of the 1.1 spec removed that sentence and actually allow this. The caveat that the spec mentions that some implementations may not support it. So an update is not needed; it's already allowed. I tend to not get too hung up on the meaning of http verbs. I sort of lost interest in the endless debates on this years ago. I like a well designed REST API of course and I try to follow the principle of the least amount of surprise when I implement one myself. Using exotic new experimental http verbs would be surprising. Using GET this way is also a bit surprising. I actually steer clear of using PATCH as well. Just use a PUT or a POST and move on. Adding new verbs to HTTP is redundant as far as I'm concerned. Not really against it but where does it stop? And what do we get out of it? It doesn't sound like it's worth the trouble.
- Spivak 3y agoSee also QUERY, imho a better name than SEARCH which is semantically a GET but with a request body -- it's cacheable! https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-02.html https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-meth...
- mlhpdx 3y agoYep, I’ve already implemented QUERY. Sorry SEARCH.
- buildfocus 3y agoThey're effectively the same I think - this post is from 2021 but a later version of the same draft renamed the concept to QUERY.
- montroser 3y agoThis seems reasonable, but unlikely to happen. GET is already pretty much sufficient, especially given that clients generally support megabytes worth of query strings in these modern times. If we're doing this though, I'll cast my vote for naming it more semantically as QUERY, and forgoing whatever little WebDAV compatibility would be had otherwise.
- ZiiS 3y agoQUERY would be more compatible with WebDAV as the would be no confusion. The relevant compatibility is with middle boxes like proxy servers that mostly already forward SEARCH but would block QUERY.
- Raqbit 3y agoMiddle boxes can arbitrarily limit the request URL size in certain cases.
- rcme 3y agoSure, but in the times of encrypted traffic, the only middle boxes capable of limiting the query should be ones explicitly added by the service owner.
- wongarsu 3y agoIn an enterprise setting you can just roll out your own root certificate to every machine's trust store, and have your middle box MitM all traffic with the help of that.
- dragonwriter 3y agoFunny you should say that: https://httpwg.org/http-extensions/draft-ietf-httpbis-safe-method-w-body.html https://httpwg.org/http-extensions/draft-ietf-httpbis-safe-m...
- austin-cheney 3y agoI understand the benefits of QUERY over GET because GET is expected to return a resource in the response. Why not just use POST though? Send a post message to a server asking to conduct a search and sending the results of that query in the response?
- AtNightWeCode 3y agoOr just use GET since it already works. I don't see any problems with large query params. You can also split the search into a POST request and a GET request for the result.
- insanitybit 3y ago> I don't see any problems with large query params. Most servers will reject URLs after a certain size, 4-8KB.
- le-mark 3y agoThis is correct, and one only needs to consider the impact of large or unlimited length urls on server memory consumption per request to see that a relatively small limit is desirable and even necessary.
- AtNightWeCode 3y agoDepends. But moving the payload between the body and the query will not solve anything performance wise if you design for it.
- AtNightWeCode 3y agoA query of that size will likely not benefit from a GET anyway so use a POST.
- imnotjames 3y agoSure except POST has its own set of issues - like not being cachable. But SEARCH or QUERY can be cached safely!
- deleted 3y ago[deleted]
- time0ut 3y ago
- hgl 3y agoMany people seem to be unenthusiastic about it because the limitation on query strings are usually large enough. I personally like this addition, because it no longer requires all queries be shoehorned into query strings. You can use any syntax you like, be it SQL or GraphQL etc.
- cratermoon 3y agoI'm trying to imagine an example of the authors primary use case: "a complicated data retrieval, sending lots of data but not changing the server state". Is he imagine sending something like a SQL statement SELECT with multiple WHERE and JOIN clauses using a recursive CTE? If defining the attributes of the thing to return takes that much complexity, maybe that's a sign the endpoint is poorly designed, rather than GET query string length being insufficiently large.
- bastawhiz 3y agoA great example of this today is GraphQL. Even if you disagree with the premise of GQL, you can understand how you can have complicated filtering and sorting logic, and conditional expansion of nested resources. For a nontrivial data model, you may want to select only a minimal number of fields on each nested resource. Having lots of different variants of the same endpoint to serve the same query with different shapes of output is difficult to maintain, so this makes some sense.
- cratermoon 3y agoI knew someone would bring up GraphQL, as I consider it a sort of way to present a database schema in JSON, so of course it would result in complex SQL-like queries. I understand why it works for Facebook, but it's not something I've ever reached for to solve any programming problem I've had.
- bastawhiz 3y agoNothing about GQL in particular is necessary to appreciate the need for this. Consider the Stripe API, where you might fetch invoices and expand the customers for each of those invoices, and then fetch the default payment methods for each of those customers. Stripe does this with a mess of query string parameters over a REST-ish API, but it could be accomplished much better with a request body. GQL solves this problem in a particular way, but even folks who don't turn to GQL still have this problem.
- kortex 3y ago
- ThaFresh 3y agobeing from 2021, I guess this one didnt pan out
- dragonwriter 3y agoIts still actively being worked (the most recent workgroup draft, and IIRC several of the internet drafts since the posted one, names it QUERY, SEARCH as a name for thr verb is probably dead) though, yes, the particular 2021 draft didn’t get adopted as anything more.
- IshKebab 3y agoThe advantage of GET is that you can always link to it. Doesn't matter with POST because you're never going to want to link to a POST. So, GET URLs with base64 encoded data is definitely ugly, but at least you can save them and share them easily.
- asattarmd 3y agoOne advantage of GET is I can just copy the URL and share it. The article makes no mention of that. While I love the proposal (apart from the name, I can see the SEARCH verb being used for something that's not search), they should also address the URL share-ablity aspect. Something like https://google.com/search<some-special-character><query> https://google.com/search<some-special-character><query> where query can be arbitrarily large (>2000 URL length restriction) and the browser is smart enough to treat only https://google.com/search https://google.com/search as the URL and anything after that as the body. The complete "URL" can be big and shared anywhere else.
- hartator 3y agoYes, that’s also lost when you do POST. Which is by design though. A HTTP Search seems like only drawbacks.
- paulddraper 3y agoSEARCH can have a request body. Many systems limit the length of the URL, so this is significant.
- adql 3y agoSo can POST. It's entirely redundant
- hakanito 3y agoFrom reading the article, one of the key items addressed by HTTP Search is caching. POST requests are usually not safe to cache
- 411111111111111 3y agoIs search safe to cache though? It would definitely heavily depend on the dataset you're querying... Very little value in cached search results if you're searching through time-sensitive data such as logs or other live-datasets. Most datasets I've searched also has a concept of permissions, so person a couldn't be served the same cached result as person b... I think search can't be cached at the http level either, its too heavily dependent on the data you're searching through so you'd have to implement it in the application anyway. the article does make a good point though: a `get` request that supports a `body` would be nice, and thats pretty much all they're arguing for with the `search` verb.
- based2 3y agoThe missing ones are HTTP OPEN/CLOSE to improve the sessions management and security audits.
- brabel 3y agoDid you know that HTTP is an inherently stateless protocol? From the very first line in the RFC[1]: "The Hypertext Transfer Protocol (HTTP) is a stateless application- level protocol for distributed, collaborative, hypertext information systems." Which is why you need to send a header every time you make a request with authorization data. Given that: no, stateful methods like OPEN/CLOSE would not fit at all with how HTTP is designed to work. [1] https://www.rfc-editor.org/rfc/rfc7231 https://www.rfc-editor.org/rfc/rfc7231
- phendrenad2 3y agoCan anyone steelman this idea for me? Seems like an incredible waste of time to make this.
- hackerbrother 3y agoI want this. SEARCH/POST APIs forever.
- mabbo 3y agoWhat I like about this proposal is that no one has to use it. There's lots of comments here saying why they prefer GET, and those people are welcome to keep using it- GET will remain unchanged. But for those applications where having a complex body in the query request is a better choice, this tool can be available. I think that's a win.
- zeroimpl 3y agoTIL sending a POST request to a URL invalidates the cache for GET requests of the same URL. Going to have to review my REST APIs to make sure that’s not a problem.
- tacone 3y agoIncredible, I did not know that. Does anybody point out to some resources to learn more about this behavior?
- brabel 3y agoI'm sorry, but isn't that the most basic thing possible about GET and POST?? GET is used to read a resource, POST to write to it... why would you NOT expect writing to a resource to invalidate caches of that resource??
- tacone 3y agoBecause usually GET /posts gets you a list of posts, POST /posts create a new one.
- JulianWasTaken 3y agoI don't know what level of incredulity is appropriate in my opinion (probably not as much as the parent comment), but isn't that another good reason to expect cache invalidation? Surely creating new posts would expectedly invalidate the list of posts cached, it has to now include at least one new one.
- zeroimpl 3y agoI think you’d have to have a fairly awkward API design in order for this to be an issue. For example, if “POST /posts” does something else, like flags a post for a moderator to review. That being said, I don’t see this feature being particularly beneficial either. Only the caches which observed the request could know to purge the cache. So you can’t actually rely on it for cache invalidation.
- yashap 3y agoI’d like to see this (or something like it) ship. Right now the two most common alternatives are: 1) GET, but base64 encode a json payload as a query param. Main downsides are that you can hit URL size limits, it makes the client and server slightly more complex (base64 encode/decode), and it sucks with browser dev tools - can’t nicely inspect the request in the browser, have to copy and decode it 2) Use POST to search. This confuses both machines and people. On the machines side, it doesn’t play well with caches, retry middleware, etc. On the people side, it confuses developers, ppl monitoring metrics, etc. - you think it’s creating something but it isn’t. Basically goes against all the reasons we have separate GET/POST/PUT/PATCH/DELETE methods in the first place Yeah, we have workarounds, but they have pretty significant downsides. I’d rather have this new method. Honestly, I think one of the main reasons ppl reach for RPC frameworks over RESTful APIs is the awkwardness around making search queries - this would really fix that issue.
- afiori 3y agoI would say that people should be more open to using HTTP in non-REST ways. Sometimes you want to send a request and receive a response, with arbirtrary restrictions and side effects that suit your cases. There are a lot of good reason for rest to exists, but also sometime you want to POST /open-garage-door?t=5-minutes and call it a day
- yashap 3y agoSure, the odd RPC style HTTP call is fine - not everything is cleanly modelled as a resource. But the lack of a clean way to make GET requests with a more complex request payload is simply a wart/shortcoming in the HTTP protocol IMO, and one that’s very correctable. Sucks to have to abandon RESTfulness even when I have a very “clean fit for the resource model” call to make, but the request body is too complex/large for HTTP query params.
- revskill 3y agoWhat if modern browser support advanced URL bar with POST method ? (Like Postman UI)
- niij 3y agoThis new method would be idempotent and safe. POST is neither. But, adding other methods than GET to the address bar in browsers is an interesting idea. It'd affect such a small percent of people it'd most likely be a net time sink than time saver, though.
- politician 3y agoI used a SEARCH verb for an REST API years ago. Granted, we only had to support clients that we wrote ourselves. Semantically, however, it made a lot of sense to include a payload in the request body describing the query. No one else needed to know or care that it existed.
- nonethewiser 3y agoAlternative naive solution: Add a body to GET requests. As I said, this is totally naive. Is there some obvious reason why this is a terrible idea, or not really possible?
- earthling8118 3y agoThere are a lot of client implementations that don't support this. You already can add a body to GETs but you'll likely run into issues somewhere due to this. I believe that JavaScript's fetch doesn't allow for this as one example. That's a big one that is probably important to have but there are many others.
- kevincox 3y agoBecause a lot of existing proxies and other middleware will just drop the body. Or even worse not consider it in caching decisions but still proxy it.
- samwillis 3y agoIf they do add this I hope they also add support to the Browser History pushState API to set the request body. You can obviously set an arbitrary JS state object, but for server rendered "html over the wire" single page apps it would be nice to be able to do a pushState that is sent to the server on refresh.
- LinuxBender 3y agoI was under the impression that new methods could not be amended into an existing RFC by revision but would require a new major.minor version all together is that not the case? Or is this worked around with the concept of extensions and if so doesn't that mean vendors of middle boxes are not required to recognize it?
- throwaway14356 3y agoi suppose you could post and store a query, give it a number and use get.
- malablaster 3y agoOne of the arguments for a separate SEARCH verb is to differentiate between the cacheing needs of POST. Aren’t cache headers sufficient for using the existing verbs?
- hcarvalhoalves 3y agoRight now, you have two main options: Use a GET, and squeeze all the parameters you need in the URL or headers somewhere Use a POST, and have the request considered as unsafe & uncacheable Third option: you POST or PUT to a resource representing the search, then you’re free to redirect and subsequently GETs of this resource can be cached. Wanting a special method for search hurts the conceptual integrity of HTTP, where resource representation is the core idea to build on top, it isn’t supposed to be just a request/response protocol.
- leftnode 3y agoThat's exactly what we do: submitting a search/filter form through a POST request which creates a record in the database representing the search/filter criteria and then redirects to the same endpoint with the ID of that record. The endpoint looks for the ID in the query, attempts to load the same record, and if found, executes and returns the search results. It works very well, is completely transparent to the end user, has automatic query logging, and provides a clean URL that can be easily shared.
- zulu-inuoe 3y agoI really wish I'd thought of this method when I was building a search based application. It was my first time and I struggled with structuring everything
- tedyoung 3y ago> POST or PUT to a resource representing the search, then you’re free to redirect and subsequently GETs of this resource can be cached. But that only means that specific instance of the search query is cacheable, not the search query itself, no? I presume the POST would create a new identifier for the resource, so that yes, that specific resource is cacheable. (Even if the server says "oh, I've just seen this POSTed query, let me return the same resource ID", another client will still have to POST to the server, a non-cacheable action.) The idea of cacheable isn't just by the server, but by anything in between the client and the resource. By using QUERY, the query itself can be cached.
- pzlarsson 3y agoThis sounds promising as a lot of use cases are shoe horned into the current verbs. A bit surprised EVALUATE isn't discussed as a name since it would make sense both for search queries and other "get using body" scenarios.
- wizofaus 3y agoOne of the user cases listed - being able to send a large body of data to a server for it to encrypt (though it could be any sort of transformation, e.g. reformatting, encoding, format conversion, or even operations like "spell check" etc.) strongly suggests to me we don't need a new verb, but should simply redefine the expected behaviour for GET requests for when they do or don't have a body - even if it means there's a expectation and method for clients to first query servers whether they have that ability (as servers that don't support "search" will return an error if that verb is used, whereas servers, esp. proxies/load balancers etc, may well simply ignore the body if GET is used.). And to not define caching semantics seems somewhat crazy - it's bound to result in implementations inventing their own (the obvious one would be to combine some hash of the body with the URL to uniquely identify the request).
- Too 3y agoBy adding the new verb, it implicitly formalizes that body in GET should not be used. Previously it was allowed, although not always expected. This may be a good thing and it could also be a bad thing if middleware starts relying on this new expectation and break old applications that previously assumed get bodies to be passed through. They should clarify the expected behavior of GET bodies more explicit, whether it is allowed or not doesn’t really matter, as long as it is crystal clear.
- pan69 3y agoAnother way to think about it, if you're doing REST as it seems the author of this article seems to strive for is to introduce a "search resource". You'd then POST to e.g. POST /searches with the query in the body, and on the server side a search resource is created. The server performs the search query and returns the result with a search ID. To do the same search again just do a GET /searches/ID.
- BugsJustFindMe 3y ago> GET requests ... can't have a request body. It's not specifically banned, but it is defined as being completely meaningless This is widely believed but false. And repeating it leads me to believe that the author is not careful enough to create new standards. They are just not defined to be meaningful, which is not the same thing. In fact the standard says that sending a message body with a GET request should not (specifically SHOULD NOT, not MAY NOT) be done unless you have confirmed that the servers accept it, because servers are not required to accept it, but that if you are talking to a server directly that you know supports it then it's perfectly fine and within the standard.
- poisonborz 3y agoAs other commenters pointed out, support is spotty and it's generally a greyzone. A new verb (I'd also support "QUERY") would clarify things.
- akvadrako 3y agoBut it doesn't actually work in many browsers.
- mdale 3y agoMaybe read only query getting larger than the GET length limit indicates it's post-processing the object for you and not a "cachable" resource from a key value retrieval perspective? I.e your POST'ing a read only query since an application has to process the query for your at that point ?