3 ms·

> Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a

> Outlook will only send GET requests, which are idempotent unless you're ignoring the spec. A message saying "this code has already been used" after sending a GET request is a bug. Alright, so imagine this: we have two endpoints GET "/page" and "POST /increment". Making a POST request to "/increment" increments a counter kept in memory and returns it's new value. The GET "/page" endpoint returns a HTML file, which contains JavaScript code that when executed, calls the "/increment" endpoint. Are we now breaking the HTTP specification saying GET requests has to be idempotent if we visit "/page" in our browser? I think not, but this is sometimes how pages are implemented, which robots are gonna have to deal with, as otherwise many would consider it broken. Don't get me wrong, I think it's a shitty implementation as well. But is it breaking the HTTP specification? Unlikely.

tidenly 4y ago
From the spec, although it doesn't explicitly say 'dont automatically send a POST upon opening a GET', I think it's fairly clear its against the spirit of what a GET should represent to the user if it isn't a safe POST request. "In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested. Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them."
jeroenhd 4y ago
I don't think it's breaking the spec per se, but web crawlers execute javascript and people hit reload on their browsers, sometimes accidentally. Automating this process may not be the solution here. Personally, I think web crawlers like Bing shouldn't be executing javascript at all but front developers can't go without their client side rendering frameworks so search engines are more or less forced to. As for a security mechanism, you want to emulate a browser as closely as possible to detect tricks like redirects from safe domains to attack domains and obfuscated URL crap. I'd expect any automated, non-interactive code to execute in a security analysis sandbox. Is this breaking the standard? Who knows. What is a cloud antivirus but a web user agent running in a data center? The email protocol doesn't specify how the client should deal with links, the robots.txt only works for spiders, not for manually submitted URLs like those clicked in emails, and without a noindex tag you're going to see your page indexed by the mail provider company regardless of what your robots file says. I think in theory your solution solves the spec breaking problem, but it doesn't solve the problem in practice because there are many other components for which there are no standards and defensive programming is required.

tidenly 4y ago

From the spec, although it doesn't explicitly say 'dont automatically send a POST upon opening a GET', I think it's fairly clear its against the spirit of what a GET should represent to the user if it isn't a safe POST request. "In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested. Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them."

jeroenhd 4y ago

I don't think it's breaking the spec per se, but web crawlers execute javascript and people hit reload on their browsers, sometimes accidentally. Automating this process may not be the solution here. Personally, I think web crawlers like Bing shouldn't be executing javascript at all but front developers can't go without their client side rendering frameworks so search engines are more or less forced to. As for a security mechanism, you want to emulate a browser as closely as possible to detect tricks like redirects from safe domains to attack domains and obfuscated URL crap. I'd expect any automated, non-interactive code to execute in a security analysis sandbox. Is this breaking the standard? Who knows. What is a cloud antivirus but a web user agent running in a data center? The email protocol doesn't specify how the client should deal with links, the robots.txt only works for spiders, not for manually submitted URLs like those clicked in emails, and without a noindex tag you're going to see your page indexed by the mail provider company regardless of what your robots file says. I think in theory your solution solves the spec breaking problem, but it doesn't solve the problem in practice because there are many other components for which there are no standards and defensive programming is required.