5 ms·

Akamai blocks unordered HTTP request headers

43 points by gwillem 9y ago

deleted 9y ago
[deleted]
idbehold 9y ago
In the second example the author does the following: $ ACCEPT="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" $ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP' The author fails to prefix $ACCEPT with the actual header key. It should be: $ curl -v -H "$UA" -H "Accept: $ACCEPT" $URL |& grep '< HTTP'
- deleted 9y ago
  [deleted]
- jstanley 9y ago
  From personal experience I'd be willing to give him the benefit of the doubt (i.e. he did it right, but wrote it up wrong). Good spot though.
- gwillem 9y ago
  Thanks! Indeed a copy paste error, I updated the article.
squeed 9y ago
Is it Akamai? Or is it a single site on Akamai? CDN customers can configure their sites in a million ways. My guess is a single site that was getting DDoS'd added this as an attack signature and forgot about it.
- jnfurst 9y ago
  This is just a single sites configuration.
  - bluesmoon 9y ago
    Well, everybody knows that in statistics, a sample size of 1 gives you a 0 margin of error ;)
- tyingq 9y ago
  My money, in this case, is something like Akamai Kona or Shape Security, that does bot blocking. Comparing user-agent against known header order for that specific user-agent sounds like something they would do.
deleted 9y ago
[deleted]
jnfurst 9y ago
This is just the configuration for a single site. The author did not even try it against www.akamai.com: $ URL=http://www.akamai.com http://www.akamai.com $ UA="User-Agent: Mozilla/5.0 My API Client" $ ACCEPT="Accept: /" $ curl -v -H "$UA" -H "$ACCEPT" $URL < HTTP/1.1 301 Moved Permanently < Content-Length: 0 < Location: https://www.akamai.com https://www.akamai.com < Date: Tue, 02 May 2017 14:46:59 GMT < Connection: keep-alive
AznHisoka 9y ago
Did Akamai recently just make this change? I'm asking because I've been running a web crawler for years now, and in the past week, I have noticed that the crawler is being rejected in more websites then usual.
michaelmior 9y ago
> most libraries use random order Most libraries use an undefined order. This is not the same as random.
- deleted 9y ago
  [deleted]
gumby 9y ago
I disagree with the author's title (and I see it was submitted with a different title). This is actually a report of two bugs: 1- the standard doesn't require an order 2 - the IETF's admonition that you be liberal in what you accept.

43 points by gwillem 9y ago

deleted 9y ago

[deleted]

In the second example the author does the following: $ ACCEPT="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" $ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP' The author fails to prefix $ACCEPT with the actual header key. It should be: $ curl -v -H "$UA" -H "Accept: $ACCEPT" $URL |& grep '< HTTP'

deleted 9y ago
[deleted]
jstanley 9y ago
From personal experience I'd be willing to give him the benefit of the doubt (i.e. he did it right, but wrote it up wrong). Good spot though.
gwillem 9y ago
Thanks! Indeed a copy paste error, I updated the article.

squeed 9y ago

Is it Akamai? Or is it a single site on Akamai? CDN customers can configure their sites in a million ways. My guess is a single site that was getting DDoS'd added this as an attack signature and forgot about it.

jnfurst 9y ago
This is just a single sites configuration.
- bluesmoon 9y ago
  Well, everybody knows that in statistics, a sample size of 1 gives you a 0 margin of error ;)
tyingq 9y ago
My money, in this case, is something like Akamai Kona or Shape Security, that does bot blocking. Comparing user-agent against known header order for that specific user-agent sounds like something they would do.

deleted 9y ago

[deleted]

jnfurst 9y ago

This is just the configuration for a single site. The author did not even try it against www.akamai.com: $ URL=http://www.akamai.com http://www.akamai.com $ UA="User-Agent: Mozilla/5.0 My API Client" $ ACCEPT="Accept: /" $ curl -v -H "$UA" -H "$ACCEPT" $URL < HTTP/1.1 301 Moved Permanently < Content-Length: 0 < Location: https://www.akamai.com https://www.akamai.com < Date: Tue, 02 May 2017 14:46:59 GMT < Connection: keep-alive

AznHisoka 9y ago

Did Akamai recently just make this change? I'm asking because I've been running a web crawler for years now, and in the past week, I have noticed that the crawler is being rejected in more websites then usual.

michaelmior 9y ago

> most libraries use random order Most libraries use an undefined order. This is not the same as random.

deleted 9y ago
[deleted]

gumby 9y ago

I disagree with the author's title (and I see it was submitted with a different title). This is actually a report of two bugs: 1- the standard doesn't require an order 2 - the IETF's admonition that you be liberal in what you accept.