9 ms·

Golang's network stack attempts to parse HTTP headers as UTF-8

jrimbault 6y ago
This article (/quasi-essay) was a good read : https://fasterthanli.me/articles/aiming-for-correctness-with-types https://fasterthanli.me/articles/aiming-for-correctness-with...
- dgellow 6y ago
  Damn that was a VERY long read, but also a fantastic one. Thanks for sharing!
panpanna 6y ago
Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. Now, what are the security implications of this? (beside the chunk thingy)
- _-___________-_ 6y ago
  > Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. > Now, what are the security implications of this? (beside the chunk thingy) Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context. Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
  - panpanna 6y ago
    > That's a terrible reason. Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang. It leads to all kind of issues, not only security problems.
    - kerng 6y ago
      But that's a Golang issue then, not http
      - panpanna 6y ago
        I know, I am just trying to explain how this could happen :)
    - deleted 6y ago
      [deleted]
    - _-___________-_ 6y ago
      Yes, but this simply means that header values are not (Go) strings, they're byte arrays. Many languages represent them this way, with a (fallible) function to get the string value iff the byte array contains valid UTF-8. The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
    - 0xjnml 6y ago
      1) Go string type is specified as a sequence of bytes. Full stop. Any encoding, binary blob, string does not care. Particular stdlib string functions _do_ care, expect UTF-8 encoding and document the fact. Other stdlib string function work with the sequence as a sequence. 3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals. 2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
  - yokaze 6y ago
    > That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1 https://tools.ietf.org/html/rfc822#section-3.1
    - Thaxll 6y ago
      nope: https://tools.ietf.org/html/rfc7230#page-25 https://tools.ietf.org/html/rfc7230#page-25
    - vaduz 6y ago
      > Actually, they are explicitly specified as ASCII They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2. https://tools.ietf.org/html/rfc7230#section-3.2.4 https://tools.ietf.org/html/rfc7230#section-3.2.4
- albinowax_ 6y ago
  I think it's worth spelling out that 'chunk thingy' can lead to complete site takeover. For example, using this behaviour I was able to get persistent control of PayPal's login page. That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015 https://news.ycombinator.com/item?id=25858015
  - panpanna 6y ago
    Do you have any pointers to the PayPal hack?
    - albinowax_ 6y ago
      Yep, the original reports are here: https://hackerone.com/reports/488147 https://hackerone.com/reports/488147 https://hackerone.com/reports/510152 https://hackerone.com/reports/510152 And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request...
      - panpanna 6y ago
        This is awesome, thanks!
      - Uberphallus 6y ago
        Wow, excellent work!
- superjan 6y ago
  It could get hairy when here is also another parser that has different assumptions. They don’t see the same URL. Psychic paper [0] comes to mind. [0] https://daringfireball.net/linked/2020/05/02/psychic-paper https://daringfireball.net/linked/2020/05/02/psychic-paper
- jlouis 6y ago
  The string type is utf8. You would think an old http header is a string. It's not. It looks like a string though :)
  - papaf 6y ago
    The string type is utf8. Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types https://golang.org/ref/spec#String_types However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.
- kerng 6y ago
  Burp employee made like 70k dollars in bug bounty some time back with smuggling attacks on various sites
- Thaxll 6y ago
  In Go a string is an immutable byte slice not UTF-8.
  - jrimbault 6y ago
    I'll also play along the role of the pedant : > In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8. Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
    - papaf 6y ago
      Its a common mistake to assume strings are UTF-8 but it is a mistake. This is not a defence of Go but more of a warning to Go programmers.
      - adonovan 6y ago
        Exactly. It's a convention, not a law of physics. If a string's UTF-8 validity is important to the correctness of your function, check it. It's not expensive.
- user-the-name 6y ago
  The real problem is that Go does not actually have a string type. It only has byte strings which can accidentally also be text strings encoded in utf-8. This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software. Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
  - naniwaduni 6y ago
    The fact is that there is no correct "text string" type, because there are no valid operations on so-called text strings in general.
    - user-the-name 6y ago
      That... makes no sense? Of course there are a lot of valid operations on text strings?
albinowax_ 6y ago
If you're wondering what kinds of attacks this enables, I think the primary risk is HTTP Request Smuggling https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... But there's a bunch of hard-to-quantify side-risks based on corner-cases that could lead to stuff like request header injection, server-side parameter pollution, and host-header attacks. I'm excited to see what shows up.
dcomp 6y ago
Technically isn't incorrect but is a violation of SHOULD. I still think a SHOULD should be a requirement for a general purpose library. With reasons given for not following it. Edit: And with HTTP having case-insensitive matching (which is most likely broken in lots of hand written implementations). This is rife with the possibility for errors Taken from RFC 7230 3.2.4 Field parsing Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.
- mytailorisrich 6y ago
  > A recipient SHOULD treat other octets in field content (obs-text) as opaque data. This is not really a 'should', IMHO, because fields are defined as OCTETS, iirc. Based on that, a compliant and robust implementation must treat them as opaque data.
  - dcomp 6y ago
    I still fight with case-sensitive matching breaking HTTP2 -> HTTP1.1 proxies
  - vaduz 6y ago
    RFC 7230 makes it a point not to make it a MUST as that would make unknown number of existing applications non-compliant with HTTP/1.1-as-redefined. They are free to treat the incoming headers as ISO-8859-1 8bit instead of dropping to 7bit US-ASCII.
    - mytailorisrich 6y ago
      RFC 2616 defined header fields as OCTETs, and regarding this change RFC 7230 states: > Non-US-ASCII content in header fields and the reason phrase has been obsoleted and made opaque (the TEXT rule was removed). RFC 2616: field-value = ( field-content | LWS ) field-content = <the OCTETs making up the field-valu and consisting of either TEXT or combinations of token, separators, and quoted-string> Hence to me fields must be treated as opaque data for backward compatibility and robustness. If anything, existing applications that are compliant with RFC 2616 already do that, right? ;)
saurik 6y ago
Every time someone creates a new programming language, the world builds yet another pile of broken HTTP implementations--clients and servers--because everyone refuses to read the specifications, thinking "none of this is hard". I remember dealing with some early Ruby web servers, 15 years ago, and was just shocked at how broken they were despite being used by everyone in production! In fact, there is a lot of nuance to HTTP, and as a civilization we seemed doomed to reliving the same mistakes over and over again every few years when the entire stack gets rewritten from scratch :/. So, I thereby was neither surprised to read that the Go HTTP server did this obviously-wrong thing... NOR was I particularly surprised to read the person complaining about it also didn't seem to understand the correct behavior, either (though I do appreciate this is a bit "unfair"--I mean, it's a tweet ;P--but I think still makes for an interesting point here) :/. I have dealt with so many broken HTTP clients and servers over the years written by people who probably also never read the manual that came with their microwave because "documentation is for idiots" or whatever, that I have just become demoralized about it :(. A few months ago, I was complaining about this, with a short list of stuff everyone tended to get wrong, and this HTTP header field encoding issue was on my shortlist :(. https://news.ycombinator.com/item?id=24835617 https://news.ycombinator.com/item?id=24835617 > FWIW, I have encountered tons of issues in peoples' "this is easy, right?" implementations of 100 Continue (including in some at-the-time major Ruby HTTP servers), non-ASCII encoding in HTTP headers (which has a ton of subtle rules, and just throwing around UTF-8 is not correct), and chunked Transfer-Encoding (though I can't remember what was wrong); further, people often don't even realize HTTP trailers exist at all, which is a bit infuriating :/. (Note, though, that I am still 100% on the side of the anti-HTTP/2 camp in this thread.) The core problem is that somehow people think they can implement a spec without even, you know, reading the spec, which is so far past hubris as to have gone through incompetence into obvious negligence. AFAIK, HTTP header fields are (in their generality) specified to be read as binary octets... but then are restricted to using ISO-8859-1 with the premise you are supposed to run them through a MIME parser? Like, some fields have strict formats, so you can't do the full decoding until you know what you are working with--and so I would argue an HTTP library should give you the raw binary data (though decoding them through ISO-8859-1 first isn't supposed to be "wrong")--but for fields with user-displayable text (including HTTP status messages!) you can seriously embed "Q" encoded strings (of all things) if you want, a la RFC 2047. You can't assume a field is restricted to ASCII too early in the parsing process. https://tools.ietf.org/html/rfc2047 https://tools.ietf.org/html/rfc2047
ipython 6y ago
Isn’t that more a Unicode normalization problem than utf8?
- albinowax_ 6y ago
  HTTP headers are generally treated as ASCII, and Golang's decision to treat them as UTF-8 opens the gate to a range of issues, including the linked unicode normalization attack.
IshKebab 6y ago
One of the hazards of text-based protocols.
- userbinator 6y ago
  Indeed --- if HTTP was a binary protocol, with headers and their values being defined constant integers, this would definitely not happen. No one would confuse 1 with 2, and implementations would probably be quite a bit more efficient... but the "lamers" really want their pile of leaky abstractions because they can't be bothered using a hex editor. </s> Nevermind the fact that some parts of HTTP are really just binary written as text, and now even most non-developers know what a 404 is...
DethNinja 6y ago
I didn’t know this kind of attack before but wouldn't any sane backend system confirm the permissions of requesting client first before handing out admin pages? I mean you need to have a client-id and their permissions for any outside request coming to server. I’m using Golang in my webserver but it wouldn’t hand-out secret pages as requesting client wouldn’t have authority to do so. I feel like Golang's UTF8 handling is the least of the worries for these kinds of attacks, it points out that entire backend system was constructed wrongly, number one rule of any backend systems: You cannot trust the client.
- pram 6y ago
  I agree. Seems like something that is easily defeated using, for example, a JWT
vaduz 6y ago
There are sequences in ISO-8859-1 that will cause UTF8 application to balk if they are read without conversion - sometimes in a part that is well away from the input itself. A common example "órny", part of a number of Polish place and personal names. ISO 8859-1 (and -2, and others) form would be F3 72 6E 79. UTF-8 treats F3 as a start of a 4 byte sequence where each subsequent character is in 80 - BF range (as specified in RFC 3629): UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) / %xF4 %x80-8F 2( UTF8-tail ) UTF8-tail = %x80-BF This nicely causes an exception in both Java and Python, for instance, if they were to do what Go does here - and there is nothing anyone from adding an X- header to the request or response to try just that.
treeman79 6y ago
Spent trying to parse old encoded documents. Spent a month doing a deep dive into internals of bytes and formats. Finally just wrote code to loop through all possible encodings and ways to change encodings. Wait until a match against a known string was found. Proper encodings found in a few minutes.
- burnthrow 6y ago
  That approach sounds like it would yield many false positives.
userbinator 6y ago
This is the cost of "extraneous hidden complexity", or as I like to think of it, "abstraction hides bugs". As others here have noted, it is all too tempting to use a language whose string type actually does something very fancy, and be unaware of it. Most of the time it "only" causes inefficiency. I don't remember if the HTTP spec has it, but I clearly remember the HTML spec has some parts that explicitly say "case-insensitive in the ASCII range only" (i.e. 61-7a = 41-5a), presumably to prevent things like this from happening.
- dgellow 6y ago
  From what I got from the spec, HTTP headers are supposed to be ASCII or opaque bytes.
- bzbarsky 6y ago
  It's for two (well, at least two) reasons: one is to prevent things like this. The other is that in generally doing case-insensitive stuff for non-ASCII is language-dependent. For example, in English "i" and "I" are case-insensitively equal but in Turkish they are not: "i" is case-insensitively equal to "İ" (U+0130) and "I" is case-insensitively equal to "ı" (U+0131). So if you wanted to do full-unicode case-insensitive things you would need to decide what happens with <DİV> or <dıv> and whether it depends on the language of the page or what. Doing ASCII-only case-insensitivity sidesteps all that, while admittedly privileging English over other languages.
_wldu 6y ago
There's a reply in that Twitter Thread. Go will be switching to ASCII: "Thanks for the ping. Our policy is that anything that requires a proxy to forward an invalid Transfer-Encoding header is a vulnerability in the proxy, not in the server. We are planning to switch all net/http to ASCII strings functions anyway for defense in depth and correctness." - Filippo Valsorda, Go Security Lead
- lucideer 6y ago
  > Our policy is that anything that requires a proxy to forward an invalid Transfer-Encoding header is a vulnerability in the proxy It's great that they're fixing this anyway, but that's a bad policy.
  - jeff-davis 6y ago
    That was my initial reaction as well, but I'd like to hear some more explanation.
jd3 6y ago
I do love the simplicity of Go's UTF-8[0] approach, though this did also burn me when I was first learning the language and did not really understand the implication of it. I translated an x-face encoder from js to go for one of my introductory projects and initially couldn't get the correct output from `b64.StdEncoding.EncodeToString` due to the fact that x-face strings are single-byte ASCII ISO-8859-1 encoded[1]. [0]: https://blog.golang.org/strings https://blog.golang.org/strings [1]: https://stackoverflow.com/questions/43461679/escaping-of-hex-values-in-string-literals https://stackoverflow.com/questions/43461679/escaping-of-hex...
- jeff-davis 6y ago
  After reading [0], I was left with the question: why not just have a utf-8 string type in addition to a byte string type? Also, as a nit, I don't think iso-8859-1 is ASCII.
  - jd3 6y ago
    > why not just have a utf-8 string type in addition to a byte string type? I'm not a go expert, but my understanding is that go "strings" are really all just a series of bytes and it's the standard library functions which treat string operations as UTF-8 encoded by default. I think this reddit post[0] explains it nicely. I suppose the simplicity of "just work with the bytes and apply encoding when necessary" follows go's simplicity and composability/interface patterns well. > Also, as a nit, I don't think iso-8859-1 is ASCII. We're splitting hairs here, but ISO-8859-1 is the eight-bit extension to 7-bit ASCII, no? It includes the standard 128 ASCII characters + up to an additional 128 characters (191 in total for ISO-8859-1 Latin-1 Western European, iirc). [0]: https://old.reddit.com/r/golang/comments/a5v3fn/are_strings_utf8_by_default/ebprjpk/ https://old.reddit.com/r/golang/comments/a5v3fn/are_strings_...
fastglass 6y ago
I think we're going to see A LOT more problems like this, with UTF, in the future.
arpa 6y ago
The real wtf here is a separate K for Kelvin, a separate N for Newton, etc. Why on gods' greenish earth do we need something that looks like a duck, quacks like a duck, but actually is not a duck but an eldritch horror?
- azornathogron 6y ago
  One of the design criteria for Unicode was to guarantee lossless round-trip conversion through Unicode for a bunch of encodings that Unicode superceded. If one of those encodings contained both a K and a K-for-Kelvin, then Unicode also had to allocate two codepoints for it. I don't know if that's the case for these characters in particular, but it is one possible explanation. Supporting round-trip conversion may sound pointless now that Unicode is ubiquitous, but it was probably necessary to make gradual adoption practical. Ref: https://en.m.wikipedia.org/wiki/Unicode_compatibility_characters https://en.m.wikipedia.org/wiki/Unicode_compatibility_charac...
  - arpa 6y ago
    That could explain it, but it doesn't make it any less crazy. I'm tempted to use K-for-Kelvin in my code now, just to fuck with anyone who works with a worse IDE or tries finding things with grep.

This article (/quasi-essay) was a good read : https://fasterthanli.me/articles/aiming-for-correctness-with-types https://fasterthanli.me/articles/aiming-for-correctness-with...

dgellow 6y ago
Damn that was a VERY long read, but also a fantastic one. Thanks for sharing!

panpanna 6y ago

Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. Now, what are the security implications of this? (beside the chunk thingy)

_-___________-_ 6y ago
> Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. > Now, what are the security implications of this? (beside the chunk thingy) Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context. Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
- panpanna 6y ago
  > That's a terrible reason. Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang. It leads to all kind of issues, not only security problems.
  - kerng 6y ago
    But that's a Golang issue then, not http
    - panpanna 6y ago
      I know, I am just trying to explain how this could happen :)
  - deleted 6y ago
    [deleted]
  - _-___________-_ 6y ago
    Yes, but this simply means that header values are not (Go) strings, they're byte arrays. Many languages represent them this way, with a (fallible) function to get the string value iff the byte array contains valid UTF-8. The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
  - 0xjnml 6y ago
    1) Go string type is specified as a sequence of bytes. Full stop. Any encoding, binary blob, string does not care. Particular stdlib string functions _do_ care, expect UTF-8 encoding and document the fact. Other stdlib string function work with the sequence as a sequence. 3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals. 2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
- yokaze 6y ago
  > That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1 https://tools.ietf.org/html/rfc822#section-3.1
  - Thaxll 6y ago
    nope: https://tools.ietf.org/html/rfc7230#page-25 https://tools.ietf.org/html/rfc7230#page-25
  - vaduz 6y ago
    > Actually, they are explicitly specified as ASCII They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2. https://tools.ietf.org/html/rfc7230#section-3.2.4 https://tools.ietf.org/html/rfc7230#section-3.2.4
albinowax_ 6y ago
I think it's worth spelling out that 'chunk thingy' can lead to complete site takeover. For example, using this behaviour I was able to get persistent control of PayPal's login page. That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015 https://news.ycombinator.com/item?id=25858015
- panpanna 6y ago
  Do you have any pointers to the PayPal hack?
  - albinowax_ 6y ago
    Yep, the original reports are here: https://hackerone.com/reports/488147 https://hackerone.com/reports/488147 https://hackerone.com/reports/510152 https://hackerone.com/reports/510152 And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request...
    - panpanna 6y ago
      This is awesome, thanks!
    - Uberphallus 6y ago
      Wow, excellent work!
superjan 6y ago
It could get hairy when here is also another parser that has different assumptions. They don’t see the same URL. Psychic paper [0] comes to mind. [0] https://daringfireball.net/linked/2020/05/02/psychic-paper https://daringfireball.net/linked/2020/05/02/psychic-paper
jlouis 6y ago
The string type is utf8. You would think an old http header is a string. It's not. It looks like a string though :)
- papaf 6y ago
  The string type is utf8. Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types https://golang.org/ref/spec#String_types However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.
kerng 6y ago
Burp employee made like 70k dollars in bug bounty some time back with smuggling attacks on various sites
Thaxll 6y ago
In Go a string is an immutable byte slice not UTF-8.
- jrimbault 6y ago
  I'll also play along the role of the pedant : > In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8. Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
  - papaf 6y ago
    Its a common mistake to assume strings are UTF-8 but it is a mistake. This is not a defence of Go but more of a warning to Go programmers.
    - adonovan 6y ago
      Exactly. It's a convention, not a law of physics. If a string's UTF-8 validity is important to the correctness of your function, check it. It's not expensive.
user-the-name 6y ago
The real problem is that Go does not actually have a string type. It only has byte strings which can accidentally also be text strings encoded in utf-8. This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software. Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
- naniwaduni 6y ago
  The fact is that there is no correct "text string" type, because there are no valid operations on so-called text strings in general.
  - user-the-name 6y ago
    That... makes no sense? Of course there are a lot of valid operations on text strings?

albinowax_ 6y ago

If you're wondering what kinds of attacks this enables, I think the primary risk is HTTP Request Smuggling https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... But there's a bunch of hard-to-quantify side-risks based on corner-cases that could lead to stuff like request header injection, server-side parameter pollution, and host-header attacks. I'm excited to see what shows up.

dcomp 6y ago

Technically isn't incorrect but is a violation of SHOULD. I still think a SHOULD should be a requirement for a general purpose library. With reasons given for not following it. Edit: And with HTTP having case-insensitive matching (which is most likely broken in lots of hand written implementations). This is rife with the possibility for errors Taken from RFC 7230 3.2.4 Field parsing Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.

mytailorisrich 6y ago
> A recipient SHOULD treat other octets in field content (obs-text) as opaque data. This is not really a 'should', IMHO, because fields are defined as OCTETS, iirc. Based on that, a compliant and robust implementation must treat them as opaque data.
- dcomp 6y ago
  I still fight with case-sensitive matching breaking HTTP2 -> HTTP1.1 proxies
- vaduz 6y ago
  RFC 7230 makes it a point not to make it a MUST as that would make unknown number of existing applications non-compliant with HTTP/1.1-as-redefined. They are free to treat the incoming headers as ISO-8859-1 8bit instead of dropping to 7bit US-ASCII.
  - mytailorisrich 6y ago
    RFC 2616 defined header fields as OCTETs, and regarding this change RFC 7230 states: > Non-US-ASCII content in header fields and the reason phrase has been obsoleted and made opaque (the TEXT rule was removed). RFC 2616: field-value = ( field-content | LWS ) field-content = <the OCTETs making up the field-valu and consisting of either TEXT or combinations of token, separators, and quoted-string> Hence to me fields must be treated as opaque data for backward compatibility and robustness. If anything, existing applications that are compliant with RFC 2616 already do that, right? ;)

saurik 6y ago

Every time someone creates a new programming language, the world builds yet another pile of broken HTTP implementations--clients and servers--because everyone refuses to read the specifications, thinking "none of this is hard". I remember dealing with some early Ruby web servers, 15 years ago, and was just shocked at how broken they were despite being used by everyone in production! In fact, there is a lot of nuance to HTTP, and as a civilization we seemed doomed to reliving the same mistakes over and over again every few years when the entire stack gets rewritten from scratch :/. So, I thereby was neither surprised to read that the Go HTTP server did this obviously-wrong thing... NOR was I particularly surprised to read the person complaining about it also didn't seem to understand the correct behavior, either (though I do appreciate this is a bit "unfair"--I mean, it's a tweet ;P--but I think still makes for an interesting point here) :/. I have dealt with so many broken HTTP clients and servers over the years written by people who probably also never read the manual that came with their microwave because "documentation is for idiots" or whatever, that I have just become demoralized about it :(. A few months ago, I was complaining about this, with a short list of stuff everyone tended to get wrong, and this HTTP header field encoding issue was on my shortlist :(. https://news.ycombinator.com/item?id=24835617 https://news.ycombinator.com/item?id=24835617 > FWIW, I have encountered tons of issues in peoples' "this is easy, right?" implementations of 100 Continue (including in some at-the-time major Ruby HTTP servers), non-ASCII encoding in HTTP headers (which has a ton of subtle rules, and just throwing around UTF-8 is not correct), and chunked Transfer-Encoding (though I can't remember what was wrong); further, people often don't even realize HTTP trailers exist at all, which is a bit infuriating :/. (Note, though, that I am still 100% on the side of the anti-HTTP/2 camp in this thread.) The core problem is that somehow people think they can implement a spec without even, you know, reading the spec, which is so far past hubris as to have gone through incompetence into obvious negligence. AFAIK, HTTP header fields are (in their generality) specified to be read as binary octets... but then are restricted to using ISO-8859-1 with the premise you are supposed to run them through a MIME parser? Like, some fields have strict formats, so you can't do the full decoding until you know what you are working with--and so I would argue an HTTP library should give you the raw binary data (though decoding them through ISO-8859-1 first isn't supposed to be "wrong")--but for fields with user-displayable text (including HTTP status messages!) you can seriously embed "Q" encoded strings (of all things) if you want, a la RFC 2047. You can't assume a field is restricted to ASCII too early in the parsing process. https://tools.ietf.org/html/rfc2047 https://tools.ietf.org/html/rfc2047

ipython 6y ago

Isn’t that more a Unicode normalization problem than utf8?

albinowax_ 6y ago
HTTP headers are generally treated as ASCII, and Golang's decision to treat them as UTF-8 opens the gate to a range of issues, including the linked unicode normalization attack.

IshKebab 6y ago

One of the hazards of text-based protocols.

userbinator 6y ago
Indeed --- if HTTP was a binary protocol, with headers and their values being defined constant integers, this would definitely not happen. No one would confuse 1 with 2, and implementations would probably be quite a bit more efficient... but the "lamers" really want their pile of leaky abstractions because they can't be bothered using a hex editor. </s> Nevermind the fact that some parts of HTTP are really just binary written as text, and now even most non-developers know what a 404 is...

DethNinja 6y ago

I didn’t know this kind of attack before but wouldn't any sane backend system confirm the permissions of requesting client first before handing out admin pages? I mean you need to have a client-id and their permissions for any outside request coming to server. I’m using Golang in my webserver but it wouldn’t hand-out secret pages as requesting client wouldn’t have authority to do so. I feel like Golang's UTF8 handling is the least of the worries for these kinds of attacks, it points out that entire backend system was constructed wrongly, number one rule of any backend systems: You cannot trust the client.

pram 6y ago
I agree. Seems like something that is easily defeated using, for example, a JWT

vaduz 6y ago

There are sequences in ISO-8859-1 that will cause UTF8 application to balk if they are read without conversion - sometimes in a part that is well away from the input itself. A common example "órny", part of a number of Polish place and personal names. ISO 8859-1 (and -2, and others) form would be F3 72 6E 79. UTF-8 treats F3 as a start of a 4 byte sequence where each subsequent character is in 80 - BF range (as specified in RFC 3629): UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) / %xF4 %x80-8F 2( UTF8-tail ) UTF8-tail = %x80-BF This nicely causes an exception in both Java and Python, for instance, if they were to do what Go does here - and there is nothing anyone from adding an X- header to the request or response to try just that.

treeman79 6y ago

Spent trying to parse old encoded documents. Spent a month doing a deep dive into internals of bytes and formats. Finally just wrote code to loop through all possible encodings and ways to change encodings. Wait until a match against a known string was found. Proper encodings found in a few minutes.

burnthrow 6y ago
That approach sounds like it would yield many false positives.

userbinator 6y ago

This is the cost of "extraneous hidden complexity", or as I like to think of it, "abstraction hides bugs". As others here have noted, it is all too tempting to use a language whose string type actually does something very fancy, and be unaware of it. Most of the time it "only" causes inefficiency. I don't remember if the HTTP spec has it, but I clearly remember the HTML spec has some parts that explicitly say "case-insensitive in the ASCII range only" (i.e. 61-7a = 41-5a), presumably to prevent things like this from happening.

dgellow 6y ago
From what I got from the spec, HTTP headers are supposed to be ASCII or opaque bytes.
bzbarsky 6y ago
It's for two (well, at least two) reasons: one is to prevent things like this. The other is that in generally doing case-insensitive stuff for non-ASCII is language-dependent. For example, in English "i" and "I" are case-insensitively equal but in Turkish they are not: "i" is case-insensitively equal to "İ" (U+0130) and "I" is case-insensitively equal to "ı" (U+0131). So if you wanted to do full-unicode case-insensitive things you would need to decide what happens with <DİV> or <dıv> and whether it depends on the language of the page or what. Doing ASCII-only case-insensitivity sidesteps all that, while admittedly privileging English over other languages.

_wldu 6y ago

There's a reply in that Twitter Thread. Go will be switching to ASCII: "Thanks for the ping. Our policy is that anything that requires a proxy to forward an invalid Transfer-Encoding header is a vulnerability in the proxy, not in the server. We are planning to switch all net/http to ASCII strings functions anyway for defense in depth and correctness." - Filippo Valsorda, Go Security Lead

lucideer 6y ago
> Our policy is that anything that requires a proxy to forward an invalid Transfer-Encoding header is a vulnerability in the proxy It's great that they're fixing this anyway, but that's a bad policy.
- jeff-davis 6y ago
  That was my initial reaction as well, but I'd like to hear some more explanation.

jd3 6y ago

I do love the simplicity of Go's UTF-8[0] approach, though this did also burn me when I was first learning the language and did not really understand the implication of it. I translated an x-face encoder from js to go for one of my introductory projects and initially couldn't get the correct output from `b64.StdEncoding.EncodeToString` due to the fact that x-face strings are single-byte ASCII ISO-8859-1 encoded[1]. [0]: https://blog.golang.org/strings https://blog.golang.org/strings [1]: https://stackoverflow.com/questions/43461679/escaping-of-hex-values-in-string-literals https://stackoverflow.com/questions/43461679/escaping-of-hex...

jeff-davis 6y ago
After reading [0], I was left with the question: why not just have a utf-8 string type in addition to a byte string type? Also, as a nit, I don't think iso-8859-1 is ASCII.
- jd3 6y ago
  > why not just have a utf-8 string type in addition to a byte string type? I'm not a go expert, but my understanding is that go "strings" are really all just a series of bytes and it's the standard library functions which treat string operations as UTF-8 encoded by default. I think this reddit post[0] explains it nicely. I suppose the simplicity of "just work with the bytes and apply encoding when necessary" follows go's simplicity and composability/interface patterns well. > Also, as a nit, I don't think iso-8859-1 is ASCII. We're splitting hairs here, but ISO-8859-1 is the eight-bit extension to 7-bit ASCII, no? It includes the standard 128 ASCII characters + up to an additional 128 characters (191 in total for ISO-8859-1 Latin-1 Western European, iirc). [0]: https://old.reddit.com/r/golang/comments/a5v3fn/are_strings_utf8_by_default/ebprjpk/ https://old.reddit.com/r/golang/comments/a5v3fn/are_strings_...

fastglass 6y ago

I think we're going to see A LOT more problems like this, with UTF, in the future.

arpa 6y ago

The real wtf here is a separate K for Kelvin, a separate N for Newton, etc. Why on gods' greenish earth do we need something that looks like a duck, quacks like a duck, but actually is not a duck but an eldritch horror?

azornathogron 6y ago
One of the design criteria for Unicode was to guarantee lossless round-trip conversion through Unicode for a bunch of encodings that Unicode superceded. If one of those encodings contained both a K and a K-for-Kelvin, then Unicode also had to allocate two codepoints for it. I don't know if that's the case for these characters in particular, but it is one possible explanation. Supporting round-trip conversion may sound pointless now that Unicode is ubiquitous, but it was probably necessary to make gradual adoption practical. Ref: https://en.m.wikipedia.org/wiki/Unicode_compatibility_characters https://en.m.wikipedia.org/wiki/Unicode_compatibility_charac...
- arpa 6y ago
  That could explain it, but it doesn't make it any less crazy. I'm tempted to use K-for-Kelvin in my code now, just to fuck with anyone who works with a worse IDE or tries finding things with grep.