7 ms·
Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. Now, what are the security implications of this? (beside the chunk
by panpanna 6y ago
Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8.
Now, what are the security implications of this?
(beside the chunk thingy)
- _-___________-_ 6y ago> Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8. That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. > Now, what are the security implications of this? (beside the chunk thingy) Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context. Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
- panpanna 6y ago> That's a terrible reason. Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang. It leads to all kind of issues, not only security problems.
- kerng 6y agoBut that's a Golang issue then, not http
- panpanna 6y agoI know, I am just trying to explain how this could happen :)
- deleted 6y ago[deleted]
- _-___________-_ 6y agoYes, but this simply means that header values are not (Go) strings, they're byte arrays. Many languages represent them this way, with a (fallible) function to get the string value iff the byte array contains valid UTF-8. The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
- 0xjnml 6y ago1) Go string type is specified as a sequence of bytes. Full stop. Any encoding, binary blob, string does not care. Particular stdlib string functions _do_ care, expect UTF-8 encoding and document the fact. Other stdlib string function work with the sequence as a sequence. 3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals. 2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
- yokaze 6y ago> That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken. Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1 https://tools.ietf.org/html/rfc822#section-3.1
- Thaxll 6y agonope: https://tools.ietf.org/html/rfc7230#page-25 https://tools.ietf.org/html/rfc7230#page-25
- vaduz 6y ago> Actually, they are explicitly specified as ASCII They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2. https://tools.ietf.org/html/rfc7230#section-3.2.4 https://tools.ietf.org/html/rfc7230#section-3.2.4
- albinowax_ 6y agoI think it's worth spelling out that 'chunk thingy' can lead to complete site takeover. For example, using this behaviour I was able to get persistent control of PayPal's login page. That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015 https://news.ycombinator.com/item?id=25858015
- panpanna 6y agoDo you have any pointers to the PayPal hack?
- albinowax_ 6y agoYep, the original reports are here: https://hackerone.com/reports/488147 https://hackerone.com/reports/488147 https://hackerone.com/reports/510152 https://hackerone.com/reports/510152 And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request...
- panpanna 6y agoThis is awesome, thanks!
- Uberphallus 6y agoWow, excellent work!
- superjan 6y agoIt could get hairy when here is also another parser that has different assumptions. They don’t see the same URL. Psychic paper [0] comes to mind. [0] https://daringfireball.net/linked/2020/05/02/psychic-paper https://daringfireball.net/linked/2020/05/02/psychic-paper
- jlouis 6y agoThe string type is utf8. You would think an old http header is a string. It's not. It looks like a string though :)
- papaf 6y agoThe string type is utf8. Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types https://golang.org/ref/spec#String_types However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.
- kerng 6y agoBurp employee made like 70k dollars in bug bounty some time back with smuggling attacks on various sites
- Thaxll 6y agoIn Go a string is an immutable byte slice not UTF-8.
- jrimbault 6y agoI'll also play along the role of the pedant : > In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8. Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
- papaf 6y agoIts a common mistake to assume strings are UTF-8 but it is a mistake. This is not a defence of Go but more of a warning to Go programmers.
- adonovan 6y agoExactly. It's a convention, not a law of physics. If a string's UTF-8 validity is important to the correctness of your function, check it. It's not expensive.
- user-the-name 6y agoThe real problem is that Go does not actually have a string type. It only has byte strings which can accidentally also be text strings encoded in utf-8. This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software. Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
- naniwaduni 6y agoThe fact is that there is no correct "text string" type, because there are no valid operations on so-called text strings in general.
- user-the-name 6y agoThat... makes no sense? Of course there are a lot of valid operations on text strings?