4 ms·

> should be sent in Latin1 Do you have a source on that "should" part. Because the spec disagrees https://www.rfc-editor.org/rfc/rfc7230#section-3.2.4 https://

by ko27 3y ago

> should be sent in Latin1 Do you have a source on that "should" part. Because the spec disagrees https://www.rfc-editor.org/rfc/rfc7230#section-3.2.4 https://www.rfc-editor.org/rfc/rfc7230#section-3.2.4: > Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. In practice and by spec, HTTP headers should be ASCII encoded.

nicktelford 3y ago
ISO-8859-1 (aka. Latin-1) is a superset of ASCII, so all ASCII strings are also valid Latin-1 strings. The section you quoted actually suggests that implementations should support ISO-8859-1 to ensure compatibility with systems that use it.
- ko27 3y ago
  You should read it again > Newly defined header fields SHOULD limit their field values to US-ASCII octets ASCII octets! That means you SHOULD NOT send Latin1 encoded headers. The opposite of what pzmarzly was saying. I don't disagree Latin-1 being a superset of ASCII or having backward compatibility in mind, but that's not relevant to my response.
  - layer8 3y ago
    SHOULD is a recommendation, not a requirement, and it refers only to newly-defined header fields, not existing ones. The text implies that 8-bit characters in existing fields are to be interpreted as ISO-8859-1.
    - verst 3y ago
      There is a RFC (2119) that specifies what SHOULD means in RFCs: > SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. https://datatracker.ietf.org/doc/html/rfc2119 https://datatracker.ietf.org/doc/html/rfc2119
  - jart 3y ago
    Haven't you heard of Postel's Maxim? Web servers need to be able to receive and decode latin1 into utf-8 regardless of what the RFC recommends people send. The fact that it's going to become rarer over time to have the 8th bit set in headers, means you can write a simpler algorithm than what Lemire did that assumes an ASCII average case. https://github.com/jart/cosmopolitan/blob/755ae64e73ef5ef7d1fe97165705767eda49a3fd/net/http/decodelatin1.c#L33 https://github.com/jart/cosmopolitan/blob/755ae64e73ef5ef7d1... That goes 23 GB/s on my machine using just SSE2 (rather than AVX512). However it goes much slower if the text is full of european diacritics. Lemire's algorithm is better at decoding those.
    - HideousKojima 3y ago
      >Haven't you heard of Postel's Maxim? Otherwise known as "Making other people's incompetence and inability to implement a specification your problem." Just because it's a widely quoted maxim doesn't make it good advice.
missblit 3y ago
The spec may disagree, but webservers do sometimes send bytes outside the ASCII range, and the most sensible way to deal with that on the receiving side is still by treating them as latin1 to match (last I checked) what browsers do with it. I do agree that latin1 headers shouldn't be _sent_ out though.

by ko27 3y ago

nicktelford 3y ago

ISO-8859-1 (aka. Latin-1) is a superset of ASCII, so all ASCII strings are also valid Latin-1 strings. The section you quoted actually suggests that implementations should support ISO-8859-1 to ensure compatibility with systems that use it.

ko27 3y ago
You should read it again > Newly defined header fields SHOULD limit their field values to US-ASCII octets ASCII octets! That means you SHOULD NOT send Latin1 encoded headers. The opposite of what pzmarzly was saying. I don't disagree Latin-1 being a superset of ASCII or having backward compatibility in mind, but that's not relevant to my response.
- layer8 3y ago
  SHOULD is a recommendation, not a requirement, and it refers only to newly-defined header fields, not existing ones. The text implies that 8-bit characters in existing fields are to be interpreted as ISO-8859-1.
  - verst 3y ago
    There is a RFC (2119) that specifies what SHOULD means in RFCs: > SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. https://datatracker.ietf.org/doc/html/rfc2119 https://datatracker.ietf.org/doc/html/rfc2119
- jart 3y ago
  Haven't you heard of Postel's Maxim? Web servers need to be able to receive and decode latin1 into utf-8 regardless of what the RFC recommends people send. The fact that it's going to become rarer over time to have the 8th bit set in headers, means you can write a simpler algorithm than what Lemire did that assumes an ASCII average case. https://github.com/jart/cosmopolitan/blob/755ae64e73ef5ef7d1fe97165705767eda49a3fd/net/http/decodelatin1.c#L33 https://github.com/jart/cosmopolitan/blob/755ae64e73ef5ef7d1... That goes 23 GB/s on my machine using just SSE2 (rather than AVX512). However it goes much slower if the text is full of european diacritics. Lemire's algorithm is better at decoding those.
  - HideousKojima 3y ago
    >Haven't you heard of Postel's Maxim? Otherwise known as "Making other people's incompetence and inability to implement a specification your problem." Just because it's a widely quoted maxim doesn't make it good advice.

missblit 3y ago

The spec may disagree, but webservers do sometimes send bytes outside the ASCII range, and the most sensible way to deal with that on the receiving side is still by treating them as latin1 to match (last I checked) what browsers do with it. I do agree that latin1 headers shouldn't be _sent_ out though.