3 ms·

But Unicode code points aren't supposed to represent semantics, they're supposed to represent... well, "abstract characters" that are either glyphs or get combi

But Unicode code points aren't supposed to represent semantics, they're supposed to represent... well, "abstract characters" that are either glyphs or get combined into glyphs, which can have multiple (ambiguous) semantic meanings. That's why there aren't two different period characters to represent the end of a sentence vs. a decimal point, or two different em dashes where one represents a pause while the other comes at the end of dialog to indicate the sentence was interrupted (literally the opposite of a pause, it's being cut off). So since the apostrophe and the right single quote are visually identical, Unicode stays consistent in recommending that they be the same character. The name Unicode gives to a character is intended to represent one of its semantic meanings, not all of them. (Unicode does have plenty of visually identical characters, but they generally belong to totally different languages, like the English "o" and the Greek omicron "ο".)

thangalin 2y ago
Thanks for shedding a little more light. Ignoring the semantics, in this case, between an apostrophe and a right single quote has resulted in many documents containing information that can not be parsed unambiguously because we have to pick a glyph and doing so with an ambiguously defined glyph loses contextual information. As a side-effect, since GPTs are based on the examples we give, they can't encode the proper punctuation for many phrases that use British English quotation mark styles, making them unable to "curl" the quotation mark properly. For example, none can curl this paragraph correctly: ''E's got a 'ittle box 'n a big 'un,' she said, 'wit' th' 'ittle 'un 'bout 2'×6". An' no, y'ain't cryin' on th' "soap box" to me no mo, y'hear. 'Cause it 'tweren't ever a spec o' fun!' I says to my frien'. The other downside to using ' or ' is that most fonts treat them as straight quotes, making for "improper" English typography when typeset into a book.
- crazygringo 2y ago
  > many documents containing information that can not be parsed unambiguously Well, and I'd suggest the unambiguous information was usually never there in the first place. It's less of an encoding problem, and more of an input "problem". People type either a single quote/apostrophe, or a double quote, and let smart quotes sort it out. And sure, smart quotes will fail spectacularly with your spectacularly pathological example! Heck, it took me a few seconds to figure out what on earth was going on with the first 5 characters. :) Your example would usually be typeset properly in a physical published book because it's done with professionals manually reviewing the typography. Just throw it in the bucket of hyphens vs. minuses vs. dashes em and en, x's versus multiplication signs... our symbols are full of ambiguities, it's not just apostrophes.
  - thangalin 2y ago
    > let smart quotes sort it out. Smart quotes fail in simple cases, too. https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/test/resources/com/whitemagicsoftware/keenquotes/texts https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/tes... I've developed a lexer/parser that can disambiguate most cases, but wow was it a chore to write. > Well, and I'd suggest the unambiguous information was usually never there in the first place. Interesting. Isn't the text ambiguous because the glyphs lack the semantics to capture the usage of apostrophes versus closing single quotes? It's a Catch-22, isn't it? If UNICODE had semantics for apostrophes versus right single quotes, then our documents would be unambiguous. But we can't make them unambiguous because UNICODE doesn't capture these semantics.
    - crazygringo 2y ago
      > If UNICODE had semantics for apostrophes versus right single quotes, then our documents would be unambiguous. No -- as I said before, it's an input problem before anything else. There aren't separate keys for apostrophe and right single quote on the keyboard. We don't even have separate keys for left and right quotes. So even if there were encodings for them, they wouldn't be used correctly. They'd be used correctly about as often as people type a proper minus sign rather than a hyphen for subtraction, which is almost never.
      - thangalin 2y ago
        > it's an input problem before anything else I see where we have our wires crossed. I'm not considering the input problem because my software (KeenQuotes) parses the source document's apostrophes into their correct semantics (99.9% of the time). My issue is that, having discerned the correct English single quotation mark (straight, apostrophe, or closing), I have no way of encoding it into a document that retains the semantics while typesetting it using common fonts (to match double quotes). My point is that if UNICODE had a way of capturing the semantics, it would at least be technically possible to create unambiguous documents, input notwithstanding. Matthew Butterick, a typographer, states, "I’ve never seen any LaTeX-created documentation that’s gotten this right": https://practicaltypography.com/straight-and-curly-quotes.html https://practicaltypography.com/straight-and-curly-quotes.ht... I sent him a screenshot showing my software typesetting the quotation marks properly, albeit with a document that has incorrect semantics (as per our discussion): https://i.ibb.co/p3TM7QM/curly-quotes.png https://i.ibb.co/p3TM7QM/curly-quotes.png

thangalin 2y ago

Thanks for shedding a little more light. Ignoring the semantics, in this case, between an apostrophe and a right single quote has resulted in many documents containing information that can not be parsed unambiguously because we have to pick a glyph and doing so with an ambiguously defined glyph loses contextual information. As a side-effect, since GPTs are based on the examples we give, they can't encode the proper punctuation for many phrases that use British English quotation mark styles, making them unable to "curl" the quotation mark properly. For example, none can curl this paragraph correctly: ''E's got a 'ittle box 'n a big 'un,' she said, 'wit' th' 'ittle 'un 'bout 2'×6". An' no, y'ain't cryin' on th' "soap box" to me no mo, y'hear. 'Cause it 'tweren't ever a spec o' fun!' I says to my frien'. The other downside to using ' or ' is that most fonts treat them as straight quotes, making for "improper" English typography when typeset into a book.

crazygringo 2y ago
> many documents containing information that can not be parsed unambiguously Well, and I'd suggest the unambiguous information was usually never there in the first place. It's less of an encoding problem, and more of an input "problem". People type either a single quote/apostrophe, or a double quote, and let smart quotes sort it out. And sure, smart quotes will fail spectacularly with your spectacularly pathological example! Heck, it took me a few seconds to figure out what on earth was going on with the first 5 characters. :) Your example would usually be typeset properly in a physical published book because it's done with professionals manually reviewing the typography. Just throw it in the bucket of hyphens vs. minuses vs. dashes em and en, x's versus multiplication signs... our symbols are full of ambiguities, it's not just apostrophes.
- thangalin 2y ago
  > let smart quotes sort it out. Smart quotes fail in simple cases, too. https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/test/resources/com/whitemagicsoftware/keenquotes/texts https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/tes... I've developed a lexer/parser that can disambiguate most cases, but wow was it a chore to write. > Well, and I'd suggest the unambiguous information was usually never there in the first place. Interesting. Isn't the text ambiguous because the glyphs lack the semantics to capture the usage of apostrophes versus closing single quotes? It's a Catch-22, isn't it? If UNICODE had semantics for apostrophes versus right single quotes, then our documents would be unambiguous. But we can't make them unambiguous because UNICODE doesn't capture these semantics.
  - crazygringo 2y ago
    > If UNICODE had semantics for apostrophes versus right single quotes, then our documents would be unambiguous. No -- as I said before, it's an input problem before anything else. There aren't separate keys for apostrophe and right single quote on the keyboard. We don't even have separate keys for left and right quotes. So even if there were encodings for them, they wouldn't be used correctly. They'd be used correctly about as often as people type a proper minus sign rather than a hyphen for subtraction, which is almost never.
    - thangalin 2y ago
      > it's an input problem before anything else I see where we have our wires crossed. I'm not considering the input problem because my software (KeenQuotes) parses the source document's apostrophes into their correct semantics (99.9% of the time). My issue is that, having discerned the correct English single quotation mark (straight, apostrophe, or closing), I have no way of encoding it into a document that retains the semantics while typesetting it using common fonts (to match double quotes). My point is that if UNICODE had a way of capturing the semantics, it would at least be technically possible to create unambiguous documents, input notwithstanding. Matthew Butterick, a typographer, states, "I’ve never seen any LaTeX-created documentation that’s gotten this right": https://practicaltypography.com/straight-and-curly-quotes.html https://practicaltypography.com/straight-and-curly-quotes.ht... I sent him a screenshot showing my software typesetting the quotation marks properly, albeit with a document that has incorrect semantics (as per our discussion): https://i.ibb.co/p3TM7QM/curly-quotes.png https://i.ibb.co/p3TM7QM/curly-quotes.png