6 ms·

Guaranteed, reST is more feature-complete and extension-friendly, but it is simply unusable for me because it wasn't designed for agglutinative languages like K

Guaranteed, reST is more feature-complete and extension-friendly, but it is simply unusable for me because it wasn't designed for agglutinative languages like Korean. Markdown is much better in this case (though CommonMark has an annoying edge case [1]). [1] https://talk.commonmark.org/t/foo-works-but-foo-fails/2528 https://talk.commonmark.org/t/foo-works-but-foo-fails/2528

do_not_redeem 1y ago
What do you mean not designed for Korean? It's just unicode. If there's some situation where RST isn't parsing inline markup, you can write the role explicitly like this: this is **bold** text this is :strong:`bold` text
- rune-space 1y ago
  But you can’t say: thisis:strong:`bold`text Whereas the equivalent is perfectly fine in markdown. Falsehoods programmers believe about written language: whitespace is used to separate atomic sequences of runes.
  - thaumasiotes 1y ago
    > Falsehoods programmers believe about written language: whitespace is used to separate atomic sequences of runes. Really? That isn't just untrue of written language in general. It's untrue of every individual written language in specific. You can't even clearly define what an "atomic sequence of glyphs" is.
    - matja 1y ago
      > You can't even clearly define what an "atomic sequence of glyphs" is. Kinda. Grapheme cluster breaks are defined in Unicode, but they have all the baggage and edge-cases you'd expect from human languages evolving over time, so they can be encoded in as a few as a thousand rules : https://github.com/unicode-org/icu/tree/main/icu4c/source/data/brkitr/rules https://github.com/unicode-org/icu/tree/main/icu4c/source/da...
    - rune-space 1y ago
      Which makes one wonder why REST puts so much weight on them being divided by WS!
- lifthrasiir 1y ago
  reST inline syntaxes are pretty much word-based, which doesn't work very well with agglutinative languages. For example if you want to apply a markup to "이 페이지" in "이 페이지는 ..." (lit. This page in This page is ...), you need to do `*이 페이지*\ 는 ...` AFAIK. That would happen every single time affixes are used, and affixes are extremely frequent in such languages.
  - do_not_redeem 1y ago
    Oh I see, you're talking about this: thisis\ **bold**\ text thisis\ :strong:`bold`\ text It's possible, but you're right, definitely more awkward than markdown.
chrismorgan 1y ago
reStructuredText and Markdown both have a bad habit of clevernesses that fall down—just in different areas. Both do at least some degree of only matching delimiters at word boundaries. I consider that to be a huge mistake. reStructuredText falls for it, but has a universally-applicable workaround (backslash-space as a separator—note that it is not an escaped space, as you might reasonably expect: it’s special-cased to expand to nothing but a syntax separator). Markdown falls for it inconsistently, which as a user of languages that divide words with spaces, is honestly worse. Its rules are more nuanced, which is generally a bad thing, because it makes it harder to build the appropriate mental model. It was also wildly underspecified, though that’s mostly settled now. For many years, Stack Overflow used at least two, I think three but I can’t remember where the third would have been, mutually-incompatible engines, and underscores and mid-word formatting were a total mess. Python in particular suffered—for many years, in comments it was impossible to get plain-text (i.e. not `-wrapped code) __init__. In CommonMark, _abc_ and *abc* get you abc, but a*b*c gets you abc while a_b_c gets you a_b_c. That’s an admission of failure in syntax. Hmm… I hadn’t thought of this, but I suppose that makes _ basically untenable in languages with no word separator. Interesting argument against Prettier, which has a badly broken Markdown mode¹, and which insists on _ for emphasis, not *. In my own lightweight markup language I’ve been steadily making and using for my own stuff for the last five years or so, there’s nothing about word boundaries. a*b*c is abc, and if a dialect² defined _ as emphasis, a_b_c would be abc. Another example of the cleverness problem in reStructuredText is how hard wrapping is handled. https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#enumerated-lists:~:text=enumerators.-,For,dude.,-Caution https://docutils.sourceforge.io/docs/ref/rst/restructuredtex... is a good example of how badly wrong this can go. (Markdown has related issues, but a little more constrained. A mid-paragraph line starting with “1. ” or “- ”—both plausible, and the latter certain to occur eventually if you use - as a dash—will start a list.) The solution here is to reject column-based hard-wrapping as a terrible idea. Yes, this is a case where the markup language should tell people “you’re doing it wrong”, because otherwise the markup language will either mangle your content, or become bad; or more likely both. Meanwhile in Markdown, it tries to be clever around specific HTML tags and just becomes hard to predict. —⁂— ¹ Prettier’s Markdown formatting is known to mangle content, particularly around underscores and asterisks, and they haven’t done anything about it. The first time I accidentally used it it deleted the rest of a file after some messy bad emphasis stuff from a WYSIWYG HTML → Markdown conversion. That was when I discovered .prettierignore is almost completely broken, too. I came away no longer just unimpressed with some of Prettier’s opinions, but severely unimpressed with the rest of it technically. Why they haven’t disabled it until such things are fixed, I don’t know. ² There’s very little fundamental syntax in it: line break, indent and parsing CSS Counter Styles is about it. The rest is all defined in dialects, for easy extension.
thaumasiotes 1y ago
> reST is more feature-complete and extension-friendly, but it is simply unusable for me because it wasn't designed for agglutinative languages like Korean. How does whether you think of the language as agglutinative affect the usability of reST? The biggest problem that occurs to me is that there isn't really a conceptual difference between an "agglutinative" language in which you have very long words expressing complex meanings, and an "isolating" language in which the same syllables occur in the same order with the same meaning but are thought of on a Platonic level as being all independent words. This is because an "agglutinative" language is one in which syntax markers are more or less independent of any other syntax markers that may apply to the same word†, which means it's always possible by definition to consider those markers to be "words" themselves. Would your problems be solved if you viewed what you had considered "long" Korean words as instead being several short words in a row? What difficulties does agglutination present? † Compare: https://glossary.sil.org/term/agglutinative-language https://glossary.sil.org/term/agglutinative-language > An agglutinative language is a language in which words are made up of a linear sequence of distinct morphemes and each component of meaning is represented by its own morpheme. https://glossary.sil.org/term/isolating-language https://glossary.sil.org/term/isolating-language > An isolating language is a language in which almost every word consists of a single morpheme.
- lifthrasiir 1y ago
  > This is because an "agglutinative" language is one in which syntax markers are more or less independent of any other syntax markers that may apply to the same word†, which means it's always possible by definition to consider those markers to be "words" themselves. I think SIL's definition is, while robust, not the usual definition because English can be regarded as agglutinative in this definition. This is particularly visible from the statement that most European languages are somewhat fusional [1], which is okay under their definitions but not the usual way we think of English. In my understanding, the analyticity is a spectrum and highly analytic languages with most (but not necessarily all) words containing just one morpheme are said to be isolating. Words in agglutinative languages can be, but not necessarily have to be, analyzed as a main morpheme ("word") with dependent morphemes attached ("affixes"). Polysynthetic languages go further by allowing multiple main morphemes in one word. As languages tend to become synthetic (as opposed to analytic), the space-separated "word" is less useful [2] and segmentation gets harder and harder. reST's failure to support those languages is all about a bad assumption about segmentation. [1] https://glossary.sil.org/term/fusional-language https://glossary.sil.org/term/fusional-language [2] So much that several agglutinative languages---in which space-separated words can still be useful---don't even think about spacing, e.g. Japanese.

do_not_redeem 1y ago

What do you mean not designed for Korean? It's just unicode. If there's some situation where RST isn't parsing inline markup, you can write the role explicitly like this: this is **bold** text this is :strong:`bold` text

rune-space 1y ago
But you can’t say: thisis:strong:`bold`text Whereas the equivalent is perfectly fine in markdown. Falsehoods programmers believe about written language: whitespace is used to separate atomic sequences of runes.
- thaumasiotes 1y ago
  > Falsehoods programmers believe about written language: whitespace is used to separate atomic sequences of runes. Really? That isn't just untrue of written language in general. It's untrue of every individual written language in specific. You can't even clearly define what an "atomic sequence of glyphs" is.
  - matja 1y ago
    > You can't even clearly define what an "atomic sequence of glyphs" is. Kinda. Grapheme cluster breaks are defined in Unicode, but they have all the baggage and edge-cases you'd expect from human languages evolving over time, so they can be encoded in as a few as a thousand rules : https://github.com/unicode-org/icu/tree/main/icu4c/source/data/brkitr/rules https://github.com/unicode-org/icu/tree/main/icu4c/source/da...
  - rune-space 1y ago
    Which makes one wonder why REST puts so much weight on them being divided by WS!
lifthrasiir 1y ago
reST inline syntaxes are pretty much word-based, which doesn't work very well with agglutinative languages. For example if you want to apply a markup to "이 페이지" in "이 페이지는 ..." (lit. This page in This page is ...), you need to do `*이 페이지*\ 는 ...` AFAIK. That would happen every single time affixes are used, and affixes are extremely frequent in such languages.
- do_not_redeem 1y ago
  Oh I see, you're talking about this: thisis\ **bold**\ text thisis\ :strong:`bold`\ text It's possible, but you're right, definitely more awkward than markdown.

chrismorgan 1y ago

reStructuredText and Markdown both have a bad habit of clevernesses that fall down—just in different areas. Both do at least some degree of only matching delimiters at word boundaries. I consider that to be a huge mistake. reStructuredText falls for it, but has a universally-applicable workaround (backslash-space as a separator—note that it is not an escaped space, as you might reasonably expect: it’s special-cased to expand to nothing but a syntax separator). Markdown falls for it inconsistently, which as a user of languages that divide words with spaces, is honestly worse. Its rules are more nuanced, which is generally a bad thing, because it makes it harder to build the appropriate mental model. It was also wildly underspecified, though that’s mostly settled now. For many years, Stack Overflow used at least two, I think three but I can’t remember where the third would have been, mutually-incompatible engines, and underscores and mid-word formatting were a total mess. Python in particular suffered—for many years, in comments it was impossible to get plain-text (i.e. not `-wrapped code) __init__. In CommonMark, _abc_ and *abc* get you abc, but a*b*c gets you abc while a_b_c gets you a_b_c. That’s an admission of failure in syntax. Hmm… I hadn’t thought of this, but I suppose that makes _ basically untenable in languages with no word separator. Interesting argument against Prettier, which has a badly broken Markdown mode¹, and which insists on _ for emphasis, not *. In my own lightweight markup language I’ve been steadily making and using for my own stuff for the last five years or so, there’s nothing about word boundaries. a*b*c is abc, and if a dialect² defined _ as emphasis, a_b_c would be abc. Another example of the cleverness problem in reStructuredText is how hard wrapping is handled. https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#enumerated-lists:~:text=enumerators.-,For,dude.,-Caution https://docutils.sourceforge.io/docs/ref/rst/restructuredtex... is a good example of how badly wrong this can go. (Markdown has related issues, but a little more constrained. A mid-paragraph line starting with “1. ” or “- ”—both plausible, and the latter certain to occur eventually if you use - as a dash—will start a list.) The solution here is to reject column-based hard-wrapping as a terrible idea. Yes, this is a case where the markup language should tell people “you’re doing it wrong”, because otherwise the markup language will either mangle your content, or become bad; or more likely both. Meanwhile in Markdown, it tries to be clever around specific HTML tags and just becomes hard to predict. —⁂— ¹ Prettier’s Markdown formatting is known to mangle content, particularly around underscores and asterisks, and they haven’t done anything about it. The first time I accidentally used it it deleted the rest of a file after some messy bad emphasis stuff from a WYSIWYG HTML → Markdown conversion. That was when I discovered .prettierignore is almost completely broken, too. I came away no longer just unimpressed with some of Prettier’s opinions, but severely unimpressed with the rest of it technically. Why they haven’t disabled it until such things are fixed, I don’t know. ² There’s very little fundamental syntax in it: line break, indent and parsing CSS Counter Styles is about it. The rest is all defined in dialects, for easy extension.

thaumasiotes 1y ago

> reST is more feature-complete and extension-friendly, but it is simply unusable for me because it wasn't designed for agglutinative languages like Korean. How does whether you think of the language as agglutinative affect the usability of reST? The biggest problem that occurs to me is that there isn't really a conceptual difference between an "agglutinative" language in which you have very long words expressing complex meanings, and an "isolating" language in which the same syllables occur in the same order with the same meaning but are thought of on a Platonic level as being all independent words. This is because an "agglutinative" language is one in which syntax markers are more or less independent of any other syntax markers that may apply to the same word†, which means it's always possible by definition to consider those markers to be "words" themselves. Would your problems be solved if you viewed what you had considered "long" Korean words as instead being several short words in a row? What difficulties does agglutination present? † Compare: https://glossary.sil.org/term/agglutinative-language https://glossary.sil.org/term/agglutinative-language > An agglutinative language is a language in which words are made up of a linear sequence of distinct morphemes and each component of meaning is represented by its own morpheme. https://glossary.sil.org/term/isolating-language https://glossary.sil.org/term/isolating-language > An isolating language is a language in which almost every word consists of a single morpheme.

lifthrasiir 1y ago
> This is because an "agglutinative" language is one in which syntax markers are more or less independent of any other syntax markers that may apply to the same word†, which means it's always possible by definition to consider those markers to be "words" themselves. I think SIL's definition is, while robust, not the usual definition because English can be regarded as agglutinative in this definition. This is particularly visible from the statement that most European languages are somewhat fusional [1], which is okay under their definitions but not the usual way we think of English. In my understanding, the analyticity is a spectrum and highly analytic languages with most (but not necessarily all) words containing just one morpheme are said to be isolating. Words in agglutinative languages can be, but not necessarily have to be, analyzed as a main morpheme ("word") with dependent morphemes attached ("affixes"). Polysynthetic languages go further by allowing multiple main morphemes in one word. As languages tend to become synthetic (as opposed to analytic), the space-separated "word" is less useful [2] and segmentation gets harder and harder. reST's failure to support those languages is all about a bad assumption about segmentation. [1] https://glossary.sil.org/term/fusional-language https://glossary.sil.org/term/fusional-language [2] So much that several agglutinative languages---in which space-separated words can still be useful---don't even think about spacing, e.g. Japanese.