5 ms·

It's strange to me that we call these things "syntax highlighters", given that they don't really seem to colour things differently according to syntax,† but rat

It's strange to me that we call these things "syntax highlighters", given that they don't really seem to colour things differently according to syntax,† but rather by the category of token. Perhaps "lexical highlighter" would be a better term for these? Because of this, unfortunately "English syntax highlighter" has two possible interpretations to me: [English [syntax highlighter]], where syntax highlighter is a noun having the idiosyncratic meaning it has in a programming context, i.e. a tool that highlights different tokens according to their category, and [[English syntax] highlighter], a tool that would presumably highlight parts of a sentence according to their syntactic function. The latter would be more exciting, but also more difficult. † There are some exceptions to this, but it does seem to be true for most of the syntax highlighters I've used.

raldu 11y ago
Your observation seems plausible at first. However when I gave it a second thought I have concluded otherwise. In the natural language context, a part of speech is actually determined syntactically. "Type" of a token in formal languages by comparison is also syntactical. Any highlighter must recognize the syntactic structure of its target language and work like a parser, that is, not like a tokenizer. You have to "push" something to the stack, if you know what I mean. To give a bad, but intuitive example, if a highlighter assigned "gray" to every "=" character it comes across, it would not be able to highlight the same characters inside a quotation mark.
- TazeTSchnitzel 11y ago
  > Any highlighter must recognize the syntactic structure of its target language and work like a parser, that is, not like a tokenizer. You have to "push" something to the stack, if you know what I mean. How often does syntax matter in "syntax highlighting"? Most highlighters I've used don't even try to recognise syntactic structure. > To give a bad, but intuitive example, if a highlighter assigned "gray" to every "=" character it comes across, it would not be able to highlight the same characters inside a quotation mark. The problem with this example is that strings are a single token.
GolDDranks 11y ago
I totally expected it to be "just" a part-of-speech-tagger. It does at least these correctly: It takes many tries to succeed. It takes only one try. It takes a try. I want to try and succeed. I'm gonna try it anyway. Man the boats! Man and the boats! What you'd want it to do? To show the parse tree? If I understand correctly, even the non-computational linguistics have had (and still have) trouble to come up with a system that parses all the natural language correctly – or even a system that could represent all the parses of natural sentences correctly.
- GolDDranks 11y ago
  Btw. to elaborate: there seems to be some troublesome sentences where a parse tree isn't just enough – you've got to have a graph. Yet, most of the linguistic phenomena can be parsed into tree structures. That's troublesome. It's like... there's a certain class of a parser in the brain, but then some curious sentences show the ability of overcome the "limitations" of that parser, and still they are considered grammatical.
kazinator 11y ago
"Lexical coloring" or "Lexical color-coding" would be even more appropriate. However, you are slightly wrong, depending on the coloring engine and language. The syntax coloring algorithm in Vim handles nested syntax. (To what extent is up to the developer of the syntax file.) You can see some of that at work here: http://www.kylheku.com/cgit/txr/tree/tests/011/txr-case.txr http://www.kylheku.com/cgit/txr/tree/tests/011/txr-case.txr (This is being colorized by invoking Vim out of Apache on-the-fly.) Here we can see that a Lisp dialect is embedded in a special-purpose language. The functions and operators of the Lisp dialect are colored green, but those in the special purpose language are purple. This is worked out even in situations when the are the same symbol. E.g. both languages have a "do" operator. Vim doesn't handle nesting with a formal context-free grammar. Rather, it allows named regions to be defined, and highlighting matches can be declared as occurring only in specific regions. Regions can also be in regions. You can get very accurate coloring; but it can also be slow at times.

raldu 11y ago

Your observation seems plausible at first. However when I gave it a second thought I have concluded otherwise. In the natural language context, a part of speech is actually determined syntactically. "Type" of a token in formal languages by comparison is also syntactical. Any highlighter must recognize the syntactic structure of its target language and work like a parser, that is, not like a tokenizer. You have to "push" something to the stack, if you know what I mean. To give a bad, but intuitive example, if a highlighter assigned "gray" to every "=" character it comes across, it would not be able to highlight the same characters inside a quotation mark.

TazeTSchnitzel 11y ago
> Any highlighter must recognize the syntactic structure of its target language and work like a parser, that is, not like a tokenizer. You have to "push" something to the stack, if you know what I mean. How often does syntax matter in "syntax highlighting"? Most highlighters I've used don't even try to recognise syntactic structure. > To give a bad, but intuitive example, if a highlighter assigned "gray" to every "=" character it comes across, it would not be able to highlight the same characters inside a quotation mark. The problem with this example is that strings are a single token.

GolDDranks 11y ago

I totally expected it to be "just" a part-of-speech-tagger. It does at least these correctly: It takes many tries to succeed. It takes only one try. It takes a try. I want to try and succeed. I'm gonna try it anyway. Man the boats! Man and the boats! What you'd want it to do? To show the parse tree? If I understand correctly, even the non-computational linguistics have had (and still have) trouble to come up with a system that parses all the natural language correctly – or even a system that could represent all the parses of natural sentences correctly.

GolDDranks 11y ago
Btw. to elaborate: there seems to be some troublesome sentences where a parse tree isn't just enough – you've got to have a graph. Yet, most of the linguistic phenomena can be parsed into tree structures. That's troublesome. It's like... there's a certain class of a parser in the brain, but then some curious sentences show the ability of overcome the "limitations" of that parser, and still they are considered grammatical.

kazinator 11y ago

"Lexical coloring" or "Lexical color-coding" would be even more appropriate. However, you are slightly wrong, depending on the coloring engine and language. The syntax coloring algorithm in Vim handles nested syntax. (To what extent is up to the developer of the syntax file.) You can see some of that at work here: http://www.kylheku.com/cgit/txr/tree/tests/011/txr-case.txr http://www.kylheku.com/cgit/txr/tree/tests/011/txr-case.txr (This is being colorized by invoking Vim out of Apache on-the-fly.) Here we can see that a Lisp dialect is embedded in a special-purpose language. The functions and operators of the Lisp dialect are colored green, but those in the special purpose language are purple. This is worked out even in situations when the are the same symbol. E.g. both languages have a "do" operator. Vim doesn't handle nesting with a formal context-free grammar. Rather, it allows named regions to be defined, and highlighting matches can be declared as occurring only in specific regions. Regions can also be in regions. You can get very accurate coloring; but it can also be slow at times.