5 ms·

Is syntax highlighting considered difficult? A regex loop that outputs HTML to a pre is a few lines of vanilla JS.

by hyperhello 3y ago

mrkeen 3y ago
Please share.
- hyperhello 3y ago
  I’m on my phone, so no code sample, but put a choice regex in a variable and loop with .exec(), emitting that text surrounded by span tags into an array that you then join(). It’s really almost a one or two line script.
  - acedTrex 3y ago
    and for languages that have contextual keywords or semantic highlighting needs?
    - hyperhello 3y ago
      You can either use CSS where it makes sense or you can keep the regex results to refer to as needed instead of throwing them out.
  - _gabe_ 3y ago
    Consider: `5 + 3 = ${5 + 3}`; const foo = async () => {} const foo = async function () {} const foo = { 'PI': 3.14 } // etc. And then also remember that HTML can embed that, and CSS. And then highlighting something like TSX which can highlight HTML etc. It quickly becomes a recursive nightmare. And all this is coming from someone who thought similarly to you. And now about 3 weeks of full time work (plus using a pre-built library oniguruma to do a lot of heavy lifting) and I just barely have a performant syntax highlighter. There are still a lot of edge cases I'm working through though.
    - hyperhello 3y ago
      It's not a nightmare, just a very small bit of mechanical work. I realize HN hates it when people don't talk about JavaScript like it's sex and quantum mechanics so I'll just back off and stop.
      - _gabe_ 3y ago
        I'm only talking about JavaScript because it was the most difficult language to correctly parse so far. The examples I chose highlight that it's not as simple as a couple regexes. Notice how curly brackets can indicate: additional JS code, objects, or interpolated strings. The objects require different syntax highlighting than the JS code. This means you need some complex regexes to provide additional insight into what your highlighting. And as someone actively working on this, yes. It is a nightmare. No, it's not trivial.
hoten 3y ago
This is essentially the bulk of grammar tools like TextMate (though it also has the very useful begin/end token construct). The hard part is likely doing it efficiently, and standardizing on a set of token names such that a theme ecosystem can exist. TextMate grammar is the only syntax highlighting tool I'm familiar with. I'm curious why these others might be preferred. I only know TextMate since it is what VS Code supports.

Is syntax highlighting considered difficult? A regex loop that outputs HTML to a pre is a few lines of vanilla JS.

mrkeen 3y ago

Please share.

hyperhello 3y ago
I’m on my phone, so no code sample, but put a choice regex in a variable and loop with .exec(), emitting that text surrounded by span tags into an array that you then join(). It’s really almost a one or two line script.
- acedTrex 3y ago
  and for languages that have contextual keywords or semantic highlighting needs?
  - hyperhello 3y ago
    You can either use CSS where it makes sense or you can keep the regex results to refer to as needed instead of throwing them out.
- _gabe_ 3y ago
  Consider: `5 + 3 = ${5 + 3}`; const foo = async () => {} const foo = async function () {} const foo = { 'PI': 3.14 } // etc. And then also remember that HTML can embed that, and CSS. And then highlighting something like TSX which can highlight HTML etc. It quickly becomes a recursive nightmare. And all this is coming from someone who thought similarly to you. And now about 3 weeks of full time work (plus using a pre-built library oniguruma to do a lot of heavy lifting) and I just barely have a performant syntax highlighter. There are still a lot of edge cases I'm working through though.
  - hyperhello 3y ago
    It's not a nightmare, just a very small bit of mechanical work. I realize HN hates it when people don't talk about JavaScript like it's sex and quantum mechanics so I'll just back off and stop.
    - _gabe_ 3y ago
      I'm only talking about JavaScript because it was the most difficult language to correctly parse so far. The examples I chose highlight that it's not as simple as a couple regexes. Notice how curly brackets can indicate: additional JS code, objects, or interpolated strings. The objects require different syntax highlighting than the JS code. This means you need some complex regexes to provide additional insight into what your highlighting. And as someone actively working on this, yes. It is a nightmare. No, it's not trivial.

hoten 3y ago

This is essentially the bulk of grammar tools like TextMate (though it also has the very useful begin/end token construct). The hard part is likely doing it efficiently, and standardizing on a set of token names such that a theme ecosystem can exist. TextMate grammar is the only syntax highlighting tool I'm familiar with. I'm curious why these others might be preferred. I only know TextMate since it is what VS Code supports.