4 ms·

Shiki Syntax Highlighter

123 points by stefankuehnel 4y ago

armchairhacker 4y ago
Question: why use TextMate instead of tree-sitter? Tree-sitter's drawbacks compared to TextMate and other regex-based solutions are slower speeds and chance of edits breaking parse trees, which are not issues here. Its advantages are that the grammar definition can be more expressive and is much more readable. Not saying you should abandon using TextMate, "if it ain't broke..." and I doubt all of the grammars you support right now have as pretty tree-sitter equivalents. But maybe add support for tree-sitter as well, especially because it seems to be growing in popularity, with growing support in emacs and neovim.
- eyelidlessness 4y ago
  The advantage is reusing existing themes. TextMate themes as a standard are the de facto not-great-but-used-everywhere thing and in this case it means you can style your blog or whatever with what you use in VSCode or Sublime or whatever.
  - doodpants 4y ago
    Ironically, the last time I checked (which was some number of years ago), TextMate 2.0 did not support the TextMate 1.x theme format, not even via any sort of import or conversion mechanism. Which is odd, considering that it's become a de facto standard supported by other editors. TextMate 1.x was the king of Mac text editors back in its day. Is anyone actually still using TextMate today?
    - eyelidlessness 4y ago
      I’m amazed I can’t remember whether I converted my custom TextMate theme when I started using 2.0! I also wish I’d realized the vivid colors in the theme (I called it “acid skittles”) were appealing to me for contrast, not their range of color spectrum. I don’t use TextMate anymore but I surely would have stuck with it longer if it had a debugger and language server. I panned all of the non-native editors that came between, but sadly had to admit that VSCode fit my workflow a lot better.
- Keats 4y ago
  Tree-sitter is going to be a lot slower to load: https://github.com/tree-sitter/tree-sitter/issues/1942 https://github.com/tree-sitter/tree-sitter/issues/1942. It's ok for things like editor but using it in a SSG for example is hard when just loading the syntaxes will take multiple times the time it takes to render the whole thing with a regex based highlighter.
- artagnon 4y ago
  Is tree-sitter really slower than TextMate grammars? Some benchmarks indicate that this isn't really the case [1]. On the other hand, breaking parse trees is a real issue, because the error-recovery in tree-sitter is pretty rudimentary [2][3], but as you said, it's not an issue for Shiki. Several TextMate grammars suffer from inaccuracy bugs, and issues of maintainability. Perhaps the biggest hindrance in the adoption of tree-sitter, is that the most popular editor, VSCode, still doesn't support it. [1]: https://github.com/microsoft/vscode/pull/161479 https://github.com/microsoft/vscode/pull/161479 [2]: https://github.com/tree-sitter/tree-sitter/issues/1870 https://github.com/tree-sitter/tree-sitter/issues/1870 [3]: https://github.com/tree-sitter/tree-sitter/issues/224 https://github.com/tree-sitter/tree-sitter/issues/224
- z3t4 4y ago
  Would use treesitter if it had more documentation and examples. For example how to find all function declarations, all variable declarations, etc
spankalee 4y ago
The problem with Shiki is that it uses TextMate grammars which in turn require the Oniguruma regex engine. Shiki compiles that to WASM I believe, but it makes Shiki larger and slower than a system that uses plain JavaScript regexes or a real parser. I'd only use it if it's absolutely necessary that your syntax highlighting exactly matches VS Code. (It's also a shame that VS Code requires Oniguruma)
- benatkin 4y ago
  Not all WASM files are in the same ballpark. This one is 500K while esbuild is almost 10MB. https://cdn.jsdelivr.net/npm/shiki@0.14.1/dist/ https://cdn.jsdelivr.net/npm/shiki@0.14.1/dist/ https://cdn.jsdelivr.net/npm/esbuild-wasm/ https://cdn.jsdelivr.net/npm/esbuild-wasm/ That's no knock on esbuild. esbuild benefits from having been built with golang and wasn't designed to be run in the browser. (tinygo can run go in much smaller but you give up some of golang's power) WASM is fast becoming a standard thing.
  - spankalee 4y ago
    500k to mostly duplicate an existing platform feature? No thanks.
    - tiagod 4y ago
      What feature are you referring to?
      - benatkin 4y ago
        Regular expressions it seems. I disagree that it's a duplicate or a waste. It's a different regex engine with different features.
- supz_k 4y ago
  Yes, I have also noticed that Shiki is slower. We use it in our blogging platform for code block highlighting and posts with code blocks are noticeably slower to save than posts without them. Are there any alternatives that supports VS Code syntax but faster?
TacticalCoder 4y ago
There's a similar one for Emacs I know of, htmlize.el. It reuses the parsing already done by Emacs (not just from the major mode but by any minor mode too) and output HTML exactly matching the colors syntax highlighting you see in your Emacs buffer.
- nine_k 4y ago
  I think it's normally used if you export an Org file with fragments of code. It can be invoked directly, too.
- anschwa 4y ago
  Yes! Using org-publish with htmlize is wonderful.
mnurzia 4y ago
This is really cool. I've been looking for a high quality (Textmate-based) syntax highlighter for rendering source code on a poster. It would be great to integrate this library with ctags or something in order to perform accurate semantic highlighting for C/C++ (distinguishing between macros and functions, etc.).
satvikpendem 4y ago
Looks nice. Are there any TreeSitter grammar based code highlighters too?
- fiddlerwoaroof 4y ago
  htmlize.el if you use tree-sitter based syntax highlighting in emacs. I’ve written one myself, it’s pretty simple to walk a tree-sitter parse tree and generate spans with classes based on the type of each node. Then syntax highlighting is “just” a matter of writing a bit of CSS.
contextfree 4y ago
Is the name 式? I always found this word interesting, it can mean "mathematical term/expression", but also "form" (as in formalism) in general, and can also mean "ceremony" as in ritual form
microflash 4y ago
Related and similar, there's also starry-night: https://github.com/wooorm/starry-night https://github.com/wooorm/starry-night
barbazoo 4y ago
What, that's so cool. I was surprised to find the code sample be runnable. Nice work.
eyelidlessness 4y ago
Shiki is awesome for code samples. It’s even better paired with a set of light/dark themes designed to be used together (and I really need to get around to open sourcing my solution for swapping inline styles with classes for that use case, it’s great for using Shiki without a client side runtime). ALSO awesome is Shiki Twoslash[1], for displaying TypeScript editor feedback in code examples. 1: https://github.com/shikijs/twoslash https://github.com/shikijs/twoslash
wodenokoto 4y ago
Being out of the loop, what does other highlighters used in static website use as engines?
c54 4y ago
Vim has :TOhtml built in https://vim.fandom.com/wiki/Pasting_code_with_syntax_coloring_in_emails https://vim.fandom.com/wiki/Pasting_code_with_syntax_colorin...
ifree 4y ago
It's cool, the svg generation feature is quite useful for embedding code snippets into a diagram.

123 points by stefankuehnel 4y ago

Question: why use TextMate instead of tree-sitter? Tree-sitter's drawbacks compared to TextMate and other regex-based solutions are slower speeds and chance of edits breaking parse trees, which are not issues here. Its advantages are that the grammar definition can be more expressive and is much more readable. Not saying you should abandon using TextMate, "if it ain't broke..." and I doubt all of the grammars you support right now have as pretty tree-sitter equivalents. But maybe add support for tree-sitter as well, especially because it seems to be growing in popularity, with growing support in emacs and neovim.

eyelidlessness 4y ago
The advantage is reusing existing themes. TextMate themes as a standard are the de facto not-great-but-used-everywhere thing and in this case it means you can style your blog or whatever with what you use in VSCode or Sublime or whatever.
- doodpants 4y ago
  Ironically, the last time I checked (which was some number of years ago), TextMate 2.0 did not support the TextMate 1.x theme format, not even via any sort of import or conversion mechanism. Which is odd, considering that it's become a de facto standard supported by other editors. TextMate 1.x was the king of Mac text editors back in its day. Is anyone actually still using TextMate today?
  - eyelidlessness 4y ago
    I’m amazed I can’t remember whether I converted my custom TextMate theme when I started using 2.0! I also wish I’d realized the vivid colors in the theme (I called it “acid skittles”) were appealing to me for contrast, not their range of color spectrum. I don’t use TextMate anymore but I surely would have stuck with it longer if it had a debugger and language server. I panned all of the non-native editors that came between, but sadly had to admit that VSCode fit my workflow a lot better.
Keats 4y ago
Tree-sitter is going to be a lot slower to load: https://github.com/tree-sitter/tree-sitter/issues/1942 https://github.com/tree-sitter/tree-sitter/issues/1942. It's ok for things like editor but using it in a SSG for example is hard when just loading the syntaxes will take multiple times the time it takes to render the whole thing with a regex based highlighter.
artagnon 4y ago
Is tree-sitter really slower than TextMate grammars? Some benchmarks indicate that this isn't really the case [1]. On the other hand, breaking parse trees is a real issue, because the error-recovery in tree-sitter is pretty rudimentary [2][3], but as you said, it's not an issue for Shiki. Several TextMate grammars suffer from inaccuracy bugs, and issues of maintainability. Perhaps the biggest hindrance in the adoption of tree-sitter, is that the most popular editor, VSCode, still doesn't support it. [1]: https://github.com/microsoft/vscode/pull/161479 https://github.com/microsoft/vscode/pull/161479 [2]: https://github.com/tree-sitter/tree-sitter/issues/1870 https://github.com/tree-sitter/tree-sitter/issues/1870 [3]: https://github.com/tree-sitter/tree-sitter/issues/224 https://github.com/tree-sitter/tree-sitter/issues/224
z3t4 4y ago
Would use treesitter if it had more documentation and examples. For example how to find all function declarations, all variable declarations, etc

spankalee 4y ago

The problem with Shiki is that it uses TextMate grammars which in turn require the Oniguruma regex engine. Shiki compiles that to WASM I believe, but it makes Shiki larger and slower than a system that uses plain JavaScript regexes or a real parser. I'd only use it if it's absolutely necessary that your syntax highlighting exactly matches VS Code. (It's also a shame that VS Code requires Oniguruma)

benatkin 4y ago
Not all WASM files are in the same ballpark. This one is 500K while esbuild is almost 10MB. https://cdn.jsdelivr.net/npm/shiki@0.14.1/dist/ https://cdn.jsdelivr.net/npm/shiki@0.14.1/dist/ https://cdn.jsdelivr.net/npm/esbuild-wasm/ https://cdn.jsdelivr.net/npm/esbuild-wasm/ That's no knock on esbuild. esbuild benefits from having been built with golang and wasn't designed to be run in the browser. (tinygo can run go in much smaller but you give up some of golang's power) WASM is fast becoming a standard thing.
- spankalee 4y ago
  500k to mostly duplicate an existing platform feature? No thanks.
  - tiagod 4y ago
    What feature are you referring to?
    - benatkin 4y ago
      Regular expressions it seems. I disagree that it's a duplicate or a waste. It's a different regex engine with different features.
supz_k 4y ago
Yes, I have also noticed that Shiki is slower. We use it in our blogging platform for code block highlighting and posts with code blocks are noticeably slower to save than posts without them. Are there any alternatives that supports VS Code syntax but faster?

TacticalCoder 4y ago

There's a similar one for Emacs I know of, htmlize.el. It reuses the parsing already done by Emacs (not just from the major mode but by any minor mode too) and output HTML exactly matching the colors syntax highlighting you see in your Emacs buffer.

nine_k 4y ago
I think it's normally used if you export an Org file with fragments of code. It can be invoked directly, too.
anschwa 4y ago
Yes! Using org-publish with htmlize is wonderful.

mnurzia 4y ago

This is really cool. I've been looking for a high quality (Textmate-based) syntax highlighter for rendering source code on a poster. It would be great to integrate this library with ctags or something in order to perform accurate semantic highlighting for C/C++ (distinguishing between macros and functions, etc.).

satvikpendem 4y ago

Looks nice. Are there any TreeSitter grammar based code highlighters too?

fiddlerwoaroof 4y ago
htmlize.el if you use tree-sitter based syntax highlighting in emacs. I’ve written one myself, it’s pretty simple to walk a tree-sitter parse tree and generate spans with classes based on the type of each node. Then syntax highlighting is “just” a matter of writing a bit of CSS.

contextfree 4y ago

Is the name 式? I always found this word interesting, it can mean "mathematical term/expression", but also "form" (as in formalism) in general, and can also mean "ceremony" as in ritual form

microflash 4y ago

Related and similar, there's also starry-night: https://github.com/wooorm/starry-night https://github.com/wooorm/starry-night

barbazoo 4y ago

What, that's so cool. I was surprised to find the code sample be runnable. Nice work.

eyelidlessness 4y ago

Shiki is awesome for code samples. It’s even better paired with a set of light/dark themes designed to be used together (and I really need to get around to open sourcing my solution for swapping inline styles with classes for that use case, it’s great for using Shiki without a client side runtime). ALSO awesome is Shiki Twoslash[1], for displaying TypeScript editor feedback in code examples. 1: https://github.com/shikijs/twoslash https://github.com/shikijs/twoslash

wodenokoto 4y ago

Being out of the loop, what does other highlighters used in static website use as engines?

c54 4y ago

Vim has :TOhtml built in https://vim.fandom.com/wiki/Pasting_code_with_syntax_coloring_in_emails https://vim.fandom.com/wiki/Pasting_code_with_syntax_colorin...

ifree 4y ago

It's cool, the svg generation feature is quite useful for embedding code snippets into a diagram.