4 ms·
Better code highlighting on the web: rehype-tree-sitter
- nullpt_rs 3y agoDefinitely considering switching over my blog to Astro since I write my posts in MDX and using this plugin to avoid doing this work on the client at all (currently use highlight.js and Next). Lovely work by my friend Haze :)
- l5870uoo9y 3y agoNextJS does however offer an experimental Rust implementation for parsing MDX which makes it a lot faster and less buggy to use than setting it yourself with various Rehype and Remark plugins. Using Rust parser with Bright[1] (if you don't need other plugins like table of contents or Github Flavoured Markdown) is a potent setup. [1]: https://bright.codehike.org/ https://bright.codehike.org/
- baby 3y agoNever heard of Astro[1] until today. I was looking for something better than Hugo and ended up using Nuxt content. How is this better than nuxt content :D ? [1]: https://astro.build/ https://astro.build/
- hazebooth 3y agoI haven't used 'Next Content', but one big difference is that Astro is framework agnostic. I'm not a huge web dev so I wouldn't know how different these frameworks are, but not being locked in/being able to try another framework is cool
- baby 3y agoIt looks like it's another framework to me. The basic example looks exactly like Nuxt to me as well! (component, pages, and layout folder!) To answer my previous question, they have a page talking about the similarities and differences here: https://docs.astro.build/en/guides/migrate-to-astro/from-nuxtjs/#key-differences-between-nuxt-and-astro https://docs.astro.build/en/guides/migrate-to-astro/from-nux... They look really really similar to me (Nuxt content and Astro)
- eyelidlessness 3y agoAstro is akin to other “meta-frameworks” with SSR/SSG like Next/Nuxt/etc, but it’s client-framework agnostic so its meta-ness is too. What primarily sets it apart is that it’s designed to ship just HTML/CSS by default, with any client side JS being opt-in where a client component is invoked (termed “islands”), and those components being authored in any client component library of your choosing (if an adapter for it exists, and many do). Or even vanilla JS, but then Astro is more akin to Jekyll or other similar SSGs.
- contravariant 3y agoTrue to their word the page loads fine without any javascript. The HTML is even somewhat readable if you ignore the embedded styling. Though this blog is a lot better in that regard.
- OJFord 3y agoHaven't heard of it before either, but their own homepage (which I assume is dogfooding) has unnatural feeling (vertical) scroll-jacking, and small amount of horizontal scroll due to incorrect width. (Mobile, Firefox, Android)
- l5870uoo9y 3y agoAnother contestant in this realm is Bright[1]. It runs entirely on the server and doesn't increase bundle size as seen here[2]. Regarding parsing speed tree-sitter is without a doubt performant since it is written in Rust, but I don't have any problems "parsing on every keystroke" with a setup containing Marked[3], highlight.js[4] and a sanitizer. I did however experience performance issues with other Markdown parser libraries than Marked. [1]: https://bright.codehike.org/ https://bright.codehike.org/ [2]: https://aihelperbot.com/test-suite https://aihelperbot.com/test-suite [3]: https://github.com/markedjs/marked https://github.com/markedjs/marked [4]: https://highlightjs.org/ https://highlightjs.org/
- eyelidlessness 3y agoNit: tree-sitter has first party Rust bindings (and its WASM bindings use Rust wasm-bindgen), but the core functionality is written in C.
- hyperhello 3y agoIs syntax highlighting considered difficult? A regex loop that outputs HTML to a pre is a few lines of vanilla JS.
- mrkeen 3y agoPlease share.
- hyperhello 3y agoI’m on my phone, so no code sample, but put a choice regex in a variable and loop with .exec(), emitting that text surrounded by span tags into an array that you then join(). It’s really almost a one or two line script.
- acedTrex 3y agoand for languages that have contextual keywords or semantic highlighting needs?
- hyperhello 3y agoYou can either use CSS where it makes sense or you can keep the regex results to refer to as needed instead of throwing them out.
- _gabe_ 3y agoConsider: `5 + 3 = ${5 + 3}`; const foo = async () => {} const foo = async function () {} const foo = { 'PI': 3.14 } // etc. And then also remember that HTML can embed that, and CSS. And then highlighting something like TSX which can highlight HTML etc. It quickly becomes a recursive nightmare. And all this is coming from someone who thought similarly to you. And now about 3 weeks of full time work (plus using a pre-built library oniguruma to do a lot of heavy lifting) and I just barely have a performant syntax highlighter. There are still a lot of edge cases I'm working through though.
- 3y ago
- meitros 3y agoI'm looking into client-side syntax highlighting as well at the moment and lezer popped up: https://lezer.codemirror.net/ https://lezer.codemirror.net/. It's directly based on tree-sitter but tailored to be more web friendly (and written by the same author as codemirror) - https://marijnhaverbeke.nl/blog/lezer.html https://marijnhaverbeke.nl/blog/lezer.html
- dunham 3y agoI think I'd prefer Lezer in a javascript-only setting. The results are compact and you don't have to deal with WASM. I've written a small tree-sitter grammar, with off-side rule even, and had a bunch of issues with the javascript bits not being the same version as the WASM bits. I ended up having to get just the right version of emscripten installed (and maybe a few other tools). The process felt a bit fragile to me.
- eyelidlessness 3y agoWhile I have no doubt tree-sitter is technically superior for the specific task of syntax highlighting, Shiki has some other appealing qualities: - Built in support for TextMate themes (and therefore also themes for VSCode, probably Sublime as well) - Shiki Twoslash[1], which supports a variety of really cool features for highlighting subsections of code, static types and type errors, module relationships 1: https://github.com/shikijs/twoslash https://github.com/shikijs/twoslash
- hardwaregeek 3y agoI was looking into implementing syntax highlighting for my hobby language and man, it’s surprisingly hard. Either you use tree-sitter which has limited support, or TextMate grammars which seem to be quite terrible. And to get JetBrains support it seems like you need to add a whole other lexer integration. Can’t we all standardize around tree-sitter?
- zamalek 3y agoAs someone who is firmly ( _firmly_ ) in the "nobody can write good manual memory management code" train, zig is probably a better fit than rust here - and this is one of the extremely rare circumstances where I would pick zig over rust. Tree-sitter is written in C - the memory benefits of rust are completely moot. Zig also has a built-in C++ compiler, so you could generate the object file with a single toolset.
- hazebooth 3y agoI chose Rust here because that was what the command line tool that I referenced were written in. I'm not going to argue on HN about memory management (because I too, prefer Zig over Rust) because it's all about the program you're trying to write. In terms of generating the object file with a single toolset, It's just not something I care about. This was like, a two day project. I don't need to make it more complicated than it needs to be
- microflash 3y agoShameless plug for the Rehype plugin [1] I wrote based on Starry Night [2]. If you're interested in syntax highlighting like GitHub or customize it with CSS properties (for example, to support multiple themes), this is one solution. [1] https://github.com/Microflash/rehype-starry-night https://github.com/Microflash/rehype-starry-night [2] https://github.com/wooorm/starry-night https://github.com/wooorm/starry-night
- siraben 3y agoI badly want this but in LaTeX. Minted is alright but it suffers from the same pitfalls as other regex-based syntax highlighters, and while you can extend the Pygments' lexer with more keywords, it's a losing battle. Having good syntax highlighting in academic papers would be amazing, since many subfields have their own DSL-like notations. Unfortunately, writing new LaTeX packages turns out to be quite difficult, especially if you want it to integrate cleanly with \usepackage and environments.