3 ms·
I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here. I
by IshKebab 2y ago
I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here.
Is this a "no third party dependencies" thing?
- jart 2y agoI don't want to require everyone who builds llamafile from source need to install rust. I don't even require that people install the gperf command, since I can build gperf as a 700kb actually portable executable and vendor it in the repo. Tree sitter I'd imagine does a really great highly precise job with the languages it supports. However it appears to support fewer of them than I am currently. I'm taking a breadth first approach to syntax highlighting, due to the enormity of languages LLMs understand.
- IshKebab 2y agoI think the Rust component of tree-sitter-highlight is actually pretty small (Tree Sitter generates C for the actual parser). But fair enough - fewer dependencies is always nice, especially in C++ (which doesn't have a modern package manager) and in ML where an enormous janky Python installation is apparently a perfectly normal thing to require.
- mdaniel 2y agoI somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth 1: https://docs.conan.io/2/introduction.html https://docs.conan.io/2/introduction.html
- IshKebab 2y agoNo, if anything vcpkg is "the C++ package manager", but it's nowhere near pervasive and easy-to-use enough to come close to even Pip. It's leagues away from Cargo, Go, and other actually good PL package managers.
- mdaniel 2y agoI knew that Microsoft used that on Windows but had no idea it was multi-platform: https://github.com/microsoft/vcpkg/releases/tag/2024.10.21 https://github.com/microsoft/vcpkg/releases/tag/2024.10.21 (MIT, like a lot of their stuff) Microsoft is such an odd duck, sometimes, but I'm glad to take advantage of their "good years" while it lasts
- deleted 2y ago[deleted]
- chubot 2y agoHave you developed against TreeSitter? Some feedback from people who use it here - https://news.ycombinator.com/item?id=39783471 https://news.ycombinator.com/item?id=39783471 And here - https://lobste.rs/s/9huy81/tbsp_tree_based_source_processing#c_zxxu35 https://lobste.rs/s/9huy81/tbsp_tree_based_source_processing...
- IshKebab 2y agoYes I have, and it worked very well for what I was using it for (assembly language LSP server). I didn't run into any of the issues they mentioned (not saying they don't exist though). For new projects I use Chumsky. It's a pure Rust parser which is nice because it means you avoid the generated C, and it also gives you a fully parsed and natively typed output, rather than Tree Sitter's dynamically typed tree of nodes, which means there's no extra parsing step to do. The main downside is it's more complicated to write the parser (some fairly extreme types). The API isn't stable yet either. But overall I like it more than Tree Sitter.