5 ms·
Let's create a Tree-sitter grammar
- tillcarlos 3y ago(unrelated to content) Wow, that site speed! In the footer there’s a link to OPs source code. The whole site gets compiled with Rust. Interesting gem to find.
- lawn 3y agoStatic sites are a breath of fresh air in this day and age.
- juanrgon 3y agoIt’s a static site. It would be strange if it wasn’t fast
- codethatwerks 3y agoIt is hard to make a slow one. Even shared hosting wordpress with the cache turned on is fast.
- viraptor 3y agoYou'd think so... There's so many versions of WP caching and so many ways it's deployed, that (with a fairly small experience) I've seen sites taking 5s+ to load after enabling the relevant option because suddenly minimisation and bundling kicked in on every request and kept the page fresh. WP is trivial to make accidentally slow.
- prabir 3y agoI did the same with my blog which is written in rust https://github.com/prabirshrestha/rblog https://github.com/prabirshrestha/rblog. While it could be static I wanted it to be server. But I don't think rust is the reason for it. Not depending on external resources such as fonts and javascript for critical path along with compression is the most critical. Since it is a blog focus on text instead of fancy images and only include when absolutely necessary. Testing the site in Fast 3G or Slow 3G is the best way to find the places where you can optimize your site.
- emmanueloga_ 3y agoThe only nitpick is that the site loads 1.2Mb of fonts... apparently through CloudFront. It seems it serves them uncompressed, perhaps because the fonts are woff2? [1] Also, the bold version of iosevka is an order of magnitud heavier than the other fonts :-). -- 1: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#compressed-content-cloudfront-file-types https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- robjwells 3y agoThis is a great post. I’d previously started to write a tree-sitter grammar for djot but bounced off when I realised I’d have to use an external scanner. Nice to have an article that sets out how to do so. Link in the post is currently broken, but here’s the author’s grammar on GitHub: https://github.com/treeman/tree-sitter-djot https://github.com/treeman/tree-sitter-djot
- lawn 3y agoOh shoot, fixed the link. Thanks!
- heresie-dabord 3y agoIn the interest of making the page the best version of itself, the following phrase makes a mistake often made by native US English speakers: > But, you might wonder how does a Tree-sitter grammar look like, How does $thing look? or What does $thing look like? Similarly, one often hears "Where are we at?" It is common enough to be seen as a regionalism. -- Your friendly tree-sitting grammarian
- lawn 3y agoHuh, I'm not a native English speaker, but I changed it to "what does a Tree-sitter grammar look like". It does sound better.
- Joker_vD 3y agoInterestingly enough, in German and Polish the analogous expression is in fact "how does X look like?" So maybe it's an interference from those two languages? Or maybe it's even a relict of the older English expression, displaced in England by "what does X look like" after the Am/Br language split had happened.
- VyseofArcadia 3y agoAnecdotally, I hear the how/what confusion from nearly every non-native English speaker I've ever met, regardless of native language. I wonder if it's one of those little things that's surprisingly rare, like the th sound.
- dugmartin 3y agoIf you want or need to stay totally within the JavaScript ecosystem, Marijn Haverbeke (author of CodeMirror, ProseMirror, and more) created Lezer which is tree-sitter like and is grammar driven with optional external tokenizers also: Announcement: https://marijnhaverbeke.nl/blog/lezer.html https://marijnhaverbeke.nl/blog/lezer.html Homepage: https://lezer.codemirror.net/ https://lezer.codemirror.net/
- mdaniel 3y agohttps://lezer.codemirror.net/docs/guide/#token-specialization https://lezer.codemirror.net/docs/guide/#token-specializatio... is pretty cool, and they make use of it in the python grammar for the new "match" and "case" soft keywords: https://github.com/lezer-parser/python/blob/ead67a397f7c7cb688e36d941a32c1527d7cc409/src/python.grammar#L45 https://github.com/lezer-parser/python/blob/ead67a397f7c7cb6...
- mingodad 3y agoFor people interested on grammars there is also: - Convert tree-sitter grammar.json to EBNF https://meimporta.eu/tree-sitter/json2ebnf.html https://meimporta.eu/tree-sitter/json2ebnf.html - An alternative syntax to write tree-sitter grammars https://mingodad.github.io/lua-wasm-playground/ https://mingodad.github.io/lua-wasm-playground/ (select "Tree-sitter-ebnf-generator" from "Examples" then click "Run" to see the tree-sitter grammar for the content in "Input Text (arg[1])") - An Yacc/Lex online editor/tester with several example grammars https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/
- antonpuz 3y agoThanks for sharing this interesting post! I'm wondering why isn't there any language agnostic AST, commonly referred to as intermediate representation, and higher level logic such as data flow graph built on Tree-sitter. I mean writing the Grammer is of course something that is mandatory but it seems like programming languages are too much different in the architectural level of Tree-sitter
- zelphirkalt 3y agoThere is probably no such thing, because it would be hard to map programming language concepts onto each other perfectly. OK, you could have a union of those concepts in the IR, but the the benefit of IR would disappear, because you will have to deal with all the things on the next layer.
- dcreager 3y agoThis is exactly right. You either end up with something very low-level (on the level of LLVM IR, for instance)—which means you aren't constructing and analyzing high-level language constructs anymore—or with something high-level but with many language-specific special cases grafted on. Where we've found success is in stepping back and creating formalisms that are language-agnostic to begin with, and then using tree-sitter to manage the language-specific translations into that formalism. A good example is how we use stack graphs for precise code navigation [1], which uses a graph structure to encode a language's name binding semantics, and which we build up for several languages from tree-sitter parse trees [2]. [1] https://dcreager.net/talks/stack-graphs/ https://dcreager.net/talks/stack-graphs/ [2] https://github.com/github/stack-graphs/tree/main/languages https://github.com/github/stack-graphs/tree/main/languages
- barrkel 3y agoLanguages are generally only similar at a superficial level and have a lot of fractal detail with high variance. For your data flow graph, are you going to handle copy constructors, assignment operator overloads, implicit conversions etc. like you see in C++? Or how about overload resolution: figuring out which method wins out over all candidates requires details about the convertibility of types, and how the language ranks candidates based on conversions required. And let's not forget Koenig lookup. Things like Tree-sitter can discover definitions, declarations and invocations, but matching the right declaration for an invocation isn't trivial in the presence of overloads.
- saltserv 3y ago[dead]
- lsllc 3y ago@lawn, There's a typo towards the end (about 80%) down in the section on nvim-treesitter, should be `TSUpdate`, not `TSUdate`: :TSUdate sdjot Excellent write up though!
- lawn 3y agoThank you, fixed.
- gcr 3y agoIs there any easy way to make tree-sitter produce smaller files? I was hoping to add support for nim to difftastic [1], but tree-sitter produces a 60MB .c file that’s too big to add to the repository. [1]: https://github.com/Wilfred/difftastic/pull/633 https://github.com/Wilfred/difftastic/pull/633
- diffxx 3y agoAbandon all hope, ye who enter here. Tree-sitter is great until you inevitably run into a problem with it and find that it is impossible to fix properly if you can even figure out why the problem even exists. The idea of a uniform api for querying syntax trees is a good one and tree-sitter deserves credit for popularizing it. It's unfortunately not a great implementation of the idea.
- dumbo-octopus 3y agoYes. It also presents itself as some sort of bastion of Rust stability, but segfaults constantly^. More than any NPM module I've ever used before. Any syntax that doesn't precisely match the grammar is liable to take down your entire thread. The wasm bindings are a necessity if you want to be anywhere near "stable", but they run 3x slower. ^ the JS/TS parser when invoked via the Node bindings, at least. Others are potentially better. The GH issues are rife with similar complaints.
- nbadg 3y agoMaybe I'm missing something here, but isn't tree sitter already used by a bunch of stuff? Last I heard, github was moving all of their syntax highlighting to it, and a bunch of editors support it. Are they just dealing with the segfaults? Genuine question, because I'm considering using tree sitter for two different projects I'm working on (a LML and a full-blown language). Or is it an issue with poorly-written grammars? The second you need to write an external parser with tree sitter, you have to start writing C, so I can see how that could lead to segfaults very quickly. I dunno, I guess I'm just a bit surprised to hear a bunch of negative feedback towards tree-sitter, because up until now most of what I've heard has been pretty positive.
- andrewla 3y agoI'm not abreast of all the latest buzz in NeoVim, but I've been hearing and seeing good things about Treesitter in general. Is the plan to at some point deprecate the existing vim-style syntax highlighting (syn match/region etc.) in favor of treesitter-based syntax highlighting? Where can I find this discussion if it is happening?