8 ms·
Show HN: I ported Tree-sitter to Go
This started as a hard requirement for my TUI-based editor application, it ended up going in a few different directions.
A suite of tools that help with semantic code entities: https://github.com/odvcencio/gts-suite https://github.com/odvcencio/gts-suite
A next-gen version control system called Got: https://github.com/odvcencio/got https://github.com/odvcencio/got
I think this has some pretty big potential! I think there's many classes of application (particularly legacy architecture) that can benefit from these kinds of analysis tooling. My next post will be about composing all these together, an exciting project I call GotHub. Thanks!
- sluongng 9mo agoOh this is really neat for the Bazel community, as depending on tree-sitter to build a gazelle language extension, with Gazelle written in Go, requires you to use CGO. Now perhaps we can get rid of the CGO dependency and make it pure Go instead. I have pinged some folks to take a look at it.
- 3rly 9mo agoWouldn't `got` be confused with OpenBSD's Got: https://gameoftrees.org/index.html https://gameoftrees.org/index.html
- odvcencio 9mo agooh wow! i really thought i was being too clever but i shouldve assumed nothing new under the sun. well im taking name suggestions now!
- boobsbr 9mo agoGoty McGotface
- sbankowi 9mo agoYAGOT (Yet Another GOT)
- bityard 9mo agoProbably taken already, better use YAGOT-NG (Next Generation) just to be safe.
- himata4113 9mo agomight be taken too so just YAGOT2 would work
- deleted 9mo ago[deleted]
- Imustaskforhelp 9mo agouGOT / uGOTme? (sort of like the idea behind uTorrent) but I will agree that sbankowi's idea of Yet another got is great as well. +1 to that as well.
- allknowingfrog 9mo agoWell, find and sed have modern "fd" and "sd" alternatives. Naming it "gt" allows you to claim that your version save 33% compared to typing "git".
- rapnie 9mo agogotgit, gotcha
- gritzko 9mo agoThat is very very interesting. I work on a similar project https://replicated.wiki/blog/partII.html https://replicated.wiki/blog/partII.html I use CRDT merge though, cause 3-way metadata-less merges only provide very incremental improvements over e.g. git+mergiraf. How do you see got's main improvement over git?
- odvcencio 9mo agoprimarily, got is structural VCS intended for concurrent edits of the same file. it does this via gotreesitter and gts-suite abstractions that enable it to: - have entity-aware diffs - not line by line but function by function - structural blame - attribution resolution for the lifetime of the entity - semver from structure - it can recommend bumps because it knows what is breaking change vs minor vs patch - entity history - because entities are tracked independently, file renames or moves dont affect the entity's history when gotreesitter cant parse a language, the 3way text merge happens as a fallback. what the structural merge enables is no conflicts unless same entity has conflicting changes
- gritzko 9mo agoI think I understand the situation.
- odvcencio 9mo agogah,. sincere apologies for formatting of this post. i ahve been on HN for basically 10 years now without ever having made a post (:
- dorianmariecom 9mo agouse four spaces " " in front of a line for <pre> formatting like " this"
- srcreigh 9mo agoIt's 2 or more spaces, not four
- skybrian 9mo agoHow about making 'got' compatible with git repos like jujutsu? It would be a lot easier to try out.
- odvcencio 9mo agoit is interoperable with git. we like git when its good but attempted to ease the pains in UX somewhat. you can take advantage of got locally but still push it to git remote forges jsut the same. when you pull stuff in this way, got will load the entity history into the git repo ensuring that you can still do got stuff locally (inspect entity histories, etc)
- irishcoffee 9mo agoIs it a go-ism that source for implementation and test code lives in the root of the repo or is this an LLM thing?
- odvcencio 9mo agoyeah the tests live with the implementation code always (Go thing) and the repo root thing is like a preference, main is an acceptable package to put stuff in (Go thing), i see this a lot with smaller projects or library type projects
- zero_shift 9mo agoInteresting. I have a similar usecase but intended to use CGo tree-sitter with Zig Are these pretty up-to-date grammars? I'm awfully tempted to switch to your project How large are your binaries getting? I was concerned about the size of some of the grammars
- odvcencio 9mo ago206 binary blobs = 15MB, so not crazy but i built for this use case where you can declare the registry of languages you want to load and not have to own all the grammar binaries by default
- zero_shift 9mo agoIf all the languages together add up to 15MB that is a game changer for me. It means the CLI I am working on can ship support for many languages whilst still being a smallish (sub 50mb) download I shall definitely check it out!
- odvcencio 9mo agore: up to date grammars, yes i found the official grammars in use by the original tree-sitter library today
- conartist6 9mo agoIt looks like porting the custom C lexers is a big part of the trouble you had to go to do this.
- odvcencio 9mo agoyes basically about 70% of the engineering effort was spent porting the external scanners and ensuring parity with original (C) tree-sitter
- acedTrex 9mo agoClaude attempted a treesitter to go port Better title
- odvcencio 9mo agowell how did it do?
- wocram 9mo agoShouldn't you be able to answer that?
- odvcencio 9mo agoyes and if you clicked the links you would know that i did answer it in the readme.
- do_not_redeem 9mo ago> Pure-Go tree-sitter runtime — no CGo, no C toolchain, WASM-ready. No you didn't. The readme is obvious LLM slop. Em-dash, rule of three, "not x, y". Why should anyone spend effort reading something you couldn't be bothered to write? Why did you post it to HN from a burner account?
- mathfailure 9mo agoBut how do we know the readme isn't also vibecoded?
- ctmnt 9mo agoI read the README and did not find answers to my questions.
- ctmnt 9mo agoHard to say. Claude’s very good at writing READMEs. In fact, Copilot often complains about docs that sound like they’re about current capabilities when in fact they’re future plans or just plan aspirational. Without downloading and testing out your software, how can we know if it’s any good? Why would we do that if it’s obviously vibed? The dilemma. I’m not at all against vibe coding. I’m just pointing out that having a nice README is trivial. And the burden of proof is on you.
- shayief 9mo agoThis is great, I was looking for something like this, thanks for making this! I imagine this can very useful for Go-based forges that need syntax highlighting (i.e. Gitea, Forgejo). I have a strict no-cgo requirement, so I might use it in my project, which is Git+JJ forge https://gitncoffee.com https://gitncoffee.com.
- odvcencio 9mo agothank you for the kind words! Very cool project! Very happy you can find some utility in it
- silverwind 9mo agoGitea is definitely watching this one. Initialy tests show a 20x increase in syntax highlighting speed compared to the previous regexp-based approach.
- trickypr 9mo agoDo you have an equivalent of TreeCursors or tree-sitter-generate? There are at least some use cases where neither queries nor walks are suitable. And I have run into cases where being able to regenerate and compile grammars on the fly is immeasurably helpful. At least for my use cases, this would be unusable. Also, what the hell is this: > partial [..] missing external scanner Why do you have a parsing mode that guarantees incorrect outputs on some grammars (html comes to mind) and then use it as your “90x faster” benchmark figure?
- odvcencio 9mo agothe 90x figure is on Go source for apples to apples against CGO bound tree-sitter. your use case is not one i designed for although yeah maybe the readme has some sections too close. the only external scanner missing atm is norg. now that i know your use case i can probably think of a way to close it
- trickypr 9mo agoSo your benchmarks are primarily just “how fast is go’s c interop” rather than any algorithmic improvement on tree-sitter? Edit: yep, you are just calling a c function in a loop. So your no-op benchmark is just the time it takes for cgo to function. I would not be able to get any perf benefits from e.g. rust
- up2isomorphism 9mo ago"rewrite" a nice code base without mentioning it is vibe coded is not great. Essentially you use AI to somehow re-implement the original code base in a different language, made it somehow work, and claim it is xx times faster. It is misleading.
- odvcencio 9mo agoi really appreciated this comment the most because of how much work "somehow" is doing here
- silverwind 9mo agoI agree it's misleading. Ideally the author would disclose how much of the tree-sitter test suite passes.
- herrington_d 9mo agoI also have a tree-sitter Rust rewrite. Though I cannot find it more useful for end users... https://github.com/HerringtonDarkholme/tree-sitter https://github.com/HerringtonDarkholme/tree-sitter
- kopirgan 9mo agoCan someone please explain what's the connection between this and LSP? For example in Helix can one use this instead of various language servers?
- mojifwisi 9mo agoTree-sitter is merely a tool for generating an AST for a given language. LSPs on the other hand have way more capabilities (formatting, diagnostics, project-wise go to definition, inlay hints, documentation on hover, etc.) as you can see in its specification.[0] They can't really replace one another. [0]: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/ https://microsoft.github.io/language-server-protocol/specifi...
- kopirgan 9mo agoThanks!
- nnx 9mo agoThis looks very interesting, but I wonder how's the rewrite approach gonna impact the long-term maintenance and porting changes _back_ from Tree Sitter. As you mention WASM-readiness, did you consider using the official Tree Sitter WASM builds nicely packaged with wazero (pure Go WASM runtime) ? It may help staying sync with upstream for the long term and, while probably a bit slower, has nice security and GC advantages too.
- yumosx 9mo ago[dead]
- CodeCompost 9mo agoI've seen a lot of "I've ported X to Go/Rust" posts lately. Is it the expectation that we're all supposed to abandon the original projects in favor of the ported versions which use newer and shinier programming languages? Is development going to continue on those new "Go/Rust" ports or are they just one-off karma farming projects?
- monster_truck 9mo agoWas excited to try this in my project but it doesn't seem like it's truly a complete port.
- brodouevencode 9mo agoNeat, but it really bothers me when projects don't use standard layouts.