4 ms·
"Creative" grammar introduces parsing difficulties, which makes IDE tooling harder to build and less effective overall. My overall guess is that Rust made the
by zozbot234 5y ago
"Creative" grammar introduces parsing difficulties, which makes IDE tooling harder to build and less effective overall. My overall guess is that Rust made the right choices here, though one can endlessly bikeshed about specifics.
- KptMarchewa 5y agoTooling should not depend on code text, but on language's AST.
- tripa 5y agoComments tending to skip on being a part of the AST make that harder.
- bobbylarrybobby 5y agoIn practice though, getting the AST from the text is a computational task in and of itself and the grammar affects the runtime of that. For instance, Rust's "turbofish" syntax, `f::<T>(args)`, is used to specify the generic type for f when calling it; this is instead of the perhaps more obvious `f<T>(args)`, which is what the definition looks like. Why the extra colons? Because parsing `f<T>(args)` in an expression position would require unbounded lookahead to determine the meaning of the left angle bracket -- is it the beginning of generics or less-than? Therefore, even though Rust could be modified to accept`f<T>(args)` as a valid syntax when calling the function, the language team decided to require the colons in order to improve worst case parser performance.
- xigoi 5y agoThis is why using the same character as a delimiter and operator is bad.
- colejohnson66 5y agoHow does C# manage to handle without the turbo fish syntax? What’s different in Rust?
- bobbylarrybobby 5y agoIt's not impossible to handle the ambiguity, it's just that you may have to look arbitrarily far ahead to resolve it. Perhaps C# simply does this. Or perhaps it limits expressions to 2^(large-ish number) bytes.
- nightski 5y agoI'm not an expert as I do not work on these tools but I don't think IDEs can rely solely on ASTs because not all code is in a compilable state. Lots of times things have to be inferred from invalid code. Jetbrains tools for example do a great job at this.
- klodolph 5y agoCreative grammar can introduce parsing difficulties, but it doesn't have to. I've made a couple small languages, and it's easy to end up lost in a sea of design decisions. But there are a lot of languages that have come before yours, and you can look to them for guidance. Do you want something like automatic semicolon insertion? Well, you can compare how JavaScript, Python[1], Haskell, and Go handle it. You can even dig up messages on mailing lists where developers talk about how the feature has unexpected drawbacks or nice advantages, or see blog posts about how it's resulted in unexpected behavior from a user standpoint. You can also take a look at some examples of languages which are easy or hard to parse, even though they have similar levels of expressivity. C++ is hard to parse... why? You'd also have as your guiding star some goal like, "I want to create an LL(1) recursive descent parser for this language." There's still a ton of room for creativity within constraints like these. [1]: Python doesn't have automatic semicolon insertion, but it does have a semicolon statement separator, and it does not require you to use a semicolon at the end of statements.
- int_19h 5y agoHave you also written tooling - e.g. code completion in an IDE - for those small languages? There are many things that might be easy to parse when you're doing streaming parsing, but a lot more complicated when you have to update the parse tree just-in-time in response to edits, and accommodate snippets that are outright invalid (because they're still being typed).
- klodolph 5y agoYes, that's a good example of exactly what I'm talking about. Code completion used to be really hard, and good code completion is still hard, but we have all these different languages to learn from and you can look to the languages that came before you when building your language. Just to give some more detail--you can find all sorts of reports from people who have implemented IDE support, talking about the issues that they've faced and what makes a language difficult to analyze syntactically or semantically. Because these discussions are available to sift through in mailing lists, or there are even talks on YouTube about this stuff, you have an wealth of information at your fingertips on how to design languages that make IDE support easier. Like, why is it that it's so hard to make good tools for C++ or Python, but comparatively easier to make tools for Java or C#? It's an answerable question. These days, making an LSP server for your pet language is within reach.