3 ms·

I think C# is really going to shine in the LLM coding era. You can write Roslyn Analyzers to fail the build on arbitrary conditions after inspecting the AST. LL

I think C# is really going to shine in the LLM coding era. You can write Roslyn Analyzers to fail the build on arbitrary conditions after inspecting the AST. LLMs are great at helping you write these too. If you get a solid architecture well defined you can then use these as guardrails to constrain development to only happen in the manner you intend. You can then get LLMs to implement features and guarantee the code comes out in the shape you expect it to. This works well for humans too, but custom analysers are abstract and not many devs know how to write them, so they are mostly provided by library authors. However, being able to generate them via LLMs makes them so much more accessible, and IMHO is a game changer for enforcing an architecture. I've been exploring this direction a lot lately, and it feels very promising.

bgrainger 1y ago
Completely agree, and I've started writing more Roslyn analyzers to provide quick feedback to the LLM (assuming you're using it in something like VS Code that exposes the `problems` tool to the model). I also want C# semantics even more closely integrated with the LLM. I'm imagining a stronger version of Structured Model Outputs that knows all the valid tokens that could be generated following a "." (including instance methods, extension properties, etc.) and prevents invalid code from even being generated in the first place, rather than needing a roundtrip through a Roslyn analyzer or the compiler to feed more text back to the model. (Perhaps there's some leeway to allow calls to not-yet-written methods to be generated.) Or maybe this idea is just a crutch I'm inventing for current frontier models and future models will be smart enough that they don't need it?
raddan 1y ago
Can you expand a little? What you’re suggesting sounds a bit like program verification, or at least program analysis. But what properties are you checking? I have written many program analyses (though never any for C#; I’ll have to check it out), and my experience is that they are quite challenging to write. Many are research-level CS, so well outside the skill set of your average vibe coder. I’m wondering if you have some insight about LLM generated code that has not occurred to me…
seanmcdirmid 1y ago
Library authors don’t really provide custom analyzers. heck, the best we can hope for are some regex based linting rules, anything that involves local data flow analysis is very rare, and anything inter procedural is non-existent. Program analysis is a dark hole, you are better off just making stronger type systems, but then type inference starts to bite you if you want to support it (and you will given how annoying type annotations are to write, unless you go with something simple like a purely structural type system so you can use Hindley Milner).
- latentsea 1y ago
  Analyzers are a first class citizen in C#. You can get access to the AST during compile time and use it to output diagnostics with error or warning level, so it's more robust than just regex. I've not seen teams personally write them because they're abstract and most devs shy away from it, but working in the C# ecosystem the one place I do see them pop up occasionally is from library authors. For example xunit has some. https://github.com/xunit/xunit.analyzers https://github.com/xunit/xunit.analyzers So far I have found the value in them to be they help you constrain the possible valid moves that can be made in a codebase. This is valuable with teams of human engineers, but even more so with LLMs. It just so happens LLMs are really good at helping you write them too given you know what you want to enforce. I doubt the tooling is as good in other languages as it is in C# in this respect, but at least for devs working in the C# ecosystem LLMs have unlocked access to writing custom analysers on a whole new level, and with that it's now significantly easier to define and enforce rules regarding what constitutes a valid program, such that the set of valid programs matches your intended architecture.
aitchnyu 1y ago
I'm looking at AST based tools in Python to lint, enforce modularity, ban certain patterns. LLMs allow me to write scripts to find recursive function calls, calling super().method() in overrides etc. https://pypi.org/project/import-linter/ https://pypi.org/project/import-linter/ https://github.com/hchasestevens/astpath https://github.com/hchasestevens/astpath
torginus 1y ago
I do quite a bit of coding in C#, and have a lot of experience, and personally I haven't found LLMs to be that great a help at writing C#. First, LLMs are great at learning new tech stacks, but good ol' ASP.NET has been pretty much stable since forever. Second, I think Rider/Resharper is the greatest piece of autocomplete tech ever made, seriously nothing ever comes, close, which means I'd rather do a refactor using them than do something similar by prompting the AI and hoping for the best. Also probably my experience makes me far less accepting of LLMisms, but that might just be on me. Lastly, AI seems to be focused around its own set of tooling, like Cursor, which is fine for TS but is far worse than Rider for things like C#. I know I could kludge things together, but still. As for Roslyn... I have some experience writing codegen/analyzers at my company and it feels like typical a Microsoft tech product, like WPF or Powershell. Brilliant idea (that's a market first as well) combined with really solid technical fundamentals, but plain confusing and overcomplicated UX, that makes it a chore to use. Seriously the amount of scaffolding you need to make even for a simple analyzer is just nuts
- theshrike79 1y ago
  > Lastly, AI seems to be focused around its own set of tooling, like Cursor Nah, the best coding LLMs are console applications like Claude Code, Codex CLI and the like. Editor integration mostly brings more tools, like tapping into different validators on VSCode and examining the "problems" view. Also Rider's autocomplete is at least partially AI powered unless you specifically disable it IIRC.
- pjmlp 1y ago
  Why I never bothered writing one is the scaffolding, and the dumb idea to write code with WriteLines instead of a nice experience like T4 templates.
- latentsea 1y ago
  My point is I can my an analyser in like 20 minutes now and it's not a chore at all. I've made like 15 for my current codebase, and when the LLM goes off the rails and generates code in a pattern I don't like, I don't tell it how to write the code much anymore, I tell it to write an analyser that prevents it from writing the code that way and then it goes about fixing it up because now the build fails. I do a lot of coding in C# with Rider, and refactoring has been a career speciality of mine. I personally find LLMs to have a tonne of value in this space.
pjmlp 1y ago
As someone whose C# is one of the main work ecosystems, I highly doubt it. What I am seeing it that LLMs will push current programming languages down the stack, like now you're enjoying C# => MSIL => Machine code. On my line of work I already can imagine the other side of the tunnel, more low-code/no-code tooling, orchestration agents, and much (much) less manually writing C#, Java and TypeScript.
- latentsea 1y ago
  I don't disagree with your take. My take on your take is that via what I'm suggesting I can envision that the low-code/no-code tooling can be expanded to produce a wider variety of more flexible programs with robust, consistent C# code underpinning them.
KurSix 1y ago
Linters are great at catching specific pattern violations, but they’re useless against bad decomposition or a poorly chosen abstraction. An LLM can generate code that passes all 100 linters and still ends up being a logical mess - with business logic in the wrong layer and completely unmaintainable.
- prmph 1y ago
  Exactly this. LLMs may do a passable job of architecture if there are many examples of high quality architecture similar to what you want to do in their training set, but to introduce some novel stuff and they are clueless
- latentsea 1y ago
  With Roslyn Analyzers and things like ArchUnit I've found its possible to actually write linters that enforce a predetermined architecture with established patterns such as enforcing usage of particular base or framework level classes in specific layers/locations. I agree with the assessment LLMs aren't great at novel architectural work. I'm merely reporting my experience that using LLMs to write analysers that enforce established patterns takes the output from random to well ordered and provides a nice productivity boost in a constrained, but valuable, set of circumstances. It's not a complete solve, but it's a big improvement over just prompting them and hoping for the best.
  - KurSix 1y ago
    This approach is perfect for mature projects with a well-established architecture. But it could be counterproductive in the early R&D stages when the architecture is still fluid and the team is constantly experimenting. The rigid constraints of the analyzers might stifle creativity. So it's a powerful tool, but for the right stage of a project's lifecycle

bgrainger 1y ago

Completely agree, and I've started writing more Roslyn analyzers to provide quick feedback to the LLM (assuming you're using it in something like VS Code that exposes the `problems` tool to the model). I also want C# semantics even more closely integrated with the LLM. I'm imagining a stronger version of Structured Model Outputs that knows all the valid tokens that could be generated following a "." (including instance methods, extension properties, etc.) and prevents invalid code from even being generated in the first place, rather than needing a roundtrip through a Roslyn analyzer or the compiler to feed more text back to the model. (Perhaps there's some leeway to allow calls to not-yet-written methods to be generated.) Or maybe this idea is just a crutch I'm inventing for current frontier models and future models will be smart enough that they don't need it?

raddan 1y ago

Can you expand a little? What you’re suggesting sounds a bit like program verification, or at least program analysis. But what properties are you checking? I have written many program analyses (though never any for C#; I’ll have to check it out), and my experience is that they are quite challenging to write. Many are research-level CS, so well outside the skill set of your average vibe coder. I’m wondering if you have some insight about LLM generated code that has not occurred to me…

seanmcdirmid 1y ago

Library authors don’t really provide custom analyzers. heck, the best we can hope for are some regex based linting rules, anything that involves local data flow analysis is very rare, and anything inter procedural is non-existent. Program analysis is a dark hole, you are better off just making stronger type systems, but then type inference starts to bite you if you want to support it (and you will given how annoying type annotations are to write, unless you go with something simple like a purely structural type system so you can use Hindley Milner).

latentsea 1y ago
Analyzers are a first class citizen in C#. You can get access to the AST during compile time and use it to output diagnostics with error or warning level, so it's more robust than just regex. I've not seen teams personally write them because they're abstract and most devs shy away from it, but working in the C# ecosystem the one place I do see them pop up occasionally is from library authors. For example xunit has some. https://github.com/xunit/xunit.analyzers https://github.com/xunit/xunit.analyzers So far I have found the value in them to be they help you constrain the possible valid moves that can be made in a codebase. This is valuable with teams of human engineers, but even more so with LLMs. It just so happens LLMs are really good at helping you write them too given you know what you want to enforce. I doubt the tooling is as good in other languages as it is in C# in this respect, but at least for devs working in the C# ecosystem LLMs have unlocked access to writing custom analysers on a whole new level, and with that it's now significantly easier to define and enforce rules regarding what constitutes a valid program, such that the set of valid programs matches your intended architecture.

aitchnyu 1y ago

I'm looking at AST based tools in Python to lint, enforce modularity, ban certain patterns. LLMs allow me to write scripts to find recursive function calls, calling super().method() in overrides etc. https://pypi.org/project/import-linter/ https://pypi.org/project/import-linter/ https://github.com/hchasestevens/astpath https://github.com/hchasestevens/astpath

torginus 1y ago

I do quite a bit of coding in C#, and have a lot of experience, and personally I haven't found LLMs to be that great a help at writing C#. First, LLMs are great at learning new tech stacks, but good ol' ASP.NET has been pretty much stable since forever. Second, I think Rider/Resharper is the greatest piece of autocomplete tech ever made, seriously nothing ever comes, close, which means I'd rather do a refactor using them than do something similar by prompting the AI and hoping for the best. Also probably my experience makes me far less accepting of LLMisms, but that might just be on me. Lastly, AI seems to be focused around its own set of tooling, like Cursor, which is fine for TS but is far worse than Rider for things like C#. I know I could kludge things together, but still. As for Roslyn... I have some experience writing codegen/analyzers at my company and it feels like typical a Microsoft tech product, like WPF or Powershell. Brilliant idea (that's a market first as well) combined with really solid technical fundamentals, but plain confusing and overcomplicated UX, that makes it a chore to use. Seriously the amount of scaffolding you need to make even for a simple analyzer is just nuts

theshrike79 1y ago
> Lastly, AI seems to be focused around its own set of tooling, like Cursor Nah, the best coding LLMs are console applications like Claude Code, Codex CLI and the like. Editor integration mostly brings more tools, like tapping into different validators on VSCode and examining the "problems" view. Also Rider's autocomplete is at least partially AI powered unless you specifically disable it IIRC.
pjmlp 1y ago
Why I never bothered writing one is the scaffolding, and the dumb idea to write code with WriteLines instead of a nice experience like T4 templates.
latentsea 1y ago
My point is I can my an analyser in like 20 minutes now and it's not a chore at all. I've made like 15 for my current codebase, and when the LLM goes off the rails and generates code in a pattern I don't like, I don't tell it how to write the code much anymore, I tell it to write an analyser that prevents it from writing the code that way and then it goes about fixing it up because now the build fails. I do a lot of coding in C# with Rider, and refactoring has been a career speciality of mine. I personally find LLMs to have a tonne of value in this space.

pjmlp 1y ago

As someone whose C# is one of the main work ecosystems, I highly doubt it. What I am seeing it that LLMs will push current programming languages down the stack, like now you're enjoying C# => MSIL => Machine code. On my line of work I already can imagine the other side of the tunnel, more low-code/no-code tooling, orchestration agents, and much (much) less manually writing C#, Java and TypeScript.

latentsea 1y ago
I don't disagree with your take. My take on your take is that via what I'm suggesting I can envision that the low-code/no-code tooling can be expanded to produce a wider variety of more flexible programs with robust, consistent C# code underpinning them.

KurSix 1y ago

Linters are great at catching specific pattern violations, but they’re useless against bad decomposition or a poorly chosen abstraction. An LLM can generate code that passes all 100 linters and still ends up being a logical mess - with business logic in the wrong layer and completely unmaintainable.

prmph 1y ago
Exactly this. LLMs may do a passable job of architecture if there are many examples of high quality architecture similar to what you want to do in their training set, but to introduce some novel stuff and they are clueless
latentsea 1y ago
With Roslyn Analyzers and things like ArchUnit I've found its possible to actually write linters that enforce a predetermined architecture with established patterns such as enforcing usage of particular base or framework level classes in specific layers/locations. I agree with the assessment LLMs aren't great at novel architectural work. I'm merely reporting my experience that using LLMs to write analysers that enforce established patterns takes the output from random to well ordered and provides a nice productivity boost in a constrained, but valuable, set of circumstances. It's not a complete solve, but it's a big improvement over just prompting them and hoping for the best.
- KurSix 1y ago
  This approach is perfect for mature projects with a well-established architecture. But it could be counterproductive in the early R&D stages when the architecture is still fluid and the team is constantly experimenting. The rigid constraints of the analyzers might stifle creativity. So it's a powerful tool, but for the right stage of a project's lifecycle