13 ms·

How Microsoft rewrote its C# compiler in C# and made it open source (2017)

• fultonfaknem42 7y ago
  I still can't believe more people aren't leveraging the Roslyn APIs to write compiler extensions or additional tools around C#. It's conceptually powerful.
  • eropple 7y ago
    Personally speaking, I'd love to, but I'd then have to figure out how to make them work with an IDE. I'm very tired of waiting for record classes to show up and I could have written a (to be clear: inferior) set of stuff around regular classes that magics one into "everything is readonly and we autogenerate a `copy` method", much like Kotlin does for its data classes...but my IDE isn't gonna understand it unless I do a lot more work, and so I never bothered.
    • moron4hire 7y ago
      I'm not sure that it would be that difficult with Roslyn and Visual Studio these days. There are Roslyn-based syntax-highlighters and linters and snippet-generators and code-transformers. I use one for making sure all of my code is not just formatted the way I want it, but auto-inserting "readonly" modifiers for fields that don't get re-written outside of the constructor, one that treats not implementing IDisposable correctly as an error (it can also track the lifetime of a Disposable object and warn when it detects that it never gets disposed, which is super cool), and another one that rainbow-highlights code blocks. The tooling is there to support just a thing, it just needs someone to put all the pieces together.
      • thrower123 7y ago
        What is the name of that IDisposable checking one? That sounds very, very useful.
        • moron4hire 7y ago
          I think the easiest way to set it all up is to manually edit your CSProj files, especially if you are still building .NET Framework projects. By default, Visual Studio will only create the new SDK Style project file format for .NET Standard and .NET Core projects, but it's still usable for .NET Framework projects if you manually change the format. Once you change it, it sticks, so you can use VS to edit the config after that, but it's still pretty easy to edit by hand now. So here is my base project config: https://github.com/capnmidnight/Juniper/blob/master/Juniper.props https://github.com/capnmidnight/Juniper/blob/master/Juniper.... The most important part is the first PropertyGroup sets values for all build configs, in particular is setting LangVersion to 8.0. Framework 4.8 taps out at C# 7.2, but you can use most of the C# 8.0 features, including fully async streams if you manually set the language version. Features that aren't available are some minor things like the array ranges and indexing: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-8.0/ranges https://docs.microsoft.com/en-us/dotnet/csharp/language-refe... And here are my base project with the analyzers I use: https://github.com/capnmidnight/Juniper/blob/master/Juniper.targets https://github.com/capnmidnight/Juniper/blob/master/Juniper.... They're all ones provided directly from Microsoft, though there are a bunch more from other vendors: https://www.nuget.org/packages?q=analyzer https://www.nuget.org/packages?q=analyzer Then here is an example project using that targets file: https://github.com/capnmidnight/Juniper/blob/master/src/Juniper.Server/Juniper.Server.csproj https://github.com/capnmidnight/Juniper/blob/master/src/Juni... You can see just how much the new SDK Style project file format simplifies things. There is no importing of any base Targets files hidden deep in Visual Studio's install directory anymore. I manually import the .props and .targets file instead of using Directory.Build.props and Directory.Build.targets because I have other projects that use these configs, included via a git submodule. Here is my .editorconfig file, where I set most of the rules related to Disposable types to errors: https://github.com/capnmidnight/Juniper/blob/master/.editorconfig https://github.com/capnmidnight/Juniper/blob/master/.editorc... And this Visual Studio extension makes .editorconfig files a lot nicer to work with: https://marketplace.visualstudio.com/items?itemName=MadsKristensen.EditorConfig https://marketplace.visualstudio.com/items?itemName=MadsKris... (BTW, I pretty much install all of Mads Kristensen's extensions) And while I'm here, I'll give a shout-out to Viasfora for its syntax highlighting modifications that rainbow-highlight code blocks: https://marketplace.visualstudio.com/items?itemName=TomasRestrepo.Viasfora https://marketplace.visualstudio.com/items?itemName=TomasRes... And VSColorOutput for making the Output window in Visual Studio actually readable: https://marketplace.visualstudio.com/items?itemName=MikeWard-AnnArbor.VSColorOutput https://marketplace.visualstudio.com/items?itemName=MikeWard...
      • eropple 7y ago
        Hey, that's cool. Sounds like things have really improved. Maybe next time I get back into C# I'll see about writing the gizmo I want, if somebody else hasn't already. Thanks a lot. (If somebody has--well, I wouldn't mind some lazyweb recommentations...)
    • simplyinfinity 7y ago
      5 years ago i wrote a C# code analyzer that found/suggested and fixed async versions of EntityFramework extension methods in your async functions. Example: if my function was async and it had a .ToList() function call inside it would suggest to use .ToListAsync() and with a click auto fix it in your function here is the repo for reference : https://github.com/aviatrix/YARA https://github.com/aviatrix/YARA It took me couple of days to wrap my head around all of the new concepts, but it was quite fun! To get it integrated with the IDE, you need "CodeAnalyzer" class, and that gets executed automagically and provides annotations in the IDE :)
    • louthy 7y ago
      I have a code-gen for records and discriminated unions [1] (as well as other useful features). It will generate a record-type at build-time (which includes the background build process in VS). All that's needed is a [Record] or [Union] attribute: [1] https://github.com/louthy/language-ext/wiki/Code-generation https://github.com/louthy/language-ext/wiki/Code-generation
      • eropple 7y ago
        You just made my day. Not kidding. Thank you.
  • tasogare 7y ago
    I looked at bit when I was in preview. It’s indeed powerful but also quite verbose and it distinguish between a lot of concepts, so there is a steep learning curve. Then there was the lack of examples beyond a few blog posts.
    • Ididntdothis 7y ago
      Agreed. I have a few ideas and it would be nice to see more examples. Right now it’s hard to make sense of things without spending a lot of time on it.
    • moron4hire 7y ago
      Oh man, the verbosity thing was a huge problem until C# got "using static". Any static classes that just hold static methods can be imported into your code module as bare functions. I frequently do "using static System.Math;" and "using static System.Console;" when tossing together little mathy processor apps.
  • rafaelvasco 7y ago
    At some point I wrote a barebones scripting system using roslyn for a project of mine. At runtime I got a piece of code compiled to a DLL in memory and then executed this DLL; Worked well; But at that time there was no support for destroying AppDomains or something, don't remember the exact name. Still pretty fun; But yeah, there's no real complete documentation anywhere; And the DLL hell was real. Dozens of DLL's just to support this; But now with .NET Core 3+ things must have improved a lot;
    • moron4hire 7y ago
      DLL Hell doesn't refer to "lots of DLLs", it refers to conflicting versions of DLLs not being easily manageable across multiple applications. Outside of putting assemblies in the GAC (which was always a hack of last resort anyway), .NET has never had "DLL Hell".
      • rafaelvasco 7y ago
        Yeah, lots of assemblies is kinda like .NET dll hell ;) .NET Core had this problem in earlier versions. Not anymore;
      • WorldMaker 7y ago
        Yeah, the closest .NET equivalent is Binding Redirection hell, and NuGet and Visual Studio and .NET Core have spent years of work making that better than it was at its worst in the early NuGet days. There's a couple of low level APIs that are still annoying problems to redirect if you need to support particular sets of .NET Framework and .NET Core, but beyond that a lot of it is managed for developers automatically these days.
        • moron4hire 7y ago
          Yeah, the VS interface for managing it is not great, but the new project file format is a lot easier to understand and manage, so I've had good luck just making manual changes.
      • qes 7y ago
  • m0xte 7y ago
    From my perspective I’d love to but from writing c# for nearly 20 years now it’s a rough ride for anyone trying to keep up with the platform. Knowledge is thrown in the trash faster than you can learn it and direction changes of all sorts jump on you. I’m not in for investing my time in that any longer.
    • yonaguska 7y ago
      I don't think that's true. If you're familiar with functional programming concepts- nothing in C# is really that new. In my experience, C# is just getting easier to use as they add more language support for things you'd normally have to implement on your own.
      • m0xte 7y ago
        The core language is fine but the surrounding ecosystem is volatile.
  • ygra 7y ago
    We've written a custom compiler from C# to Java and JavaScript based on Roslyn. Over time it gained more features and target languages as well (Python is in progress, we can also emit a working GWT wrapper for the JavaScript output, we can emit TypeScript typings or just normal TypeScript as well, etc.). For us this helps us in offering our products on various different platforms without having to write the code in different places anew. Since our product is a library, not an application, we couldn't really take advantage of existing conversion tools that mostly take the path of converting IL to hideous code and an entry point. The whole thing is now used in basically every library build we have at some point, even for the C# versions, as it ties in with our documentation writing process and places the correct API names and links for that product into the documentation, even though the docs start with mostly the same content for each. I agree that lack of documentation makes working with Roslyn a bit daunting at times, although the API is very well designed and oftentimes it's very obvious where to look for something. I was also very impressed by their compatibility efforts. We started while Roslyn was in beta and upgrading through the releases worked without a hitch.
    • fultonfaknem42 7y ago
      I'll second the API being very well designed. It is incredibly legible from a technical perspective and it has actually been generally a joy to figure it out, as opposed to just being told how it works. That being said: I need some bathroom friendly reading material every now and again.
• chrisseaton 7y ago
  Why isn't the native code compiler also now written in C#, like Java is doing?
  • eropple 7y ago
    I imagine it's on a roadmap somewhere. Having the code->IL layer be readily accessible from managed code will have bigger developer impact than the IL->machine code step, though.
  • phillipcarter 7y ago
    Depending on what you consider the runtime, a lot of it is already written in C#. And many new concepts are written in C# when they may have been written with C++ in the past, in part due to C# supporting more low-level concepts than before. That said, large-scale rewrites don't happen just because engineers feel like doing them. In the case of Roslyn, there were multiple compilers (C# language compiler, VB language compiler, C# tooling compiler, VB tooling compiler, etc.), requiring an enormous cost to evolve C# and VB as languages while ensuring that end users of these languages had a good experience using tools like Visual Studio. Beyond that, there was a host of additional tooling to provide more advanced code analysis that was equally expensive to maintain and evolve as the languages evolved. This meant that the kinds of language and tooling innovations end users expected was more challenging to meet. A similar set of challenges - focused on problems centered around the end users - must exist to justify the enormous cost of rewriting a massive engineering system.
  • kevingadd 7y ago
    People have done research before into having the C#->native JIT be written in C#, I recall seeing prototypes. There are a ton of barriers between a prototype and a shipping implementation though, so I'm not sure we'll ever see it. In particular you risk regressions in startup time or memory usage since the amount of infrastructure needed to run C# (pre-jitted?) to generate all your jitcode is much higher than a small blob of hand-written C/C++ that's spitting out jitcode. See https://www.mono-project.com/news/2018/09/11/csharp-jit/ https://www.mono-project.com/news/2018/09/11/csharp-jit/ for one example. There's an old saying (from one of Unity's developers, I think?) that you can't run Hello World in C# or Java without an XML parser because so much configuration for things like locales ends up pulling in serialization libraries, reflection, etc. Things have improved in this area for both platforms but it's definitely still very difficult to trim managed code down as far as you can trim native code. For something as critical as the JIT you also want to keep the generated code small (for cache efficiency) and if possible, PGO it - things that existing managed code generators aren't especially great at compared to clang or modern MSVC. From my past experience writing/maintaining C# compiler and runtime code, I would never bother trying to port the JIT to C#. I don't think the returns are worth the massive investment. I'd sooner port it to a language like Rust for safety benefits or to some sort of hypothetical language that enables producing smaller/faster code to improve JIT performance. I'm not sure I'd invest in that either though because for most workloads JIT time is not the bottleneck (and you can optimize that out in many cases by pre-generating the JITcode). EDIT: Also, for workloads where JIT is the bottleneck, it's questionable whether it's possible to extract big gains out of optimizing the JIT because of the nature of JIT workloads - you may just be bottlenecked on memory bandwidth or instructions per clock. A JIT converting IR to machine code is not trivially vectorized. People are figuring this out (again, from scratch) now with webassembly as all the modern browsers go through the churn and angst involved in answering the question 'can we actually JIT this whole 50mb executable from scratch at load time?' even though we already knew the answer back when WebAssembly wasn't a spec yet. [DISCLOSURE: I get paid to work on Mono right now and was paid to help draft the initial WebAssembly spec, and before that my work on the JSIL MSIL->JS compiler was sponsored. So I have some massive biases here.]
• nudpiedo 7y ago
  I find amusing how microsoft moved from the closed source referent in the industry into such an open source player. Right now even allows to hook some of their tools and platforms to its competing platforms and tools.
  • alexlesuper 7y ago
    I think they are really repositioning the company as powering general computing through their cloud platform as opposed to powering general computing through their OS.
    • Ididntdothis 7y ago
      I think they will slowly turn into IBM. Maybe a better version of IBM though.
      • thrower123 7y ago
        I hope that they have no interest in becoming a global services company...
        • Ididntdothis 7y ago
          I think they are becoming more and more enterprise oriented so global services company is the way to go.
          • Mountain_Skies 7y ago
            Wonder if this will eventually lead to the company splitting into separate consumer and enterprise companies as the overlap between the two halves grows smaller.
    • deleted 7y ago
      [deleted]
  • m0xte 7y ago
• The_rationalist 7y ago
  It should be noted they are now contributing to openjdk too!
• _hardwaregeek 7y ago
  Roslyn's parser and syntax tree is pretty amazing. You can recreate the precise source text from the parse tree up to whitespace. This sort of "bijective parsing" is truly incredible and probably one of the cooler innovations I've seen in parsing technology. I can see a bunch of really interesting ideas that you could do with bijective parsing. For instance, imagine Rails style boilerplate generation but done at a semantics aware, within file basis. You could conceivably have a code generator that finds a class, introspects if it has the corresponding method, then generates it if not. Or imagine syntax reformatting but purely locally. Or semantically aware git diffs that can actually compare the underlying parse trees instead of just raw text. There's so much cool stuff you can do. I wish every language had a parser like this.
  • munificent 7y ago
    Dart's parser can do this too. The formatter (dartfmt) takes advantage of this in order to preserve comments and use the original formatting as a hint to the output in some cases.
  • Sharlin 7y ago
    Pardon me if I'm missing something, but doesn't just about any AST implementation allow recovering the source text up to whitespace – or indeed including whitespace, given that most real-world parsers have to retain lexical information in order to support reasonable diagnostic messages anyway.
    • _hardwaregeek 7y ago
      I'm not sure to be honest. I suspect most parse trees preserve most of the info but not up to bijection. For instance, Ruby has ways of taking the Ripper output and generating source text but there's no guarantee that this output will be precisely the original source text. After all, the parser would need to distinguish between spaces and tabs and store them as tokens to reproduce the text fully.
    • fierarul 7y ago
      > doesn't just about any AST implementation allow recovering the source text up to whitespace That's correct. Especially for parsers used in IDEs you most definitely care about exact locations of everything, including whitespace.
• pcunite 7y ago
  If .NET/C# could output native binaries for Windows Desktop apps, I could seriously think about switching to C# over C++. How do people deal with securing their source code otherwise? Quite trivial to get the source from a decompiled C# exe file.
  • dancek 7y ago
    You can reverse engineer native binaries, too. Many companies are happily publishing web apps where the actual source is available, in minified form. Why does your source code need to be so secure?
  • mads_ravn 7y ago
    You can use an obfuscator (see e.g. https://stackoverflow.com/questions/19163701/how-can-i-obfuscate-my-c-sharp-code-so-it-cant-be-deobfuscated-so-easily https://stackoverflow.com/questions/19163701/how-can-i-obfus...). Note that you can still decompile the obfuscated code and look around (I’ve done so to attempt to debug a 3rd party library), but mangling all identifiers makes it quite hard to read.
  • LeifCarrotson 7y ago
    It's quite trivial to get the source from a C# exe file in the same way as a decompiler makes it trivial to get the source from a C++ exe file. Obfuscators are common if you're concerned about the symbols leaking. If you don't include the .PDB debugging symbol files it's much the same as native binary code. The .NET virtual machine code is a little more expressive but is superficially similar to x86 native code. In my opinion, if you're concerned about people reading your compiled machine code, the only solution is to run your app on a server and give users an API.
    • pcunite 7y ago
      A C# binary can be decompiled back to easy to read source code. Show me how this can be done with a compiled C++ binary. It is a valid concern.
      • 7y ago
• jaked89 7y ago
  While the rewrite enabled faster development of the language as a whole, it also gradually destroyed the IDE's performance. The editor in VS 2019 is simply unworkable. I blame this directly on the immutable AST. While a nice concept in theory, it causes too many allocations, and is cumbersome to work with. I predict another rewrite in 2 or 3 years.
  • alkonaut 7y ago
    I had to ditch ReSharper to get to 2019 because together with the performance of the IDE itself, it just wasn't usable. R# ate gigs of Ram and Roslyn does the same. It's not surprising since they basically do the same thing, in managed code! But I can't pay the CPU time and memory to analyze my code TWICE on every edit. I also suspect things like switching build configuration offers thousands of opportunities to have some IDE widget hold references to old compilation data structures which are never garbage collected. On the bright side at least 2019 works pretty well without R#.
    • hadrien01 7y ago
      I did the exact opposite: I migrated to Rider, an IDE built around ReSharper. In my experience, it is much faster than VS2019 for my use case (including a huge monolith with hundreds of projects)
    • to11mtm 7y ago
      > I had to ditch ReSharper to get to 2019 because together with the performance of the IDE itself, it just wasn't usable. R# ate gigs of Ram and Roslyn does the same. It's not surprising since they basically do the same thing, in managed code! But I can't pay the CPU time and memory to analyze my code TWICE on every edit. I'm holding off on 2019 as much as I can. Between 'forcing' an upgrade for .NET Core 3.0 and the fact Resharper slows it down too much, I decided to give Rider a try. I'm finding myself not missing VS a whole lot; on one hand Rider is taking way more RAM to start and load, but it stays pretty constant after the first debug session, winds up staying under VS for memory on longer loads (Especially if I've got multiple solutions open) and it's smoother than VS the whole time.
• swasheck 7y ago
  I've always loved C# but can't quite get past having to target specific .net frameworks and keeping the different frameworks in my different servers straight and targeting each of them differently. Maybe that'll change with .net core (and once I'm able to get that on my servers), but for now I've discovered a personal love for Go as a way around this for my small utilities.
  • cpascal 7y ago
    The madness will hopefully be ending soon with .NET 5. https://devblogs.microsoft.com/dotnet/introducing-net-5/ https://devblogs.microsoft.com/dotnet/introducing-net-5/ > There will be just one .NET going forward, and you will be able to use it to target Windows, Linux, macOS, iOS, Android, tvOS, watchOS and WebAssembly and more.
  • manigandham 7y ago
    .NET Core doesn’t need to be preinstalled. You can package a self contained deployment now and it comes with all of its dependencies. Or use one of the docker images to build on and run your app in a container.
  • wvenable 7y ago
    I say it's a mess but it isn't a hassle. Targeting .NET standard or .NET core is pretty straight forward and easy. But the fact that all this exists is a mess and thankfully MS finally has a good plan to clean it up.
  • WorldMaker 7y ago
    .NET Core 3.1 is a good LTS release and no time like the present to start migrating to it. Unlike .NET Framework, there's a lot less emphasis in .NET Core on machine-wide installs and a lot more capabilities for application-specific framework deployments. .NET Core even has tools to bundle all of your framework dependencies into a single "EXE", and tree-shaking that down to at least something of a minimal bundle size. There's a small but growing world of "go-like" small utilities that are entirely self-contained .NET Core dependencies. https://www.hanselman.com/blog/MakingATinyNETCore30EntirelySelfcontainedSingleExecutable.aspx https://www.hanselman.com/blog/MakingATinyNETCore30EntirelyS... There's even fun experiments of AOT compiling to get interesting in "EXE golf" results from .NET Core applications such as getting them below 8 KB or running on Windows 3.1 (because why not): https://www.hanselman.com/blog/NETEverywhereApparentlyAlsoMeansWindows311AndDOS.aspx https://www.hanselman.com/blog/NETEverywhereApparentlyAlsoMe... .NET 5 will integrate further AOT capabilities as the Mono world is merging in, in addition to the raw marketing advantage that 5 > 4 for anyone still struggling to convince non-technical managers that .NET Core is a better investment in 2020 than .NET Framework.
• ablekh 7y ago
  How would you compare special features, if any, as well as advantages (and disadvantages) of using C# for developing an embedded DSL versus using Julia macros for the same purpose versus using specialized toolsets (e.g., MPS) for developing an external DSL? Please note that I'm aware of the Modeling SDK for Visual Studio. However, since it only allows integration with / targets the Visual Studio environment, it is not a good general approach, hence the question.
  • nudpiedo 7y ago
    Maybe you can look at F#, it will compile for Android, iOS, windows, Linux, MAC, javascript frontends and server side. The feature set is different than Julia’s but it is very powerful and versatile.
    • ablekh 7y ago
      Thank you for the suggestion. Will definitely take a more detailed look at F#. I have read about it some time ago, including some very positive general feedback. However, even if F# is an excellent fit feature-wise, I can see two potential issues: 1) lack of a decent package ecosystem (since the planned DSL is only one part of multi-feature multi-aspect platform puzzle) and 2) lack of a significant enough pool of experienced developers, which would make building a good team a challenge (due to its relative popularity, C# is IMO much better than F# in this regard).
• ryuukk_ 7y ago
  they wasted precious time wile they spent time rewriting the same shit people came with swift, go and rust c#, just like java, both old people who refuse to get rid of their JIT bullshit, RIP to both of them