6 ms·

"LINQ Performance improvements" really should read: "our own List<T> implementation performance improvements". Microsoft seem to spend their time improving wha

by louthy 2y ago

"LINQ Performance improvements" really should read: "our own List<T> implementation performance improvements". Microsoft seem to spend their time improving what they need, rather than spending time on general improvements. LINQ (the syntax, not the method extensions) needs investment. Primarily around lambda allocation and potentially on compile-time reduction of lambdas. It's time to have value-type local lambdas, or a strategy that doesn't make lambda allocation the overhead that it is. And also there really should be wildcard support (`_`) for LINQ variables by now. Which was been completely ignored when they were brought in for lambdas. It should also be possible to use a lifted-type (like IEnumerable<T>, Option<T>, etc.) as the final item in a LINQ expression, rather than `select ...`. The select adds overhead that isn't needed for certain use-cases and limits things like tail-recursive LINQ expressions. Libraries like mine that go all-in on LINQ [1], but aren't using `IEnumerable` or `IQueryable`, or any of the LINQ extensions, continually get ignored because MS are focusing purely on improving the performance of their own projects. A good example is the improved lambda inference. It was only brought forward because ASP.NET Core needed it for its minimal API. It seems like many of the features of the language/framework are driven by their own needs rather than those of the community. The absolute worst thing is the ever expanding set of 'magic methods' (like the LINQ extensions Select, SelectMany, and Where, but also GetAwaiter, and the rest). MS are adding capabilities for themselves (the compiler mostly) rather than do what is really needed and add proper higher-kinded traits to resolve the magic. So everything is weakly typed and only vaguely discoverable by the compiler :/ LINQ is one of those key differentiators between languages, yet it's been wallowing, pretty much untouched since C# 3. I think it's a crying shame that it has basically been ignored since then and, even now, they think LINQ is only useful for iterating lists. And primarily only their own list implementations. /rant Don't get me wrong, I appreciate all performance improvements, I'm sure it will help plenty of users. But the focus always seems to be narrowly targeted, which limits the potential. [1] https://github.com/louthy/language-ext/ https://github.com/louthy/language-ext/

neonsunset 2y ago
If you do have useful feedback, please submit an issue or contribute a PR to dotnet/runtime. That’s how many performance improvements in LINQ come to be, that the post discusses.
- louthy 2y ago
  7 years ago: https://github.com/dotnet/csharplang/issues/1060 https://github.com/dotnet/csharplang/issues/1060 3 years ago: https://github.com/dotnet/csharplang/blob/main/meetings/2021/LDM-2021-04-07.md#cost-free-delegates-and-linq https://github.com/dotnet/csharplang/blob/main/meetings/2021... I've given up submitting to csharplang now. It seemed like a positive when it was first open-sourced, but the proposals/issues are often just circular discussions that go nowhere. It doesn't feel like (at least from the outside) any proposal by a non-MS employee would make it thru the process unless MS needed it themselves. Which is what I was alluding to in my original post. I may well be wrong and maybe some proposals have made it through, but it feels like an illusion of community collaboration rather than actual community collaboration imho. I don't expect preferential treatment or anything like that, but it would be good if one of the most standout features of C# got some proper attention after a decade of neglect. There has been a lot of effort on performance in many areas of the framework and the language. Leveraging that work for LINQ (the language feature) would be very welcome.
  - neonsunset 2y ago
    C# is already big enough language. This is more about .NET in general and less about C# specifically. From what I've heard - there is an awareness that the current cost of delegates, while not that problematic anymore, can be improved. Whether this happens via explicit syntax to enable (ref) struct closures and value delegate pattern implementations or via object escape analysis and better devirtualization remains to be seen. p.s.: I really wish that instead of LanguageExt, companies would have adopted F# instead.
    - louthy 2y ago
      > p.s.: I really wish that instead of LanguageExt, companies would have adopted F# instead. Ah the standard "why don't you just use F#" line. I have a standard response to that now ;) https://github.com/louthy/language-ext/wiki/%22Why-don't-you-just-use-F%23%3F%22 https://github.com/louthy/language-ext/wiki/%22Why-don't-you...
progmetaldev 2y ago
Your library looks very interesting, but I think you might be setting things up ahead of time to be dismissed. There are a large number of using statements, which is not a big deal for those who understand breaking up projects into appropriate pieces for exactly what you need, and keeping boundaries between different concerns. Most developers don't structure their projects the way that you do, and I say that as a positive towards your implementation, but often little things like this can hold back "the most common-denominator" when it comes to developers. I will admit right off the bat that you are far more well-versed in C# and .NET than I am, but I'm coming to this as someone that has worked with lots of junior developers that already struggle with standard LINQ syntax, methods, and (in particular) performance. You do mention these things in your README, and I appreciate it, as most are set on "selling" their library instead of actually listing what it exceeds at and the purpose. Your comment on being non-idiomatic is another issue that I see (again, talking about people learning and getting used to C#/.NET). Only in my opinion, I feel like Microsoft want to keep their tools and languages following certain practices, and I get that your naming follows more functional programming practices to make them natural to FP. I feel like often, little issues like this are very large to Microsoft when it comes to looking into improvements. I starred your repo, and am very interested in what you've put together here. On the last few major applications I've built (I normally build content management systems that are highly customized, that I tend to not call apps), I've used a Result<T> type that seems to be roughly the same as Option<T>. I wrote that sentence, and then went back to look at your library, and as much as I thought I knew about FP, I really don't. I think I'm really decent at C# and doing some complex things with it (I mainly use Umbraco CMS, and I've gotten my code to run websites under the minimal recommended requirements of the base Umbraco software). With that said, I read a lot about FP, but still have tons of issues picking it up. F# For Fun And Profit is the closest it's made sense to me. In the end, all this is to say that it's nothing that you're doing wrong. Microsoft is going to target the average or beginner developer, because that is the vast majority of developers that they have in their environment. I hope that you library can gain some internal improvements, because it looks like you've spent an enormous amount of time on this, and you have enough stars on GitHub to at least indicate that people are actively using your library and getting benefit from it. I apologize if any of this came off as dismissive, I think what you've done here is exciting, and feel like you've set up enough documentation where I can slowly pick up concepts that I'm unfamiliar with.

by louthy 2y ago

neonsunset 2y ago

If you do have useful feedback, please submit an issue or contribute a PR to dotnet/runtime. That’s how many performance improvements in LINQ come to be, that the post discusses.

louthy 2y ago
7 years ago: https://github.com/dotnet/csharplang/issues/1060 https://github.com/dotnet/csharplang/issues/1060 3 years ago: https://github.com/dotnet/csharplang/blob/main/meetings/2021/LDM-2021-04-07.md#cost-free-delegates-and-linq https://github.com/dotnet/csharplang/blob/main/meetings/2021... I've given up submitting to csharplang now. It seemed like a positive when it was first open-sourced, but the proposals/issues are often just circular discussions that go nowhere. It doesn't feel like (at least from the outside) any proposal by a non-MS employee would make it thru the process unless MS needed it themselves. Which is what I was alluding to in my original post. I may well be wrong and maybe some proposals have made it through, but it feels like an illusion of community collaboration rather than actual community collaboration imho. I don't expect preferential treatment or anything like that, but it would be good if one of the most standout features of C# got some proper attention after a decade of neglect. There has been a lot of effort on performance in many areas of the framework and the language. Leveraging that work for LINQ (the language feature) would be very welcome.
- neonsunset 2y ago
  C# is already big enough language. This is more about .NET in general and less about C# specifically. From what I've heard - there is an awareness that the current cost of delegates, while not that problematic anymore, can be improved. Whether this happens via explicit syntax to enable (ref) struct closures and value delegate pattern implementations or via object escape analysis and better devirtualization remains to be seen. p.s.: I really wish that instead of LanguageExt, companies would have adopted F# instead.
  - louthy 2y ago
    > p.s.: I really wish that instead of LanguageExt, companies would have adopted F# instead. Ah the standard "why don't you just use F#" line. I have a standard response to that now ;) https://github.com/louthy/language-ext/wiki/%22Why-don't-you-just-use-F%23%3F%22 https://github.com/louthy/language-ext/wiki/%22Why-don't-you...

progmetaldev 2y ago

Your library looks very interesting, but I think you might be setting things up ahead of time to be dismissed. There are a large number of using statements, which is not a big deal for those who understand breaking up projects into appropriate pieces for exactly what you need, and keeping boundaries between different concerns. Most developers don't structure their projects the way that you do, and I say that as a positive towards your implementation, but often little things like this can hold back "the most common-denominator" when it comes to developers. I will admit right off the bat that you are far more well-versed in C# and .NET than I am, but I'm coming to this as someone that has worked with lots of junior developers that already struggle with standard LINQ syntax, methods, and (in particular) performance. You do mention these things in your README, and I appreciate it, as most are set on "selling" their library instead of actually listing what it exceeds at and the purpose. Your comment on being non-idiomatic is another issue that I see (again, talking about people learning and getting used to C#/.NET). Only in my opinion, I feel like Microsoft want to keep their tools and languages following certain practices, and I get that your naming follows more functional programming practices to make them natural to FP. I feel like often, little issues like this are very large to Microsoft when it comes to looking into improvements. I starred your repo, and am very interested in what you've put together here. On the last few major applications I've built (I normally build content management systems that are highly customized, that I tend to not call apps), I've used a Result<T> type that seems to be roughly the same as Option<T>. I wrote that sentence, and then went back to look at your library, and as much as I thought I knew about FP, I really don't. I think I'm really decent at C# and doing some complex things with it (I mainly use Umbraco CMS, and I've gotten my code to run websites under the minimal recommended requirements of the base Umbraco software). With that said, I read a lot about FP, but still have tons of issues picking it up. F# For Fun And Profit is the closest it's made sense to me. In the end, all this is to say that it's nothing that you're doing wrong. Microsoft is going to target the average or beginner developer, because that is the vast majority of developers that they have in their environment. I hope that you library can gain some internal improvements, because it looks like you've spent an enormous amount of time on this, and you have enough stars on GitHub to at least indicate that people are actively using your library and getting benefit from it. I apologize if any of this came off as dismissive, I think what you've done here is exciting, and feel like you've set up enough documentation where I can slowly pick up concepts that I'm unfamiliar with.