6 ms·

its a shame that linq had the overhead it does when used with collections of objects. at work right now i am often having to de-linq things

manigandham 7y ago
Try LinqFaster which uses different techniques for better performance and less allocations: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster
- gameswithgo 7y ago
  i wrote that :)
sebazzz 7y ago
Linq should not be used on hot paths, as it causes GC pressure. If this is what bothers your program, you can find out by profiling. However, if you have some slow SQL query or redundant SQL queries, I bet it is performance wise better to do something about that. In the case of Roslyn, the C# compiler, they avoid Linq on the hot paths (which run many times, for instance when typing in Visual Studio!) but outside hot paths linq is still used.
- oaiey 7y ago
  Which is annoying as hell once you are used to LINQ. I completely forgot how to write a for loop till .NET Core came up with Span<T> and friendly forced us to re-think about memory. I hope they update LINQ in .NET to address these use cases.
fuzzy2 7y ago
I’m interested in what performance problems you’re seeing. Maybe then I won’t fall into the same trap in the future. Off the top of my head, the only possibility I see is allocating objects with `Select` or the like. If one can avoid that, the only overhead should be the predicate/selector method calls...?
- gameswithgo 7y ago
  best way to get familiar with the degree of overhead is benchmark some things with benchmarkdotnet using linq and with for loops. lists and arrays and ienumerables. you will build up an intuition for the costs involved. groupby in particular is really nasty. any time you want to use that just put things into a dictionary instead.
  - fuzzy2 7y ago
    BenchmarkDotNet looks cool, thanks for that! The problem is that I never had problems! (With LINQ-to-Objects performance.) That’s why I’m asking in the first place. Maybe I intuitively avoided all the pitfalls, who knows. It would be cool if you could share some (abstract) operator chains that caused problems in whatever you’re working on.
    - gameswithgo 7y ago
      So I have some benchmarks in this repo showing some comparisons: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster While those can look really bad - orders of magnitude slower! Keep in mind with things like Where/Select, its a fixed amount of overhead. If you are larger amounts of work, or IO in those clauses, then the fixed amount of overhead is small in comparison. There are some common pitfalls you can avoid: Don't use GroupBy - it actually has a lot of overhead. Put things into a dictionary by hand instead. Instead of chaining together multiple Where clauses, just use one Where clause and the && operator. This saves you some overhead and has the same semantics. Consider whether you can presize a collection and then fill it up, rather than calling "ToList()" or "ToDictionary" because those will not presize the collection for you. The list will have to reallocate as it grows etc. Keep in mind that depending on your .NET version, calls to Count() may have to actually iterate over the whole collection even if that isn't necessary. .NET Core 3 fixes some of those cases, older versions may iterate over an IEnumerable backed by a List just to get the count, when the count is already there. At work right now, we just operate at very high scale, so in high traffic endpoints its worth it to pull microseconds of time out, and reduce memory use by bytes. Because everything gets multiplied by large numbers of requests.

manigandham 7y ago

Try LinqFaster which uses different techniques for better performance and less allocations: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster

gameswithgo 7y ago
i wrote that :)

sebazzz 7y ago

Linq should not be used on hot paths, as it causes GC pressure. If this is what bothers your program, you can find out by profiling. However, if you have some slow SQL query or redundant SQL queries, I bet it is performance wise better to do something about that. In the case of Roslyn, the C# compiler, they avoid Linq on the hot paths (which run many times, for instance when typing in Visual Studio!) but outside hot paths linq is still used.

oaiey 7y ago
Which is annoying as hell once you are used to LINQ. I completely forgot how to write a for loop till .NET Core came up with Span<T> and friendly forced us to re-think about memory. I hope they update LINQ in .NET to address these use cases.

fuzzy2 7y ago

I’m interested in what performance problems you’re seeing. Maybe then I won’t fall into the same trap in the future. Off the top of my head, the only possibility I see is allocating objects with `Select` or the like. If one can avoid that, the only overhead should be the predicate/selector method calls...?

gameswithgo 7y ago
best way to get familiar with the degree of overhead is benchmark some things with benchmarkdotnet using linq and with for loops. lists and arrays and ienumerables. you will build up an intuition for the costs involved. groupby in particular is really nasty. any time you want to use that just put things into a dictionary instead.
- fuzzy2 7y ago
  BenchmarkDotNet looks cool, thanks for that! The problem is that I never had problems! (With LINQ-to-Objects performance.) That’s why I’m asking in the first place. Maybe I intuitively avoided all the pitfalls, who knows. It would be cool if you could share some (abstract) operator chains that caused problems in whatever you’re working on.
  - gameswithgo 7y ago
    So I have some benchmarks in this repo showing some comparisons: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster While those can look really bad - orders of magnitude slower! Keep in mind with things like Where/Select, its a fixed amount of overhead. If you are larger amounts of work, or IO in those clauses, then the fixed amount of overhead is small in comparison. There are some common pitfalls you can avoid: Don't use GroupBy - it actually has a lot of overhead. Put things into a dictionary by hand instead. Instead of chaining together multiple Where clauses, just use one Where clause and the && operator. This saves you some overhead and has the same semantics. Consider whether you can presize a collection and then fill it up, rather than calling "ToList()" or "ToDictionary" because those will not presize the collection for you. The list will have to reallocate as it grows etc. Keep in mind that depending on your .NET version, calls to Count() may have to actually iterate over the whole collection even if that isn't necessary. .NET Core 3 fixes some of those cases, older versions may iterate over an IEnumerable backed by a List just to get the count, when the count is already there. At work right now, we just operate at very high scale, so in high traffic endpoints its worth it to pull microseconds of time out, and reduce memory use by bytes. Because everything gets multiplied by large numbers of requests.