6 ms·

Sorry to be "that guy", but what is System.Linq.Parallel? Can anyone provide a brief description/ list of links where to read more about it? Why is open sourcin

by blubbi2 12y ago

Sorry to be "that guy", but what is System.Linq.Parallel? Can anyone provide a brief description/ list of links where to read more about it? Why is open sourcing it such a big deal?

useerup 12y ago
http://msdn.microsoft.com/en-us/library/dd460688(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/dd460688(v=vs.110).a... Parallel LINQ is an extension to LINQ (Language Integrated Querying). The "parallel" hints that it is a way to process results in parallel - using multiple threads, possibly on multiple CPUs/cores. It allows you to express patterns like fork/join in a very elegant, fluent way. If I have a set of Orders each with a collection of OrderItems, and each OrderItem* has a Consolidate method (I know, contrived, but bear with me). Orders.SelectMany(o => o.OrderItems).AsParallel().ForAll(Consolidate); will "consolidate" all of the order items in parallel (or with the optimal degree of parallelism given CPUs/cores).
ultimape 12y ago
Linq is a library that injects a whole bunch functional style operations on top of list-like objects. Technically speaking, its implementing a type-safe query (think SQL) language on top of them, but the difference is very minimal in practice. What this does is allow for certain operations to automatically be run in parallel. If you wanted to add 1 to every element in an array, you could do that with with Linq: IEnumerable<int> numbers = new List<int>() { 1, 2, 3, 4, 5 }; var newList = from num in numbers.AsParallel() select (x => x+1); but by adding ".asParallel()" to the chain: var newList = from num in numbers.AsParallel() select (x => x+1); your code is automatically run on each element in a parallel fashion and the runtime handles marshalling it all into a queue for processing... or however it does it's magic. The MSDN document on the way it does it is quite clear: http://msdn.microsoft.com/en-us/library/dd997399(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/dd997399(v=vs.110).a... - describing many linq operations as 'delightfully parallel'. I personally love Linq. To me it feels like using a Functional Relational Mapping system when combined with Linq-To-SQL, and the same semantics can be used to create a reactive programming model similar to what is intended with kefir.js and Akka ( http://msdn.microsoft.com/en-us/data/gg577609.aspx http://msdn.microsoft.com/en-us/data/gg577609.aspx ) The idea itself isn't new, Perl 6 has something like this planned for trivial cases like for (1..10) { $_ = $_ +1; } ---- I don't know the full story, but for me the significance of open sourcing this code is that it will potentially encourage forks and extensions that allow running stuff upon GPGPUs or other massively parallel architectures while still using Linq style syntax. TL;DR: With Linq, Anything that implements iEnumerable (lists, arrays, streams, etc) can be made to automatically support all sorts of nifty transforms. The Parallel bits let many of them run in parallel quite trivially
- mattmanser 12y ago
  I hate to be nit-picky, what you're presenting is non-idiomatic C#, I would guess you're not a C# programmer. No-one really uses the SQL-esque langugage. The simple methods reads better, are quicker to type, are clearer and are more succinct. 'p', 'q' also tends to be the de facto standard operator in lambda expressions, rather than 'x', 'y' or 'i', 'j'. And why would you use IEnumerable<int> where you used it, it makes no sense? Your code should have read: var numbers = new List<int>() { 1, 2, 3, 4, 5 }; var newList = numbers.AsParallel().Select(p => p + 1);
  - schrodinger 12y ago
    No one uses the sql esque language? Where do you get that idea from? I've been a c# dev for 7 years and have seen plenty. For complicate queries it's significantly more compact (and readable, in my opinion). And using IEnumerable was probably making the point that it didn't have to be a list, it could be anything implementing that interface.
    - mattmanser 12y ago
      Ach, come on, the briefest trawl through recent SO questions and you'll see the % of DSLs has dropped to a small amount since it launched. Downvoters be damned, they know not what they talk about. ultimape is clearly not a C# developer.
      - chton 12y ago
        I know people with 10 years experience with the language that still use the DSL for simple cases. There is a lot of discussion about the usage of var, and ultimape's non-usage of it in his example is a personal preference, as well as making sense in an educational post. And lastly, I think you'll find many developers, young and old alike, use x, y and z as their lambda parameters. I do too. It comes from the mathematical function origin of the syntax. In most early examples when it was introduced, and loads of educational materials now, lambda syntax is still presented akin to a mathematical formula: x => x+1 In that context, using x makes a lot more sense than p, and has stuck around as a habit. It's a style thing, and everybody has their own style. You don't get to claim superior experience or skill based on a 2 line snippet that doesn't match your style. And as an aside: you're not being downvoted by "the ignorant masses". You're being downvoted for being confrontational and not contributing to a conversation about the topic, which is Parallel LINQ and not coding style.
CmonDev 12y ago
LINQ = Enumeration Monad Parallel = executes tasks on multiple cores simultaneously if possible
Quequau 12y ago
This is actually becoming my chief complaint about submissions. It appears to me as if folks now believe that it's cool to submit links which resolve as deeply and closely as possible to subject (in this case a the subdirectory inside the project's repo) with total disregard to the fact that there is insufficient (or no) information describing why we should be interested in the link or what, if anything, we should discuss. I'm willing to bet the submitter went through a series of links beginning with some sort of announcement that something changed to get to the link they submitted. Why do folks think that we would not also benefit from that information or that it would not help to foster more interesting discussion?
- protomyth 12y ago
  Why? Probably because the "blog spam" or "only submit the original source" comments will show up and submissions get flagged. Happens enough that I would imagine the submitter probably followed all the links in response to prior submissions.
- untog 12y ago
  It would be better if we could change titles, but more often than not mods revert them back to whatever the page title is.

by blubbi2 12y ago

Sorry to be "that guy", but what is System.Linq.Parallel? Can anyone provide a brief description/ list of links where to read more about it? Why is open sourcing it such a big deal?

useerup 12y ago

http://msdn.microsoft.com/en-us/library/dd460688(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/dd460688(v=vs.110).a... Parallel LINQ is an extension to LINQ (Language Integrated Querying). The "parallel" hints that it is a way to process results in parallel - using multiple threads, possibly on multiple CPUs/cores. It allows you to express patterns like fork/join in a very elegant, fluent way. If I have a set of Orders each with a collection of OrderItems, and each OrderItem* has a Consolidate method (I know, contrived, but bear with me). Orders.SelectMany(o => o.OrderItems).AsParallel().ForAll(Consolidate); will "consolidate" all of the order items in parallel (or with the optimal degree of parallelism given CPUs/cores).

ultimape 12y ago

Linq is a library that injects a whole bunch functional style operations on top of list-like objects. Technically speaking, its implementing a type-safe query (think SQL) language on top of them, but the difference is very minimal in practice. What this does is allow for certain operations to automatically be run in parallel. If you wanted to add 1 to every element in an array, you could do that with with Linq: IEnumerable<int> numbers = new List<int>() { 1, 2, 3, 4, 5 }; var newList = from num in numbers.AsParallel() select (x => x+1); but by adding ".asParallel()" to the chain: var newList = from num in numbers.AsParallel() select (x => x+1); your code is automatically run on each element in a parallel fashion and the runtime handles marshalling it all into a queue for processing... or however it does it's magic. The MSDN document on the way it does it is quite clear: http://msdn.microsoft.com/en-us/library/dd997399(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/dd997399(v=vs.110).a... - describing many linq operations as 'delightfully parallel'. I personally love Linq. To me it feels like using a Functional Relational Mapping system when combined with Linq-To-SQL, and the same semantics can be used to create a reactive programming model similar to what is intended with kefir.js and Akka ( http://msdn.microsoft.com/en-us/data/gg577609.aspx http://msdn.microsoft.com/en-us/data/gg577609.aspx ) The idea itself isn't new, Perl 6 has something like this planned for trivial cases like for (1..10) { $_ = $_ +1; } ---- I don't know the full story, but for me the significance of open sourcing this code is that it will potentially encourage forks and extensions that allow running stuff upon GPGPUs or other massively parallel architectures while still using Linq style syntax. TL;DR: With Linq, Anything that implements iEnumerable (lists, arrays, streams, etc) can be made to automatically support all sorts of nifty transforms. The Parallel bits let many of them run in parallel quite trivially

mattmanser 12y ago
I hate to be nit-picky, what you're presenting is non-idiomatic C#, I would guess you're not a C# programmer. No-one really uses the SQL-esque langugage. The simple methods reads better, are quicker to type, are clearer and are more succinct. 'p', 'q' also tends to be the de facto standard operator in lambda expressions, rather than 'x', 'y' or 'i', 'j'. And why would you use IEnumerable<int> where you used it, it makes no sense? Your code should have read: var numbers = new List<int>() { 1, 2, 3, 4, 5 }; var newList = numbers.AsParallel().Select(p => p + 1);
- schrodinger 12y ago
  No one uses the sql esque language? Where do you get that idea from? I've been a c# dev for 7 years and have seen plenty. For complicate queries it's significantly more compact (and readable, in my opinion). And using IEnumerable was probably making the point that it didn't have to be a list, it could be anything implementing that interface.
  - mattmanser 12y ago
    Ach, come on, the briefest trawl through recent SO questions and you'll see the % of DSLs has dropped to a small amount since it launched. Downvoters be damned, they know not what they talk about. ultimape is clearly not a C# developer.
    - chton 12y ago
      I know people with 10 years experience with the language that still use the DSL for simple cases. There is a lot of discussion about the usage of var, and ultimape's non-usage of it in his example is a personal preference, as well as making sense in an educational post. And lastly, I think you'll find many developers, young and old alike, use x, y and z as their lambda parameters. I do too. It comes from the mathematical function origin of the syntax. In most early examples when it was introduced, and loads of educational materials now, lambda syntax is still presented akin to a mathematical formula: x => x+1 In that context, using x makes a lot more sense than p, and has stuck around as a habit. It's a style thing, and everybody has their own style. You don't get to claim superior experience or skill based on a 2 line snippet that doesn't match your style. And as an aside: you're not being downvoted by "the ignorant masses". You're being downvoted for being confrontational and not contributing to a conversation about the topic, which is Parallel LINQ and not coding style.

CmonDev 12y ago

LINQ = Enumeration Monad Parallel = executes tasks on multiple cores simultaneously if possible

Quequau 12y ago

This is actually becoming my chief complaint about submissions. It appears to me as if folks now believe that it's cool to submit links which resolve as deeply and closely as possible to subject (in this case a the subdirectory inside the project's repo) with total disregard to the fact that there is insufficient (or no) information describing why we should be interested in the link or what, if anything, we should discuss. I'm willing to bet the submitter went through a series of links beginning with some sort of announcement that something changed to get to the link they submitted. Why do folks think that we would not also benefit from that information or that it would not help to foster more interesting discussion?

protomyth 12y ago
Why? Probably because the "blog spam" or "only submit the original source" comments will show up and submissions get flagged. Happens enough that I would imagine the submitter probably followed all the links in response to prior submissions.
untog 12y ago
It would be better if we could change titles, but more often than not mods revert them back to whatever the page title is.