11 ms·
C#: IEnumerable, yield return, and lazy evaluation
- keewee7 4y agoStack Overflow is a remarkable company. They bet on C# and ASP.NET MVC 1.0 in 2008 and they still host their own server infrastructure in their basement while being one of the 20 most visited sites in the world.
- DeathArrow 4y agoThey also utilize just a few servers even if they have a massive traffic because they optimize the code. :) A lazy company would use some dynamic language and tons of cloud VMs and pay 100x for the hosting.
- fendy3002 4y agoSome businesses cannot afford to optimize the code because the requirements are kept changing/unclear. Any attempts to optimize will be obsolete in some years. It's worse when the project is outsourced to vendors, since time used for optimization is a wasted cost for vendor, while business is unlikely to pay for optimization unless it's really severe.
- deleted 4y ago[deleted]
- arunprakashb 4y agoThis is my most favorite feature in C#. It is the right combination of elegance and utility.
- Eji1700 4y agoI'll just say if you enjoy using this stuff in C# (and linq) you really owe it to yourself to at least look at F#. I'm biased as hell because I think it's a wonderful language, but even if you decide it's not for you, getting the basics down helped me better understand how to use such tools across languages.
- dlkmp 4y agoIf you had to start over, would you still recommend to yourself to learn F# in 2022? I am asking this as someone who has dabbled in functional languages (F#/ Haskell) multiple times over the years but, despite liking the elegance of functional code, never reached a level of productivity that came close to what I am used to from imperative languages. Specifically for C#, given the trend of baking more and more of the cool functional stuff into the language (e.g. [1]), I am wondering whether F# still has enough unique features to justify the effort. [1] see e.g. https://devblogs.microsoft.com/dotnet/early-peek-at-csharp-11-features/#c-11-preview-list-patterns https://devblogs.microsoft.com/dotnet/early-peek-at-csharp-1...
- shirogane86x 4y agoAs a former F# full-time dev, I'd say that there's a few things that really make it worth to use F# (and I will still use it instead of C#, where there's a possibility, in the future): - Computation expressions make it possible to do ergonomically use value based error and nil handling, with option and result (very awkward to do in C#). - type providers are a way to generate types at compile time based on some input. They let you do amazing stuff, like compile-time checking SQL against a specific database and automatically generating the correct types for inputs and outputs, or automatically generating types for stuff like XML or JSON files, based on a sample. - really good type inference and (in my opinion) a better type system than C#'s, mostly wrt generics. You reap all the benefits of static typing, while also typing very very few types, and you get to express some things that just aren't unfeasible to express safely in C#. For example, the compiler will infer things like: let add a b = a + b * b //has type ^a -> ^c -> ^d when (^a or ^b) : (static member ( + ) : ^a * ^b -> ^d) and (^a or ^c) : (static member ( * ) : ^a * ^c -> ^b) //which means: a function from 'a' and 'c' to 'd', where either 'a' or 'b' has an add operator, and either 'a' or 'c' has a multiplication operator you could also explicitly type it out as a type signature, by the way. there's probably more, but these are the things that came to my head. DISCLAIMER: I usually reach for F# anyway, as I feel a lot more comfortable programming functionally compared to imperatively. F# is the only real option for actual FP on the CLR - so I'm a bit biased (and for much of the same reason, I would rather use Scala compared to Java on the JVM, for example). My actual favourite programming language to use is Haskell, so take that as you will :)
- travisgriggs 4y agoI did not know C# had this. That’s cool. I admit I see only a little of the C# that others at our company do. But I have used the similar feature in Python for years. And I love it. Is it just me, or are a lot of these languages starting to all feel a lot like each other?
- StellarScience 4y agoThis is indeed an amazing feature, and good languages copy ideas from each other. I believe C# has had this for years. C++ just got it with C++20 coroutines, though it still takes some work to use since std::generator didn't make it into C++20. Lest folks think this is a new feature, it was invented by Barbara Liskov in her CLU language, as explained in this 1977 paper: https://web.eecs.umich.edu/~weimerw/2011-6610/reading/liskov-clu-abstraction.pdf https://web.eecs.umich.edu/~weimerw/2011-6610/reading/liskov... (PDF, see section 4.2.)
- m1el 4y agoCross-pollination of ideas is a good thing. The only problem, what if your language adopts an idea which prevents you from having a better API? As far as I know, coroutine/generator doesn't prevent you from doing anything.
- tokamak-teapot 4y agoI enjoy learning enough of novel (to me) languages so that I can do some fun stuff. Usually when I look at a language One of the first things I do is look for familiarity: How to map, filter and reduce - and how to work with collections and lazy enumerables, whatever they may be called - IEnumerable, generator, Stream, etc. It’s amazing how similar many popular languages are these days, at least in terms of developer experience. This doesn’t mean they are trivially interchangeable of course. You can’t write Elixir in an idiomatic C# style and you shouldn’t try to!
- int_19h 4y agoIn case of C# and Python specifically, there's been quite a bit of cross-pollination. And now designers of both languages work at the same company...
- StellarScience 4y agoThis incredibly powerful facility is also available in C++20 with coroutines. 'yield return' is written as 'co_yield'. Also since "iterator" means something else in C++, this special usage of coroutines is called a "generator". 'co_yield' makes iterators/generators super easy to write as shown in this article. But also it enables great abstraction/encapsulation for where you want to provide an interface allowing callers to iterate over elements, without exposing the underlying container type. (Note that 'std::generator' is not yet officially standardized, but with a little work we're using Microsoft's 'std::experimental::generator' across platforms, compiling with clang and gcc. Alternatively, I hear the cppcoro library is pretty great.)
- mgraczyk 4y agoFacebook already uses coroutines pretty heavily in certain C++ services. Was kind of surprising to see, I haven't seen coroutines used much in C++ because of the lack of standard library support (no composeable promises yet)
- insaider 4y agoVery interesting, would love a follow up deep dive into the IAsyncEnumerable and await foreach() features!
- LeonB 4y agoAs I was reading it I thought … okay next it’ll go into IAsyncEnumerable and could feel my self start to sweat. Then it was over!
- MauroIksem 4y agoYield return is magical.
- delta_p_delta_x 4y agoIEnumerable is to C#/.NET, what std::begin() and std::end() are to C++, and LINQ is to C#/.NET what <algorithm> is to C++. Both interfaces and libraries respectively are the bread and butter of both languages. I love both languages, and sincerely hope that C# is able to compile to native at some point in the future; there are already some efforts to compile it to WebAssembly with LLVM [0]. [0]: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-LLVM https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-...
- DeathArrow 4y agoAOT is coming with NET 7 which will be released in November. https://visualstudiomagazine.com/articles/2022/04/15/net-7-preview-3.aspx https://visualstudiomagazine.com/articles/2022/04/15/net-7-p...
- pooper 4y agoWhen I read the advantages, all but restricted platforms sound like things that would help with server less (aws lambda, azure functions, etc) where we are doing seemingly stupid things like pinging a service once a while to keep it "warm". However, they don't mention this at all. Is asp.net not just another console app?
- DeathArrow 4y agoASP.NET are console apps, yes.
- neonsunset 4y agoSpeaking of serverless, this might be of interest to you: https://github.com/awslabs/dotnet-nativeaot-labs https://github.com/awslabs/dotnet-nativeaot-labs
- GordonS 4y agoMicrosoft also said it was coming in NET 6, NET 5, Dotnet Core 3... Hopefully they actually mean it this time!
- SideburnsOfDoom 4y agoThey use "ToArray()" in several places, "ToList()" is perhaps more common. This operation is technically called "to reify" the enumeration", or "Reification" From the dictionary: > reify: make (something abstract) more concrete or real. https://dictionary.cambridge.org/dictionary/english/reify https://dictionary.cambridge.org/dictionary/english/reify
- phito 4y agoToList shouldn't be used as much as it is, most of the time ToArray is good enough, and it doesn't allocate a List on the heap
- Limeray 4y ago
- spErased 4y agoJust to add to this, they're essentially the same with ToList beating out ToArray at certain sizes due to how the internal array of a list is resized vs a raw array (true for .net core). A better rule of thumb is deciding if the result needs to added to/modified etc then picking accordingly.
- WorldMaker 4y agoI've written several rants about "ToList considered harmful". I think it is unfortunately too easy to (ab)use and often ToArray is better overall for the use cases of a reifying a query only to simply iterate it later (more than once; otherwise leave things lazy). That rule of thumb "are there calls to Add after this query" is the biggest one where "you might need List<T>" (and even then maybe you are taking a too imperative approach to something that could be reified differently). Though often I find what projects really need are ToDictionary or ToLookup, especially ToLookup, a lot of C# developers sleep on that, because it doesn't have an Add and doesn't implement ICollection<T> and you have to search NuGet for a mutable ILookup<T> implementation that you can just `new` yourself, but ILookup<T> is often the best shape of reified results that have multiple iterations of subsets. Take your O(n^2) operations and make them more O(n log n) or whatever if you pick the right key(s) for your subsets. It's not uncommon for me to start a performance optimization pass on a C# app simply by doing a Find All for ToList and removing every single use.
- steve_taylor 4y agoIn the early 2010s while learning C# and porting a Delphi app to Windows Forms, I learnt about this and many other C# features and patterns thanks to a ReSharper free trial which I used heavily for all of those 30 days. If it's as good now as it was over 10 years ago, I highly recommend it for every C# developer who uses Visual Studio.
- keewee7 4y agoWhenever new C# language features are announced people outside the .Net world criticize the language for becoming too bloated. They forget that 99% of C# developers use full fledged IDEs that gradually teach you new language features by suggesting useful refactorings. The only important C# specific language features that takes effort to understand is LINQ that came out in 2007 and async/await that came out in 2012. Any C# programmer who has been in coma since 2012 can get up and running with the more recent language features in just a few hours.
- danjc 4y agoSurprised to see a 2022 post about this without mentioning IAsyncEnumerable which is the companion to IEnumerable for async and allows async lazy eval of *each iteration*. We’re using it extensively and coupled with the syntactic sugar that enables its consumption in foreach, it makes the code look as elegant as a traditional synchronous enumeration.
- DeathArrow 4y agoYou should read the article, not just the title: "And to muddy the waters just a little, not all iterators are synchronous; there’s also an IAsyncEnumerable interface (you can loop through it with await foreach)."
- bouke 4y agoEven though it is mentioned in the article, it deserves more than just mention. I find the Async LINQ extensions somewhat lacking. I need to have IAsyncEnumerable all the way down: returning an array in `SelectMany` is not possible. Sometimes you mix and match with synchronous code and the `ToAsyncEnumerable()` isn't elegant.
- WorldMaker 4y agoHave you tried importing the System.Interactive.Async package? [0] It's sometimes referred to as "Ix" (Interactive Extensions) to relate it to its older brother System.Reactive (ReactiveX or Rx). It adds a ton of LINQ-style extensions that make flowing between async and sync code easier. Despite the "System" name you always have to install it from NuGet and it's not just available out of the box, but it lights up so many LINQ extensions when installed and paired with a `using System.Linq.Async;`. Though I also think IAsyncEnumerable is even better when paired with Rx, too, as some things are easier to express as Observable "in the middle" as an AsyncEnumerable->Observable->AsyncEnumerable "sandwich". (You can sort of think of it as an pullable Event Source being spread into a push-based message bus/pipeline and then being collected back into a pullable Result feed.) [0] https://www.nuget.org/packages/System.Interactive.Async/ https://www.nuget.org/packages/System.Interactive.Async/
- steve_taylor 4y agoIt's worth mentioning that JavaScript has had this feature for a while now, most likely inspired by C#. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- agumonkey 4y agoI don't know the timeline but it's interesting how c#, js, python, php and maybe others got generators around the same period.
- dragonwriter 4y ago> I don't know the timeline but it's interesting how c#, js, python, php and maybe others got generators around the same period. AFAICT, not really all that close in time: Python 2.2 (2001) .NET 1.1 (2003) PHP (2012-ish, based on RFC date) JS ES2015 (2015) [though nonstandardized impls in some engines prior]
- steve_taylor 4y agoPretty sure IEnumerable was part of the .NET 2.0 release in 2005. https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-version-history#c-version-20 https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csh...
- bzxcvbn 4y agoI don't see how you get to that conclusion. These release notes mention enumerators: > C# version 2.0 brought iterators. To put it succinctly, iterators let you examine all the items in a List (or other Enumerable types) with a foreach loop. Having iterators as a first-class part of the language dramatically enhanced readability of the language and people's ability to reason about the code. The implicit assumption in that sentence is that enumerables existed before iterators.
- SideburnsOfDoom 4y ago
- eknkc 4y agoUnity elegantly abuses these to provide a coroutine support. Basically you can “schedule” an enumerator to be run on the game loop. Engine iterates one by one on the iterator each frame. Can use for animations, delays etc. It looked weird first time but makes sense.
- DeWilde 4y agoWhat a coincidence, I am just going through the C# in Depth chapter about iterators and this pops up on the first page of HN :D.
- code_runner 4y agoThat is such a great book. Read it a while back… I imagine the modern version is every bit as good. I don’t remember it ever getting boring, but if it does, push through! It’s worth it. Jon Skeet also has a (now old) blog series called edu-linq that basically implements linq just using generator blocks and it really makes you think about stuff like validation of arguments vs actually doing the operation. It’s very interesting. A couple of caveats in there.
- rednab 4y agoUnity (the game engine, not the desktop environment) uses this concept heavily, although they use the slightly older IEnumerator instead of IEnumerable ¹). It usually takes people a while to wrap their heads around it and (also hinted at in the article) forgetting to add "yield return" in front of an IEnumerator is a common mistake, but it is an awesome tool to flatten out callback hell or complicated every-frame checks. ¹) https://docs.unity3d.com/Manual/Coroutines.html https://docs.unity3d.com/Manual/Coroutines.html
- DeathArrow 4y agoThey simulate concurrency with coroutines because due to design all code that uses the engine functions should run in main thread, so you can't use Tasks or Threads.
- rednab 4y agoThe threading thing is a common requirement for game engines, though. Unreal also assumes engine functions are called only on the game thread (unless they're specifically marked otherwise).
- jayd16 4y agoYou can use async Tasks as long as you keep them on the main thread. They end up making a lot of garbage though.
- DeathArrow 4y agoLazy evaluation is nice, but as a beginner, if you don't know if a particular IEnumerable or IQueryable is lazy evaluated or not, you are going to get burned.
- alkonaut 4y agoOne must assume you don't know whether it is (so it could be). If you want to realize it into a list, you can do that - but if you don't know anything about the enumerable, you might get burned anyway. If I make an IEnumerable<long> which generates all even numbers, then anyone doing ToList() on it is in for a surprise.
- beltsazar 4y agoI found it surprising that when critizing Go, no one hardly talked about Go's lack of a common iterator interface like Java's Iterator (let alone a more powerful generator like C#'s IEnumerable). This was partly due to the lack of generics back then. Now that we have generics, hopefully we can have it in Go soon.
- steve_taylor 4y agoOracle Database beat C# to language-level iterator support by four years with pipelined functions as part of the 9i release in 2001. Its syntax uses "pipe row" instead of "yield return". In fact, you can connect a .NET app to an Oracle Database, invoke a PL/SQL pipelined function and, while that function is emitting results via pipe row inside a PL/SQL for loop, your C# wrapper streams the results to its called via yield return inside a C# for loop.
- piersj225 4y agoFor for anyone wanted to try C# and a yield method, https://try.dot.net/ https://try.dot.net/ I've had trouble running this in Firefox recently, but Chromium seems fine
- AtNightWeCode 4y agoVery powerful but also very error prone. I am almost thinking that it could use a strict/pure mode with more restrictions. Or maybe just compile warnings for some things.
- sdfhdhjdw3 4y ago> Every time we iterate over numbers, it will start over at the beginning of the iterator method and yield all the same values over again. (You can’t expect this from every iterator; more on that later.) They never did, or did I miss it?
- hddqsb 4y agoThey did: > An iterator could query a database, for example—including the unfortunate possibility that it might alter data, or that iterating through it twice might yield completely different results! Some tools (like ReSharper) will warn you against multiple enumeration for this reason.
- sdfhdhjdw3 4y agoOh I saw that. I was expecting that they were gonna discuss stateful iterators more generally. Database iterators are just a tiny subset of stateful iterators. I think discussing the latter more generally would have been more useful than a one-liner about databases.
- infogulch 4y agoIEnumerable is fun. A while ago I wrote a lazy streaming no-accumulation Batch extension method using a bunch of features like yield return and manual enumeration that happens to be one of my more upvoted answers. public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> source, int size) https://stackoverflow.com/a/44505349/213246 https://stackoverflow.com/a/44505349/213246
- mcaravey 4y agoOne of my favorite ways to show off enumerables to someone is with an endless list. It really makes someone think when I show an IEnumerable instance that contains every prime number, for example [0]. Obviously you can never evaluate the whole thing, but because of the lazy evaluation you can get enumerators and pull out values all day long. Just don’t .ToList() the darn thing. [0] https://gist.github.com/arlm/6647474#file-sieveoferatosthenes-cs https://gist.github.com/arlm/6647474#file-sieveoferatosthene...
- aliswe 4y agoBut EFCore doesnt support async enumerables does it?
- WorldMaker 4y agoIt's not very well documented, but EF Core has supported IAsyncEnumerable<T> since EF Core 3.0 or thereabouts. Just about any of its IQueryable<T> LINQ implementations are castable to IAsyncEnumerable<T>, easiest through .AsAsyncEnumerable() and generally just work when you do that. Most people just use the .ToListAsync() and .ToArrayAsync() "materialization" tools which I believe use IAsyncEnumerable<T> under the hood, but you can use IAsyncEnumerable<T> directly if you want. Generally though, EF Core's support for IAsyncEnumerable<T> is mostly known for the confusion it causes with "Ix" (Interactive Extensions; the System.Interactive.Async nuget package that adds a bunch of LINQ extensions for IAsyncEnumerable<T> to the `using System.Linq.Async;` namespace) because if you install that package and have such using statements there's overload conflicts between IQueryable<T> and IAsyncEnumerable<T> for many LINQ operators and you have to explicitly add .AsQueryable() to almost all of your EF Core LINQ to make sure to avoid those operator conflicts and force a cast to just IQueryable<T>, not both IQueryable<T> and IAsyncEnumerable<T>.