17 ms·
LInQer – C# Integrated Queries ported to Javascript
- skrowl 7y agoMore info here - https://siderite.dev/blog/linq-in-javascript-linqer/ https://siderite.dev/blog/linq-in-javascript-linqer/ Pretty neat project. LINQ is a great time saver in .NET development, I'm looking forward to messing around with it in javascript.
- gameswithgo 7y agoits a shame that linq had the overhead it does when used with collections of objects. at work right now i am often having to de-linq things
- manigandham 7y agoTry LinqFaster which uses different techniques for better performance and less allocations: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster
- gameswithgo 7y agoi wrote that :)
- sebazzz 7y agoLinq should not be used on hot paths, as it causes GC pressure. If this is what bothers your program, you can find out by profiling. However, if you have some slow SQL query or redundant SQL queries, I bet it is performance wise better to do something about that. In the case of Roslyn, the C# compiler, they avoid Linq on the hot paths (which run many times, for instance when typing in Visual Studio!) but outside hot paths linq is still used.
- oaiey 7y agoWhich is annoying as hell once you are used to LINQ. I completely forgot how to write a for loop till .NET Core came up with Span<T> and friendly forced us to re-think about memory. I hope they update LINQ in .NET to address these use cases.
- fuzzy2 7y agoI’m interested in what performance problems you’re seeing. Maybe then I won’t fall into the same trap in the future. Off the top of my head, the only possibility I see is allocating objects with `Select` or the like. If one can avoid that, the only overhead should be the predicate/selector method calls...?
- gameswithgo 7y agobest way to get familiar with the degree of overhead is benchmark some things with benchmarkdotnet using linq and with for loops. lists and arrays and ienumerables. you will build up an intuition for the costs involved. groupby in particular is really nasty. any time you want to use that just put things into a dictionary instead.
- fuzzy2 7y agoBenchmarkDotNet looks cool, thanks for that! The problem is that I never had problems! (With LINQ-to-Objects performance.) That’s why I’m asking in the first place. Maybe I intuitively avoided all the pitfalls, who knows. It would be cool if you could share some (abstract) operator chains that caused problems in whatever you’re working on.
- gameswithgo 7y agoSo I have some benchmarks in this repo showing some comparisons: https://github.com/jackmott/LinqFaster https://github.com/jackmott/LinqFaster While those can look really bad - orders of magnitude slower! Keep in mind with things like Where/Select, its a fixed amount of overhead. If you are larger amounts of work, or IO in those clauses, then the fixed amount of overhead is small in comparison. There are some common pitfalls you can avoid: Don't use GroupBy - it actually has a lot of overhead. Put things into a dictionary by hand instead. Instead of chaining together multiple Where clauses, just use one Where clause and the && operator. This saves you some overhead and has the same semantics. Consider whether you can presize a collection and then fill it up, rather than calling "ToList()" or "ToDictionary" because those will not presize the collection for you. The list will have to reallocate as it grows etc. Keep in mind that depending on your .NET version, calls to Count() may have to actually iterate over the whole collection even if that isn't necessary. .NET Core 3 fixes some of those cases, older versions may iterate over an IEnumerable backed by a List just to get the count, when the count is already there. At work right now, we just operate at very high scale, so in high traffic endpoints its worth it to pull microseconds of time out, and reduce memory use by bytes. Because everything gets multiplied by large numbers of requests.
- hermitdev 7y agoI got dinged pretty heavily for using Linq in an interview test at a hedge fund several years back. I was a little miffed because they jad issues with the memory allocations and speed. Thing is, this was an ahead of time test a3md there was no explanations about expectations beyond "do X" and "do Y", etc. Being a nonpaid test, I did what was expedient to me that met the requirements. Most of the questions were 1-liners using Linq, but that wasn't acceptable. I probably didn't make any friends when I pointed out that I solved all of their problems and that there were no other conditions on the solution. Fuck 'em. Interviewing with hedgefunds is a pain in the ass. Bunch of premadonnas that think their shit doesnt stink (but is probably the rankest WTF code you'll ever see) - and I say this having spent a decade at one. A separate fund gave me a private hackerrank challenge. My code passed all of the public requirements and examples, but failed a private test case. No info given about the failure. Emailed the recruiter about my frustration with it, said fuck it, I'm not interested. Figured if theyre like that in an interview, theyre probably colossal pricks to actually work with. No surprise, I didnt get an on site interview.
- knocte 7y agoWhy not just use C# instead of staying in the shitty alternative? You know... https://twitter.com/knocte/status/1207544365607985152 https://twitter.com/knocte/status/1207544365607985152
- mythz 7y agoBoth these expressions return the same result in both C# and JavaScript: 0.1 + 0.1 == 0.2; //= true 0.1 + 0.2 == 0.3; //= false It's not the language, it's https://floating-point-gui.de https://floating-point-gui.de
- knocte 7y agoI was going to reply that I thought the point of the video/tweet is that JS does choose floating point by default while others don't. But I stand corrected, C# default sucks too: it uses the double type! And here I thought that by using F# I would even be protected from this shit but unfortunately it has the same problem... Performance should be opt-in! (favouring correctness over it) :facepalm: Thanks
- wruza 7y agoFirst you should state the correctness level you need. 0.2? 1/7? sqrt(2)+pi? int n=0, n<inf, sum(1/n^3)?
- sasmithjr 7y agoLook at the decimal type; it's less performant, but you'll avoid floating point issues. https://docs.microsoft.com/en-us/dotnet/api/system.decimal?view=netcore-3.1 https://docs.microsoft.com/en-us/dotnet/api/system.decimal?v...
- ygra 7y agodecimal is still floating-point, just decimal floating-point instead of binary. Things like 1/3 can still not be represented exactly. It's just that for values given in base 10 there's no transformation of the representation and 0.3 remains exactly 0.3.
- pmoleri 7y agoIt looks pretty good. However, I find it similar to what you can do with wu.js or immutable.js@4, although of course LInQer would be more appealing to someone coming from C#. I wish you good luck with this project.
- mythz 7y agoAs LINQ has been positioned as one of C#/.NET's biggest strengths their 101 LINQ Examples are a nice set of simple code examples to compare how well different languages fare against each other: - Swift https://github.com/mythz/swift-linq-examples https://github.com/mythz/swift-linq-examples - Kotlin https://github.com/mythz/kotlin-linq-examples https://github.com/mythz/kotlin-linq-examples - Java https://github.com/mythz/java-linq-examples https://github.com/mythz/java-linq-examples (Java 1.7) - Clojure https://github.com/mythz/clojure-linq-examples https://github.com/mythz/clojure-linq-examples - Dart https://github.com/mythz/dart-linq-examples https://github.com/mythz/dart-linq-examples - #Script LISP https://sharpscript.net/linq/restriction-operators?lang=lisp https://sharpscript.net/linq/restriction-operators?lang=lisp (Interactive, .NET Lisp) - #Script Code https://sharpscript.net/linq/restriction-operators?lang=code https://sharpscript.net/linq/restriction-operators?lang=code (Interactive, .NET JS-Like) - Elixir https://github.com/omnibs/elixir-linq-examples https://github.com/omnibs/elixir-linq-examples - Python https://github.com/rogerwcpt/python-linq-samples https://github.com/rogerwcpt/python-linq-samples - Groovy https://gitlab.com/svkj/groovy-linq-samples https://gitlab.com/svkj/groovy-linq-samples Most languages fare well in both verbosity and readability so I don't view LINQ as a major strength of C# anymore, it's just a well designed, typed query language with the USP of being able to capture and traverse an expression's AST which different LINQ providers can take advantage of by translating the intent of the query into a different DSL, most commonly used by ORMs to convert to SQL and execute the typed C# Expression logic on the RDBMS Server.
- bob1029 7y agoIf you have the patience to dive really deep into the Expression Tree pool, you can build some incredibly powerful tools. See: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/expression-trees/ https://docs.microsoft.com/en-us/dotnet/csharp/programming-g... http://twistedoakstudios.com/blog/Post2540_optimizing-just-in-time-with-expression-trees http://twistedoakstudios.com/blog/Post2540_optimizing-just-i...
- mythz 7y agoI'm the author of 2 such LINQ providers that rewrite expression trees for execution in different remote persistent providers: - OrmLite https://github.com/ServiceStack/ServiceStack.OrmLite https://github.com/ServiceStack/ServiceStack.OrmLite (C# POCO ORM) - PocoDynamo https://github.com/ServiceStack/PocoDynamo https://github.com/ServiceStack/PocoDynamo (C# DynamoDB Client)
- duxup 7y agoIs there a good high level explanation for what this is for someone not already familiar with LInQer or C#? I'm googling around and still kinda not sure.
- np_tedious 7y agoIt's a bunch of functions that can be performed on an enumerable (array, set, , etc) type. Commonly several are changed together. Select (aka map) Where (aka filter) SelectMany (aka Flatmap) Accumulate (aka fold) First (with optional predicate function) Max Min GroupBy Count And so on. The readme lists them all
- bmurphy1976 7y agoIt's more than that though. It's also a special monad like syntax that takes advantage of those methods as well as transformation that takes the declarative form of those methods and converts them into something else ala SQL.
- james_s_tayler 7y agoIIRC only SelectMany meets the criteria of a monad. The powerful part about LINQ is it builds an AST and there are many different LINQ providers which can translate that AST into some instructions to interact with other systems. The typical one is converting it to SQL commands but you could plug anything into it.
- np_tedious 7y agoDoes this project do any of that?
- Rapzid 7y agoDoes it AST? Time for a TypeORM 2.0; Entity TypeWork?
- jeswin 7y agoA big plus over previous attempts seems to be that this is written in TypeScript. Previous HN discussion on LazyLinq, which is another JS implementation - https://news.ycombinator.com/item?id=15913571 https://news.ycombinator.com/item?id=15913571
- scarface74 7y agoPublic Service Announcement: LINQ is not just syntactic sugar for processing collections. The beauty and power of Linq is that it is translated to expression trees at runtime, can be interpreted by a provider, transformed to basically anything - sql, MongoQuery, intermediate language, etc. - and you can pass around types of: Expression<Func<T,bool>> to different providers. Like repo.Find<Customer>(c => c.Age > 50 && c.Sex == “M”);
- jeswin 7y ago> The beauty and power of Linq is that it is translated to expression trees at runtime It's compiled into Expression Trees at compile-time (if it's an IQueryable). If it's an IEnumerable, Expression Trees are never created in the first place (you get a function).
- scarface74 7y agoThere is a reason I wrapped everything in Expression<> to avoid the mistake I made way too often when sending a Func<T,bool> to an EF context....
- jeswin 7y ago> There is a reason I wrapped everything in Expression<> to avoid the mistake I made way too often when sending a Func<T,bool> to an EF context. Sorry, I did not follow.
- scarface74 7y agoNothing will stop you from passing in a Func<T,bool> type where you should be using an expression, but if you pass it to a Where on an EF context it will be run on the client instead of being converted to sql.
- jeswin 7y agoI was saying that Expression Trees are created at "compile-time", and not at runtime. Here's the CIL that gets generated in the case of IQueryable - https://stackoverflow.com/questions/3894598/how-is-linq-compiled-into-the-cil https://stackoverflow.com/questions/3894598/how-is-linq-comp...
- thrower123 7y agoI enjoy LINQ, but the real power comes from when the implementation is more complex than just renaming your standard functional programming map, fold, filter, etc functions to more SQL-y aliases. With that power comes the responsibility to actually understand what it is that you are doing, and where the abstraction leaks. If you are not careful, and do not understand the ramifications of what you are writing, it is very possible to write some horrifically inefficient code, or code that looks correct, but will break at runtime. I have seen many a horror with Entity Framework or Linq-to-SQL when I dug in to fix badly performing code, and hooked up tracing to track the SQL code that is actually generated and executed. Multiple iteration of an IEnumerable is very easy to run into, although static analysis is getting good at flagging that with a warning. Trying to use an IEnumerable that hasn't been materialized from the expression tree into the actual data after the database connection or other resource has been disposed is another common gotcha. Understanding what operations can be translated by the expression tree engine is very important. In LINQ database ORMs, the way that foreign keys behave can vary wildly depending on how the query is written and on how the ORM happens to handle those accesses.
- nizmow 7y agoHonestly this just sounds more like a complaint about ORMs than LINQ in general, though I do agree with it.
- kntmnf 7y agoYou can't port linq to javascript because it needs that Expression stuff. The code quality of this library is fucking terrible. TypeScript sucks, god knows why you or anyone would use it. The author of this library is not even implementing it in a high-performance way (you have to gather up all the calls and then use a reducer to process all the calls at once in one single reducer once a ToList or whatever is called) despite it saying high performance. Who the hell uses generators in javascript. c# is a terrible language. Just learn reducers, filters, maps, etc. You probably came from c# and decided not to learn those things. It goes with choosing TypeScript. Downvotes from the moron brigade of social programmers who spend more type pressing up and downvote buttons then programming.
- efdee 7y agoYou created a new account just to hate on a programming language? Who did this to you?
- kjlkjljklkjl 7y ago"ts sucks"? cuz?
- deleted 7y ago[deleted]
- mb7733 7y agoAccording to him C# sucks too, so something tells me he's not the primary audience of a port from C# to TS
- treve 7y agoNewb question: Can this translate to a MySQL query?
- remontoire 7y agoI would be remiss if I didn't mention something from clojurescript land that accomplishes something similar: https://github.com/tonsky/datascript https://github.com/tonsky/datascript Datascript allows you to write datalog queries in ClojureScript(and JavaScript) that operate on a b+ tree maintained by the library. It includes first class relationships and unique keys.
- hebrox 7y agoIf you like this syntax and work with TypeScript and Knex, you can checkout my typed-knex library: https://github.com/wwwouter/typed-knex https://github.com/wwwouter/typed-knex All feedback is welcome :)
- jeswin 7y agoBecause there is no built in support to create expression trees from query expressions, your library is forced to re-implement common query patterns with a custom API. The result is not as clean as C#. And there is some loss of type-safety (The 'SUM' in the groupBy() example). An alternative approach could be to parse real JS expressions using babel (at runtime) - essentially giving you the same flexibility as C#. So you could allow users to write: typedKnex.query(User).where(u => u.id > 100 && u.city === "Calicut")
- recursive 7y ago> Because there is no built in support to create expression trees from query expressions, your library is forced to re-implement common query patterns with a custom API. Well, it could convert the "delegates" back to strings, and re-parse an AST from that. Not that I'm volunteering to write it.
- jeswin 7y agoThat works. An alternative could be to do it during build with a babel based tool. Earlier, I wrote a tool called chimpanzee[1] to parse such ASTs into SQL. Not easy (but not too hard either); there are so many different ways to express something in JS. Here's an example (using chimpanzee) which checks if an expression is doing a sort (on an array/table) and to extract sort fields and order. https://github.com/jeswin-unmaintained/isotropy-ast-analyzer-db/blob/master/src/schemas/sort.js https://github.com/jeswin-unmaintained/isotropy-ast-analyzer... [1] chimpanzee: https://github.com/jeswin/chimpanzee https://github.com/jeswin/chimpanzee
- greatjack613 7y agoReally cool, but the description of the project states, "ported to javascript for performance" In what alternative universe are things ported to javascript for performance?
- czechdeveloper 7y agoI think word "performance" is not proper here. Maybe productivity.
- mhh__ 7y agoI haven't looked into it yet but I see quite a few projects thinking they can outsmart the big boys by tuning for either extremely specific use cases or very small benchmarks. The bigger argument against JavaScript in my view is the way the language is so antithetical to every lesson you'd have thought the industry has learned - even Go is safer than JS and Go is designed like it's from the 70s
- HereBeBeasties 7y agoYou have the wrong end of the stick. The author means that he's ported it to achieve better performance than using Javascript's built-in map/filter./etc. operators because those do not lazy-evaluate and do not chain together efficiently, creating an array at each invocation.
- yread 7y agoYou can do a lot with pure js map, filter and apply: https://medium.com/@aikeru/a-c-linq-to-javascript-translation-guide-6e1558fc1905 https://medium.com/@aikeru/a-c-linq-to-javascript-translatio...
- adgasf 7y agoThe pipeline operator and some free functions is a superior alternative. const map = f => xs => ({ [Symbol.iterator]: function * () { for (const x of xs) { yield f(x); } } }); xs |> map(x => x * 2);
- achou 7y agoSee also IxJS, which is a pull-based version of RxJS: https://github.com/ReactiveX/IxJS https://github.com/ReactiveX/IxJS
- aabbcc1241 7y agoWhen would you really need this complex library over the native methods supported by Array? I know there are overhead to convert map and set into array but it should be fine for non huge data set.
- oaiey 7y agoInteresting question. I share this train of thought. When I restarted writing JavaScript (thank god with TypeScript ;)) and I realized filter/map/reduce array methods, my fear of lacking LINQ vanished away. Reduce is painful in the beginning (because it is different to SQL's GroupBy) but it is a foundation for so much more (once you REALLY understand it). Some more helpers are useful (first, last, max, take, skip ... all with variation). They should added to the core language.
- jolux 7y agoLinq has Reduce as well, known as Aggregate (foldr/foldl in most functional languages).
- jermaustin1 7y agoI wrote as a fun little learning experiment linq.js[1], which is 90% renamed wrappers for the array prototype methods, then some logic thrown in for things like any, all, first/single[OrDefault], and orderBy. It is just under 90 lines of code. 1: https://github.com/jeremyaboyd/linq.js https://github.com/jeremyaboyd/linq.js
- brazilianabroad 7y agoYou need to study more. You're too ignorant to be judging whole ecosystems as "shit". http://0.30000000000000004.com/ http://0.30000000000000004.com/
- phillc73 7y agoThought it also worth mentioning the Julia package, Query.jl, which supports LINQ-like syntax: https://github.com/queryverse/Query.jl/blob/master/README.md https://github.com/queryverse/Query.jl/blob/master/README.md http://www.queryverse.org/Query.jl/stable/linqquerycommands/ http://www.queryverse.org/Query.jl/stable/linqquerycommands/
- macca321 7y agoThese libraries miss the point of linq. Yes it provides a nifty syntax for munging lists, but Real C# LINQ provides that munge as a abstract syntax tree which can be interpreted before executing it. This means it can rewrite the native language query, and say, turn it into a SQL or graphql query, or a stream filter, or check against cached data or all kinds of clever things.
- flamtap 7y agoIt does seem to miss the "Query" part of "Language INtegrated Query". Though the methods are convenient in any case. One of the most mind-blowing moments I can remember in my career is when I realized just how powerful the tandem of LINQ and IQueryable was. Now, writing your own LINQ provider is no walk in the park, but is a really satisfying project and worth doing once, if only to get a deeper understanding of what's going on with expression trees and the inner workings LINQ. It's almost like a language within a language.
- brazilianabroad 7y agoYou need to study more. You lack sufficient knowledge to be judging whole ecosystems as "shit". http://0.30000000000000004.com/ http://0.30000000000000004.com/
- brazilianabroad 7y agoI don't understand why my comment keeps getting flagged. "You need to study more" is not meant as an insult, simply as advice. 'You're too ignorant to be judging whole ecosystems as "shit".' is also not meant offensively, simply as a statement of fact. Someone who presumably writes code for a living, but doesn't understand binary floating-point numbers, and finds themselves in a position to criticise JavaScript as being "shit" while not even having a sufficient understanding of their preferred technology to realize that it has the same (expected) behavior is objectively someone who "needs to study more" and who is "too ignorant to judge whole ecosystems as 'shit'". Edit: Before someone says it's because I wasn't being helpful, both my comments also linked to http://0.30000000000000004.com/ http://0.30000000000000004.com/ which contains an explanation.
- codetrotter 7y agoYes well, it comes off as flaming, and no matter who started it it is only adding fuel to the fire. Flamewars are not productive, no matter which side of it each of the people are at. Statement of fact or not, and insult or not, the question is, does the comment contribute to the conversation? And does it do so in a tone that is likely to be received well by others? Even if someone is inexperienced, is telling them to study more and telling them that they are ignorant useful? Not really IMO. Aside from the thing about tone, it is also a question of whether it is giving actual actionable advice. Actionable advice includes pointing to specific sources of information, like one of the sibling comments did. Non-actionable advice includes telling someone to just study more in general.
- brazilianabroad 7y ago> Yes well, it comes off as flaming, Saying that someone who doesn't understand binary floating point numbers is ignorant and needs to study, while pointing them to a study resource, isn't flaming, it's an act of selflessness. It's a favor, a grant of knowledge and a nudge in the right direction. > Even if someone is inexperienced, is telling them to study more and telling them that they are ignorant useful? Yes > Aside from the thing about tone, it is also a question of whether it is giving actual actionable advice. The sibling comments did not exist when I posted my first one. It had also a link to actionable advice. Mind you, no-one is forced to know how binary floating point operations work, as long as they don't intend on passing incorrect judgement that's brash and impolite.
- isuckatcoding 7y agoDid I miss the benchmarks somewhere?
- haapanen 7y agoHere's another one that decided to use JS naming conventions while providing Linq API: https://github.com/tomi/fromfrom https://github.com/tomi/fromfrom