4 ms·

What's wrong with Nim's iterators?

_dain_ 4y ago
I wrote a comment about it a while go: https://news.ycombinator.com/item?id=34669902 https://news.ycombinator.com/item?id=34669902 basically Nim has several features which on their own seem great, but don't compose with each other properly. also it's been many months since I used Nim so some of the details may be outdated or misremembered but the overall gist of what i'm saying is >90% true so i don't care if anyone nitpicks this: Nim has UFCS (uniform function call syntax). so whenever you have f(a, b), you can write a.f(b), and it means literally the same thing (until you read the fine print for the parts where it's not). and you can also omit parentheses when it's unambiguous, so you can write stuff resembling method-chaining or unix pipelines in a totally natural way: foo.bar.baz(x, y).qaaz().quux is the same as quux(qaaz(baz(bar(foo()), x, y))) and way more readable bc less parens and the execution order is left-to-right. it's so, SO, tempting to write lots of code this way, because it's such a rare language feature, and it fits with the way I think. but it's not just functions, there are a lot of function-like things, i.e. the stuff you can write as f(a, b): methods, ad-hoc overloaded functions, generic functions, templates, macros, iterators, etc. but things don't work consistently across all combinations of function-like objects, argument types, and function-call syntaxes. iterators is one part where the abstraction breaks, and iirc there are a few others (certain kinds of templates?). there are two kinds of iterators: inline, and closure. inline iterators are meant to be a zero-cost abstraction, basically a for loop that gets inlined at the call site by the compiler, and they're not first-class objects. closure iterators otoh are first-class objects and can be passed around, but they have runtime overhead (idk how much). so lets say you want to make an API that consumes an iterator and does something useful with it, e.g. higher-order functions like python's itertools (Nim advertises itself as being similar to python in important respects like this so it's a valid comparison). great, so then you try to use it on the builtin "lines" iterator (the one that splits a file at newlines and yields each line as a string). oh no, you can't, because "lines" is an inline iterator so you can't pass it to a function. fine, so the language has handed me this pile of dogshit that I can't do anything useful with directly. surely this would be a common enough problem that there must be a helper macro in the stdlib that lets you convert between inline and closure iterators? nope, there isn't. okay but writing one should be trivial, right, given the vaunted metaprogramming capabilities of Nim? well, I tried writing one in the most obvious way. and it behaved differently when I used UFCS vs not. and when I experimentally called it on a closure iterator (expecting it to be effectively a no-op), the compiler segfaulted. and when I asked about this in the discord, nobody could figure out why it didn't work. later I found an implementation of inline<->closure conversion macro and it was like hundreds of lines of gensym fuckery and god knows what. i don't know why it should be so complicated. oh and btw, even with the supposedly "first-class" closure iterators, it is impossible to write a generic function like f: iterable[T] -> iterator[T]. you have to use a template. and the syntax for doing it is counterintuitive. and the "iterable" keyword in the function signature doesn't mean what you think anyway. when you ask in the chat "oh my god just please tell me how to do lazy iteration", they say "don't worry it's easy, just use [x]", where x is one of half a dozen mutually incompatible 3rd party libraries, each one expecting you to learn its own mad little DSL that you can't really extend or integrate with normal Nim code. and then you look at the implementation: enormous inscrutable macros, which are that way presumably to work around all the crap I described in the last paragraphs and even more I don't know about. I don't want that in my dependencies! the docs don't prepare you for all this complexity; templates/macros/iterators are documented like that "draw the rest of the fucking owl" meme. the "sugar" module has a macro to let you write lambdas more compactly. but if you try to do something straightforward like pass one into sort() as a comparison function (ie the kind of stuff that comprises 95% of the legit uses of anonymous functions in grugbrain code), it doesn't fucking work reliably. like it will probably work on a simple case like sorting a bunch of strings by their length, but the minute you introduce some complication like generics it all falls apart and the compiler spams ten pages of indecipherable type errors at you. and then they'll say "just use sortIt" -- no, fuck you, i will never use sortIt, i hate anaphoric macros, just let me pass in an honest function. what I'm getting at is, you know how frictionless it is to sling around iterators and generators and first-class functions and so on in Python, even with mypy --strict? and you read about Nim's UFCS and macros and generics and you think "great, best of both worlds", imagining yourself as a ninja doing "myfile.txt".lines.map(func1).filter(func2).max(key=blah).frobnicate, and it all Just Works[tm]? well the reality is as frictionless as dragging your face across raw asphalt.
- planetis 4y ago
  > what i'm saying is >90% true so i don't care if anyone nitpicks this Can't argue with that logic :-P You've obviously spent years writing nim and know what you're talking about /s But in all seriousness, as the author of the sugar.collect macro, I am fond of it. I understand that not everyone likes how fp is done nim but it's ok.
  - _dain_ 4y ago
    thanks for the work you do! my comment was very harsh, but I would never have written such a long comment if I didn't fundamentally like what Nim is trying to do. it comes from disappointment, not dislike. fwiw I've put in maybe 100 hours honestly trying to learn and use the language, and I don't have much to show for it. I think that has to count as some kind of failure, of the language and/or the documentation. I haven't run into this with any other language.
    - planetis 4y ago
      Thank you for your kind words sir. I didn't expect it. And I am sorry for your lost time. But I hope the situation improves.
      - summarity 4y ago
        > how fp is done nim It's great that a composition of libraries (sequtils, sugar, algorithm) can get us close to the effortless quasi-FP people may be used to from LINQ. I just wish this would be merged into one library. Pretty much every single module that uses one of them imports all three in my code. We need one, consistent, awesome library to interact with collections.

I wrote a comment about it a while go: https://news.ycombinator.com/item?id=34669902 https://news.ycombinator.com/item?id=34669902 basically Nim has several features which on their own seem great, but don't compose with each other properly. also it's been many months since I used Nim so some of the details may be outdated or misremembered but the overall gist of what i'm saying is >90% true so i don't care if anyone nitpicks this: Nim has UFCS (uniform function call syntax). so whenever you have f(a, b), you can write a.f(b), and it means literally the same thing (until you read the fine print for the parts where it's not). and you can also omit parentheses when it's unambiguous, so you can write stuff resembling method-chaining or unix pipelines in a totally natural way: foo.bar.baz(x, y).qaaz().quux is the same as quux(qaaz(baz(bar(foo()), x, y))) and way more readable bc less parens and the execution order is left-to-right. it's so, SO, tempting to write lots of code this way, because it's such a rare language feature, and it fits with the way I think. but it's not just functions, there are a lot of function-like things, i.e. the stuff you can write as f(a, b): methods, ad-hoc overloaded functions, generic functions, templates, macros, iterators, etc. but things don't work consistently across all combinations of function-like objects, argument types, and function-call syntaxes. iterators is one part where the abstraction breaks, and iirc there are a few others (certain kinds of templates?). there are two kinds of iterators: inline, and closure. inline iterators are meant to be a zero-cost abstraction, basically a for loop that gets inlined at the call site by the compiler, and they're not first-class objects. closure iterators otoh are first-class objects and can be passed around, but they have runtime overhead (idk how much). so lets say you want to make an API that consumes an iterator and does something useful with it, e.g. higher-order functions like python's itertools (Nim advertises itself as being similar to python in important respects like this so it's a valid comparison). great, so then you try to use it on the builtin "lines" iterator (the one that splits a file at newlines and yields each line as a string). oh no, you can't, because "lines" is an inline iterator so you can't pass it to a function. fine, so the language has handed me this pile of dogshit that I can't do anything useful with directly. surely this would be a common enough problem that there must be a helper macro in the stdlib that lets you convert between inline and closure iterators? nope, there isn't. okay but writing one should be trivial, right, given the vaunted metaprogramming capabilities of Nim? well, I tried writing one in the most obvious way. and it behaved differently when I used UFCS vs not. and when I experimentally called it on a closure iterator (expecting it to be effectively a no-op), the compiler segfaulted. and when I asked about this in the discord, nobody could figure out why it didn't work. later I found an implementation of inline<->closure conversion macro and it was like hundreds of lines of gensym fuckery and god knows what. i don't know why it should be so complicated. oh and btw, even with the supposedly "first-class" closure iterators, it is impossible to write a generic function like f: iterable[T] -> iterator[T]. you have to use a template. and the syntax for doing it is counterintuitive. and the "iterable" keyword in the function signature doesn't mean what you think anyway. when you ask in the chat "oh my god just please tell me how to do lazy iteration", they say "don't worry it's easy, just use [x]", where x is one of half a dozen mutually incompatible 3rd party libraries, each one expecting you to learn its own mad little DSL that you can't really extend or integrate with normal Nim code. and then you look at the implementation: enormous inscrutable macros, which are that way presumably to work around all the crap I described in the last paragraphs and even more I don't know about. I don't want that in my dependencies! the docs don't prepare you for all this complexity; templates/macros/iterators are documented like that "draw the rest of the fucking owl" meme. the "sugar" module has a macro to let you write lambdas more compactly. but if you try to do something straightforward like pass one into sort() as a comparison function (ie the kind of stuff that comprises 95% of the legit uses of anonymous functions in grugbrain code), it doesn't fucking work reliably. like it will probably work on a simple case like sorting a bunch of strings by their length, but the minute you introduce some complication like generics it all falls apart and the compiler spams ten pages of indecipherable type errors at you. and then they'll say "just use sortIt" -- no, fuck you, i will never use sortIt, i hate anaphoric macros, just let me pass in an honest function. what I'm getting at is, you know how frictionless it is to sling around iterators and generators and first-class functions and so on in Python, even with mypy --strict? and you read about Nim's UFCS and macros and generics and you think "great, best of both worlds", imagining yourself as a ninja doing "myfile.txt".lines.map(func1).filter(func2).max(key=blah).frobnicate, and it all Just Works[tm]? well the reality is as frictionless as dragging your face across raw asphalt.

planetis 4y ago
> what i'm saying is >90% true so i don't care if anyone nitpicks this Can't argue with that logic :-P You've obviously spent years writing nim and know what you're talking about /s But in all seriousness, as the author of the sugar.collect macro, I am fond of it. I understand that not everyone likes how fp is done nim but it's ok.
- _dain_ 4y ago
  thanks for the work you do! my comment was very harsh, but I would never have written such a long comment if I didn't fundamentally like what Nim is trying to do. it comes from disappointment, not dislike. fwiw I've put in maybe 100 hours honestly trying to learn and use the language, and I don't have much to show for it. I think that has to count as some kind of failure, of the language and/or the documentation. I haven't run into this with any other language.
  - planetis 4y ago
    Thank you for your kind words sir. I didn't expect it. And I am sorry for your lost time. But I hope the situation improves.
    - summarity 4y ago
      > how fp is done nim It's great that a composition of libraries (sequtils, sugar, algorithm) can get us close to the effortless quasi-FP people may be used to from LINQ. I just wish this would be merged into one library. Pretty much every single module that uses one of them imports all three in my code. We need one, consistent, awesome library to interact with collections.