10 ms·

Mindset shifts for functional programming (with Clojure)

162 points by janetacarr 4y ago

twawaaay 4y ago
> Transformations over Instructions Hell yeah! The problem with "functional" programming is that a lot of people simply don't get you want to push as much of your program to be generic tools that transform things (ie FUNCTIONS!) When I start on a new problem I typically look at what kind of tools (functions) would make the solution concise and readable. Then create the tools and then write the solution with the tools. Unfortunately most developers come with preconceived notions of how the program should be laid out and what the development process should be. If they come from OOP world, you will see code that pretty much looks like objects just without OOP machinery. If they come from scripting/procedural then you will see long stretches of instructions just split into smaller "functions". > Recursion over Looping Recursion is elegant but it is also hard to understand for many people. I object putting anything into code when the only purpose is making the code look elegant / more advanced at the cost of narrowing audience that can effectively work with it. My goal is to make the code stupid simple. My main challenge is preventing my ego from trying to impress the reader. Frequently recursion is more readable than looping. I don't hesitate using recursion in that case. But I make sure that the reader should be able to instantly recognise the pattern and the pattern is not too complex. Another thing to push recursion to be more manageable is just structuring your code to extract recursion from everything else. Make one or two very small functions that are only responsible for recursion, extract all other logic into other functions that are meaningful on its own (and just happen to be used in recursive context). If the recursion would be too complex, usually iteration is going to be easier to represent the solution in manageable chunks without requiring the reader to take everything in in one go.
- urthor 4y ago
  The problem with recursion is we don't teach people what corecursion is. Corecursion, aka literally just plain looping, is a lot simpler and better for many, many activities. If we taught recursion by contrast, then it makes sense.
  - Liberonostrud 4y ago
    [dead]
- mrkeen 4y ago
  > If the recursion would be too complex, usually iteration is going to be easier to represent the solution in manageable chunks without requiring the reader to take everything in in one go. Perhaps? But the reason I reach for recursion is so that I don't have to first produce the "wrong answer", and then modify it until it's the "right answer". The sum of 1..10 is always 55. It isn't first 0, then 1, then 3, then 6, etc.
  - twawaaay 4y ago
    I don't understand where you come from with your argument. Recursion is also producing results that are "wrong answers" although I prefer to call them partial results. Just like a loop that is producing partial results as long as it has not finished yet.
    - wizofaus 4y ago
      Out of curiosity is there actually any chipset with the algorithmic ability to add more than 2 numbers altogether in one operation (without having to store intermediate results)? I'd also think looping is far closer than recursion to how we tend to manually calculate things (either entirely in our heads or using a pen & paper or even a basic calculator). (Edit: looks like a number of other posters have made the same point already, but in dead/downvoted posts, not sure why...)
      - twawaaay 4y ago
        Chips have ability to run vector operations that can add a bunch of numbers. The issue is that recursion in software development is relatively rarely about arithmetic operation. For example, you might want to traverse a tree of objects (like a filesystem folder) to run some operation on them, etc. In this case the intermediate information is pointers to where you are currently processing various folders in your tree structure, etc. In a loop you would have to create a data structure to hold these. In recursive function this would typically be spread in multiple frames on your stack pointing to objects in your heap.
- janetacarr 4y ago
  I agree with your sentiment about recursion. It took me a damn long time to get used to it. But that's why I call it a mindset shift ;). If the codebase was written in Haskell, then there'd be no looping. Clojure is a bit odd in this case as it has a form called "loop" but it's really just a let binding over a fn, providing a point for `recur` to, well, recur to.
  - twawaaay 4y ago
    Again, the ultimate goal is to make the code readable (without sacrificing too much other qualities like performance). When you have a language and context that makes recursion easier to understand than loop -- go for recursion. One other reason to go for loops is to make sure you control the stack. With recursion it is not always immediately clear that the loop is going to get tail call optimisation. And good 9/10ths of developers meet me with a blank stare when I mention it. At least with a loop it is clearly visible how much space and in what way you are allocating. I had one dev who said he likes recursion because he says it is more memory efficient. To which I had to point out that each level of recursion creates a new stack frame. The guy just wasn't aware of it...
    - ColonelPhantom 4y ago
      Recursion isn't strictly worse though, due to TCE. And if you can manage to avoid allocating a list by recursing instead, you do save memory. (But I find it hard to think of such a case, as you can usually just loop over an iterator/generator instead?)
- epgui 4y ago
  This “recursion is more complicated” idea is a myth. Yes, a lot of people seem to have trouble with it, but it isn’t fundamentally more complex or difficult. The hardest thing about learning recursion is unlearning the other patterns. It’s totally a matter of familiarity.
tourgen 4y ago
[dead]
Liberonostrud 4y ago
[dead]
dandeere 4y ago
The biggest problem of functional languages is that they teach things of doing things that are not how computers work internally. For that C is much better. Assembly language the best. Or any other procedural language will teach you more about how computers run under the hood. Recursion is a purely math concept, in real life the chips don't work with recursion, they work strictly with if-then (more current/less current) logic. Assembly language is also procedural. Also, there is another fundamental thing that is a blocker for functional programming - our brain. For recursion you have to keep in brain many things while you call yourself. This is extremely hard (hmm, I wonder why the Lisp Machine was so pricey ;) ) - especially for the majority of people whose brain has problem to remember more than 2 things in the span of 5 seconds. If you like math, yeah, sure use Haskell or Clojure, but for the most people it's an overkill and for the most part people still tend to write very procedural code - you have for loops in Clojure - I am wondering why ;D I like some things that are often associated with functional programming, e.g. immutability, but I am not a fan of recursion at all. Also there seem to be some obsession of having dynamic types in many Lisp and functional programming languages, for example Clojure or Elixir, and I hate it.
- chii 4y ago
  > For recursion you have to keep in brain many things while you call yourself. This is extremely hard it depends on your problem domain. Recursion makes some things easier - much easier than iteration. Try solving tower of hanoi without using recursion and see how hard it is: https://www.geeksforgeeks.org/iterative-tower-of-hanoi/ https://www.geeksforgeeks.org/iterative-tower-of-hanoi/ Or try writing a merge sort algorithm without using recursion.
margorczynski 4y ago
I think the main thing is not really FP vs whatever else but the most fundamental thing is declarative vs imperative. That is probably the biggest jump in mindset when coming from e.g. C or C++ to something like Haskell - that instead of telling the computer/compiler what to do, you describe what you want.
- crop_rotation 4y ago
  You are still telling the computer what to do, describing what you want is more like prolog. The main difference between Haskell and C in a case like this would be the level at which you tell it what to do.
  - margorczynski 4y ago
    That's a truism that when extend equates Firefox or Excel with programming with C because at the end of the day you tell the computer what to do via some interface (text programming language or graphical with much more abstraction layers).
- janetacarr 4y ago
  I think this would be correct for declarative programming languages, but I don't agree that Haskell is a declarative programming language. Haskell is pure functional programming in my mind. A declarative programming language might be something more akin to DML SQL for a RDBMS, or HCL for Terraform (pre-looping, v0.X).
  - margorczynski 4y ago
    Pure FP languages are declarative just with additional restrictions like referential transparity, no explicit handling of state and immutability of the underlying state. For these restrictions to be effectively held it requires for it to be declarative.
    - janetacarr 4y ago
      Oh I see what you mean. I had to look into this a bit. Sorry for the confusion!
roenxi 4y ago
> Recursion over Looping Part of what makes Clojure a great programming language is that you don't have to believe this if you don't want to. Nobody has yet convinced me that recursion has any sustained advantage over looping. Using a loop is generally bad practice if a more specialised operation is available (don't loop if something is a simple map or reduce for example). But if the situation justifies a recursion then it usually justifies a loop unless the recursion is particularly neat. Recursion has the same problem as looping - it doesn't tell anyone anything about what the code is really doing. If I see map then I have implicit and explicit expectations about what is about to happen. I enjoyed the article though.
- nerdponx 4y ago
  I think the difference between recursion and looping is that the former requires you to be very explicit about the state you intend to modify and persist between iterations, while it's generally somewhat of a free-for-all in imperative loops.
  - slifin 4y ago
    The most common looping construct I see in clojure is the for statement Which isn't quite the free for all it is in traditional languages You have to be very conscious and coding against the grain to use it to bang on values in place
    - nerdponx 4y ago
      This is a general design principle that I appreciate about the Lisp family of languages in general. You can write highly imperative code that mutates things in-place, but it's a little harder to do and it's something you do only when you really need to do it. Whereas the default idioms are functional, and functional design is the path of least resistance. Perl is morally the opposite of functional programming in many ways, but I actually think this design is a great embodiment of the "make easy things easy, make hard things possible" ethos of Larry Wall.
- js8 4y ago
sharas- 4y ago
The advantage of clojure is in data centricity. It is not about shifting mind to recursion from looping. Clojure has a 'for' which is it's list comprehension loop. The actual fun of clojure: https://bitslap.it/blog/posts/fun-of-clojure.html https://bitslap.it/blog/posts/fun-of-clojure.html
- janetacarr 4y ago
  Clojure data type's are fantastic, but the main thesis of my post isn't what's required for FP in Clojure, rather, what's required to become comfortable with pure functional programming concepts which is why I reference Haskell a lot in the post. The examples just happen to be in Clojure. Sorry for the confusion.
  - sharas- 4y ago
    I get it. But my point is that clojure is not snobbish about pure functional programming. It's angle is data centricity. Very different to haskell in that respect and type centricity of it.
    - janetacarr 4y ago
      I agree with you, but I also never said it's snobbish about pure functional programming. The way I see it, pure functional programming, like anything, is just a tool in the belt to help think about solutions in a different manner.
samsquire 4y ago
If your team and you can read it later or when it goes wrong, then it's fine. I don't want to need to think hard about what code does, it should be clear. Or jump through 100 files to find a simple algorithm that has been divided into 100 pieces. I think people refactor to their own understanding or mental model or refactor-to-understand. I think there are cases where imperative code is intuitive and others where functional code is intuitive. I've worked on two professional projects in Clojure at a surface level but I still find Python easier to read, but that's my experience YMMV. There's a point in my programming projects when my own understandability is weakened and I this week I've been trying to think of a mindset that simplifies the problem. I am experimenting with multithreaded code in Java that implements left-right concurrency control. The idea is readers don't block writers and writers don't block readers and writers don't block writers. Give each thread a shard and rely on commutative property of your tree data structure and have a coordinator thread handle merging. The benefit: each thread can operate at single core speeds without any synchronization for reading OR writing. Buffer flipping is handled by the coordinator thread. Each thread has its own copy of state which is merged by the coordinating thread. So it's an eventually consistent system. The result: each thread can modify its own copy of the data as much as it wants and it can see its own snapshot of global state at the last snapshot point. Cross thread writes always happen on the inactive buffer. How do you think about data transformation pipelines? If only there was an IDE for kafka or clojure transformation pipelines.
- janetacarr 4y ago
  Agreed, cyclomatic complexity is definitely something to be aware of when designing functional programming systems. Although, I disagree about Clojure's readability, but that's probably because I've been doing it for so long. Interesting Java project you've got there. Reminds me of Clojure's core.async library and software transaction memory ;) (though, I know it's not exactly the same thing). If I was transforming large streaming input from Kafka or Kinesis, I would probably reach for Cortex (a map-reduce style lib for Clojure), or I might rig up some strange, stream-to-lazy-seq adapter and reach for transducers as they have much better performance when dealing with larger input.
  - 4y ago
twic 4y ago
I am not a Clojure programmer, and am pretty skeptical of LISPs and FP more generally, but i have to say that transducers are pretty great. The descriptions of them, and the way the interface is expressed, are a bit off-putting, but once you grok them they're actually simple and useful. They occupy the same space as Java's streams, but manage to do the same stuff with a smaller, more generic, more extensible interface, that can do more stuff (eg intermediate stages get told when input is finished). I'm jealous. EDIT: IIUC, in Java terms, Clojure's "reducing functions" are like Collector, and a transducer is a function which takes a Collector and returns another one. There is then a little bit of top-level sugar so you can give a sequence of transducers, terminating in a concrete reducing function, and get back a reducing function which itself feeds things through the chain of reducing functions built by the transducers. You could probably replicate this in Java, but it might be too clunky even for Java programmers.
- janetacarr 4y ago
  Transducers are great! They were an small obsession of mine last week as I wrote an accompanying blog post to demystify them.
  - twic 4y ago
    Aha, this one i suppose (your blog does not have a browseable index, although it does have search): https://blog.janetacarr.com/clojure-transducers-your-composable-big-data-pipelines/ https://blog.janetacarr.com/clojure-transducers-your-composa... Personally, i would say that a blog post which starts "We can think of a transducer as a context-independent transformation composed of, say, many reducers" and then starts adding parentheses is not really demystifying. But perhaps i am not the target audience.
    - janetacarr 4y ago
      Oh yeah. If you didn't know what a reducer was, the article might be a bit confusing, eh?
- twic 4y ago
vrglvrglvrgl 4y ago
[dead]
koito17 4y ago
As a Clojure programmer, I find it a bit unfortunate that most of this post, like many other posts demonstrating Clojure, tend to paint a rosey picture and never mention the realities one has to reconcile in a Clojure codebase. For instance, one of the sections in this blog post is titled "Functions over Objects" but any Clojure programmer with experience will agree that most code ends up becoming some sort of map munging. Rather than "functions over objects" you really have "map-munging and function instrumentation in dev over banging on concrete instances of classes". There is also this statement > functional programming languages often facilitate iteration through recursion then the author proceeds to mention loop/recur, which is nice, but it is dishonest at best and lying at worst to imply loop/recur is the most natural or common way we iterate over a structure. Off the top of my head, I always see idioms like `(for [[k v] some-map] (do things with k v))`. It would also be nice to demonstrate how faux-recursive looks like with loop/recur rather than stating Clojure's lack of TCO and not elaborating further on how the idiomatic version of `recursive-map` would look like. Next, > by (nearly) eliminating side-effects, functional programming (nearly) elminates this whole class of bugs. That may be the intent of purely functional code in general, but in Clojure one regularly interoperates with the host, to take advantage of their rich ecosystem. Many of the libraries one will interop with will involve some imperative or stateful things! So I think it'd be better if the author had mentioned the idea of "functional core, imperative shell." That is a pattern most Clojurists would agree with, and it is what people really do in a code base, or at least attempt to design. Lastly, while Clojure allows one very naturally to mock up a finite state machine with a single map and writing a few functions to describe transitions given the current state of the machine, I want to emphasize that not much Clojure codebases I've seen ever used FSMs explicitly :-) Overall, the article is okay, but my brain registers it as another "Clojure portrayed with rose-tinted glasses and contrived examples" article. Not that this is bad. It's just not the kind of sales pitch I'd want to show to non-Clojure programmers, because they will likely want to ask questions about maintainability, testing, instrumentation, etc. And it is possible to show Clojure being nice for each of these things. At a previous job I got to see ClojureScript code dating back to 2014, untouched, and surviving 8 years worth of language and library updates.

162 points by janetacarr 4y ago

> Transformations over Instructions Hell yeah! The problem with "functional" programming is that a lot of people simply don't get you want to push as much of your program to be generic tools that transform things (ie FUNCTIONS!) When I start on a new problem I typically look at what kind of tools (functions) would make the solution concise and readable. Then create the tools and then write the solution with the tools. Unfortunately most developers come with preconceived notions of how the program should be laid out and what the development process should be. If they come from OOP world, you will see code that pretty much looks like objects just without OOP machinery. If they come from scripting/procedural then you will see long stretches of instructions just split into smaller "functions". > Recursion over Looping Recursion is elegant but it is also hard to understand for many people. I object putting anything into code when the only purpose is making the code look elegant / more advanced at the cost of narrowing audience that can effectively work with it. My goal is to make the code stupid simple. My main challenge is preventing my ego from trying to impress the reader. Frequently recursion is more readable than looping. I don't hesitate using recursion in that case. But I make sure that the reader should be able to instantly recognise the pattern and the pattern is not too complex. Another thing to push recursion to be more manageable is just structuring your code to extract recursion from everything else. Make one or two very small functions that are only responsible for recursion, extract all other logic into other functions that are meaningful on its own (and just happen to be used in recursive context). If the recursion would be too complex, usually iteration is going to be easier to represent the solution in manageable chunks without requiring the reader to take everything in in one go.

urthor 4y ago
The problem with recursion is we don't teach people what corecursion is. Corecursion, aka literally just plain looping, is a lot simpler and better for many, many activities. If we taught recursion by contrast, then it makes sense.
- Liberonostrud 4y ago
  [dead]
mrkeen 4y ago
> If the recursion would be too complex, usually iteration is going to be easier to represent the solution in manageable chunks without requiring the reader to take everything in in one go. Perhaps? But the reason I reach for recursion is so that I don't have to first produce the "wrong answer", and then modify it until it's the "right answer". The sum of 1..10 is always 55. It isn't first 0, then 1, then 3, then 6, etc.
- twawaaay 4y ago
  I don't understand where you come from with your argument. Recursion is also producing results that are "wrong answers" although I prefer to call them partial results. Just like a loop that is producing partial results as long as it has not finished yet.
  - wizofaus 4y ago
    Out of curiosity is there actually any chipset with the algorithmic ability to add more than 2 numbers altogether in one operation (without having to store intermediate results)? I'd also think looping is far closer than recursion to how we tend to manually calculate things (either entirely in our heads or using a pen & paper or even a basic calculator). (Edit: looks like a number of other posters have made the same point already, but in dead/downvoted posts, not sure why...)
    - twawaaay 4y ago
      Chips have ability to run vector operations that can add a bunch of numbers. The issue is that recursion in software development is relatively rarely about arithmetic operation. For example, you might want to traverse a tree of objects (like a filesystem folder) to run some operation on them, etc. In this case the intermediate information is pointers to where you are currently processing various folders in your tree structure, etc. In a loop you would have to create a data structure to hold these. In recursive function this would typically be spread in multiple frames on your stack pointing to objects in your heap.
janetacarr 4y ago
I agree with your sentiment about recursion. It took me a damn long time to get used to it. But that's why I call it a mindset shift ;). If the codebase was written in Haskell, then there'd be no looping. Clojure is a bit odd in this case as it has a form called "loop" but it's really just a let binding over a fn, providing a point for `recur` to, well, recur to.
- twawaaay 4y ago
  Again, the ultimate goal is to make the code readable (without sacrificing too much other qualities like performance). When you have a language and context that makes recursion easier to understand than loop -- go for recursion. One other reason to go for loops is to make sure you control the stack. With recursion it is not always immediately clear that the loop is going to get tail call optimisation. And good 9/10ths of developers meet me with a blank stare when I mention it. At least with a loop it is clearly visible how much space and in what way you are allocating. I had one dev who said he likes recursion because he says it is more memory efficient. To which I had to point out that each level of recursion creates a new stack frame. The guy just wasn't aware of it...
  - ColonelPhantom 4y ago
    Recursion isn't strictly worse though, due to TCE. And if you can manage to avoid allocating a list by recursing instead, you do save memory. (But I find it hard to think of such a case, as you can usually just loop over an iterator/generator instead?)
epgui 4y ago
This “recursion is more complicated” idea is a myth. Yes, a lot of people seem to have trouble with it, but it isn’t fundamentally more complex or difficult. The hardest thing about learning recursion is unlearning the other patterns. It’s totally a matter of familiarity.

tourgen 4y ago

[dead]

Liberonostrud 4y ago

[dead]

dandeere 4y ago

The biggest problem of functional languages is that they teach things of doing things that are not how computers work internally. For that C is much better. Assembly language the best. Or any other procedural language will teach you more about how computers run under the hood. Recursion is a purely math concept, in real life the chips don't work with recursion, they work strictly with if-then (more current/less current) logic. Assembly language is also procedural. Also, there is another fundamental thing that is a blocker for functional programming - our brain. For recursion you have to keep in brain many things while you call yourself. This is extremely hard (hmm, I wonder why the Lisp Machine was so pricey ;) ) - especially for the majority of people whose brain has problem to remember more than 2 things in the span of 5 seconds. If you like math, yeah, sure use Haskell or Clojure, but for the most people it's an overkill and for the most part people still tend to write very procedural code - you have for loops in Clojure - I am wondering why ;D I like some things that are often associated with functional programming, e.g. immutability, but I am not a fan of recursion at all. Also there seem to be some obsession of having dynamic types in many Lisp and functional programming languages, for example Clojure or Elixir, and I hate it.

chii 4y ago
> For recursion you have to keep in brain many things while you call yourself. This is extremely hard it depends on your problem domain. Recursion makes some things easier - much easier than iteration. Try solving tower of hanoi without using recursion and see how hard it is: https://www.geeksforgeeks.org/iterative-tower-of-hanoi/ https://www.geeksforgeeks.org/iterative-tower-of-hanoi/ Or try writing a merge sort algorithm without using recursion.

margorczynski 4y ago

I think the main thing is not really FP vs whatever else but the most fundamental thing is declarative vs imperative. That is probably the biggest jump in mindset when coming from e.g. C or C++ to something like Haskell - that instead of telling the computer/compiler what to do, you describe what you want.

crop_rotation 4y ago
You are still telling the computer what to do, describing what you want is more like prolog. The main difference between Haskell and C in a case like this would be the level at which you tell it what to do.
- margorczynski 4y ago
  That's a truism that when extend equates Firefox or Excel with programming with C because at the end of the day you tell the computer what to do via some interface (text programming language or graphical with much more abstraction layers).
janetacarr 4y ago
I think this would be correct for declarative programming languages, but I don't agree that Haskell is a declarative programming language. Haskell is pure functional programming in my mind. A declarative programming language might be something more akin to DML SQL for a RDBMS, or HCL for Terraform (pre-looping, v0.X).
- margorczynski 4y ago
  Pure FP languages are declarative just with additional restrictions like referential transparity, no explicit handling of state and immutability of the underlying state. For these restrictions to be effectively held it requires for it to be declarative.
  - janetacarr 4y ago
    Oh I see what you mean. I had to look into this a bit. Sorry for the confusion!

roenxi 4y ago

> Recursion over Looping Part of what makes Clojure a great programming language is that you don't have to believe this if you don't want to. Nobody has yet convinced me that recursion has any sustained advantage over looping. Using a loop is generally bad practice if a more specialised operation is available (don't loop if something is a simple map or reduce for example). But if the situation justifies a recursion then it usually justifies a loop unless the recursion is particularly neat. Recursion has the same problem as looping - it doesn't tell anyone anything about what the code is really doing. If I see map then I have implicit and explicit expectations about what is about to happen. I enjoyed the article though.

nerdponx 4y ago
I think the difference between recursion and looping is that the former requires you to be very explicit about the state you intend to modify and persist between iterations, while it's generally somewhat of a free-for-all in imperative loops.
- slifin 4y ago
  The most common looping construct I see in clojure is the for statement Which isn't quite the free for all it is in traditional languages You have to be very conscious and coding against the grain to use it to bang on values in place
  - nerdponx 4y ago
    This is a general design principle that I appreciate about the Lisp family of languages in general. You can write highly imperative code that mutates things in-place, but it's a little harder to do and it's something you do only when you really need to do it. Whereas the default idioms are functional, and functional design is the path of least resistance. Perl is morally the opposite of functional programming in many ways, but I actually think this design is a great embodiment of the "make easy things easy, make hard things possible" ethos of Larry Wall.
js8 4y ago

sharas- 4y ago

The advantage of clojure is in data centricity. It is not about shifting mind to recursion from looping. Clojure has a 'for' which is it's list comprehension loop. The actual fun of clojure: https://bitslap.it/blog/posts/fun-of-clojure.html https://bitslap.it/blog/posts/fun-of-clojure.html

janetacarr 4y ago
Clojure data type's are fantastic, but the main thesis of my post isn't what's required for FP in Clojure, rather, what's required to become comfortable with pure functional programming concepts which is why I reference Haskell a lot in the post. The examples just happen to be in Clojure. Sorry for the confusion.
- sharas- 4y ago
  I get it. But my point is that clojure is not snobbish about pure functional programming. It's angle is data centricity. Very different to haskell in that respect and type centricity of it.
  - janetacarr 4y ago
    I agree with you, but I also never said it's snobbish about pure functional programming. The way I see it, pure functional programming, like anything, is just a tool in the belt to help think about solutions in a different manner.

samsquire 4y ago

If your team and you can read it later or when it goes wrong, then it's fine. I don't want to need to think hard about what code does, it should be clear. Or jump through 100 files to find a simple algorithm that has been divided into 100 pieces. I think people refactor to their own understanding or mental model or refactor-to-understand. I think there are cases where imperative code is intuitive and others where functional code is intuitive. I've worked on two professional projects in Clojure at a surface level but I still find Python easier to read, but that's my experience YMMV. There's a point in my programming projects when my own understandability is weakened and I this week I've been trying to think of a mindset that simplifies the problem. I am experimenting with multithreaded code in Java that implements left-right concurrency control. The idea is readers don't block writers and writers don't block readers and writers don't block writers. Give each thread a shard and rely on commutative property of your tree data structure and have a coordinator thread handle merging. The benefit: each thread can operate at single core speeds without any synchronization for reading OR writing. Buffer flipping is handled by the coordinator thread. Each thread has its own copy of state which is merged by the coordinating thread. So it's an eventually consistent system. The result: each thread can modify its own copy of the data as much as it wants and it can see its own snapshot of global state at the last snapshot point. Cross thread writes always happen on the inactive buffer. How do you think about data transformation pipelines? If only there was an IDE for kafka or clojure transformation pipelines.

janetacarr 4y ago
Agreed, cyclomatic complexity is definitely something to be aware of when designing functional programming systems. Although, I disagree about Clojure's readability, but that's probably because I've been doing it for so long. Interesting Java project you've got there. Reminds me of Clojure's core.async library and software transaction memory ;) (though, I know it's not exactly the same thing). If I was transforming large streaming input from Kafka or Kinesis, I would probably reach for Cortex (a map-reduce style lib for Clojure), or I might rig up some strange, stream-to-lazy-seq adapter and reach for transducers as they have much better performance when dealing with larger input.
- 4y ago

twic 4y ago

I am not a Clojure programmer, and am pretty skeptical of LISPs and FP more generally, but i have to say that transducers are pretty great. The descriptions of them, and the way the interface is expressed, are a bit off-putting, but once you grok them they're actually simple and useful. They occupy the same space as Java's streams, but manage to do the same stuff with a smaller, more generic, more extensible interface, that can do more stuff (eg intermediate stages get told when input is finished). I'm jealous. EDIT: IIUC, in Java terms, Clojure's "reducing functions" are like Collector, and a transducer is a function which takes a Collector and returns another one. There is then a little bit of top-level sugar so you can give a sequence of transducers, terminating in a concrete reducing function, and get back a reducing function which itself feeds things through the chain of reducing functions built by the transducers. You could probably replicate this in Java, but it might be too clunky even for Java programmers.

janetacarr 4y ago
Transducers are great! They were an small obsession of mine last week as I wrote an accompanying blog post to demystify them.
- twic 4y ago
  Aha, this one i suppose (your blog does not have a browseable index, although it does have search): https://blog.janetacarr.com/clojure-transducers-your-composable-big-data-pipelines/ https://blog.janetacarr.com/clojure-transducers-your-composa... Personally, i would say that a blog post which starts "We can think of a transducer as a context-independent transformation composed of, say, many reducers" and then starts adding parentheses is not really demystifying. But perhaps i am not the target audience.
  - janetacarr 4y ago
    Oh yeah. If you didn't know what a reducer was, the article might be a bit confusing, eh?
twic 4y ago

vrglvrglvrgl 4y ago

[dead]

koito17 4y ago

As a Clojure programmer, I find it a bit unfortunate that most of this post, like many other posts demonstrating Clojure, tend to paint a rosey picture and never mention the realities one has to reconcile in a Clojure codebase. For instance, one of the sections in this blog post is titled "Functions over Objects" but any Clojure programmer with experience will agree that most code ends up becoming some sort of map munging. Rather than "functions over objects" you really have "map-munging and function instrumentation in dev over banging on concrete instances of classes". There is also this statement > functional programming languages often facilitate iteration through recursion then the author proceeds to mention loop/recur, which is nice, but it is dishonest at best and lying at worst to imply loop/recur is the most natural or common way we iterate over a structure. Off the top of my head, I always see idioms like `(for [[k v] some-map] (do things with k v))`. It would also be nice to demonstrate how faux-recursive looks like with loop/recur rather than stating Clojure's lack of TCO and not elaborating further on how the idiomatic version of `recursive-map` would look like. Next, > by (nearly) eliminating side-effects, functional programming (nearly) elminates this whole class of bugs. That may be the intent of purely functional code in general, but in Clojure one regularly interoperates with the host, to take advantage of their rich ecosystem. Many of the libraries one will interop with will involve some imperative or stateful things! So I think it'd be better if the author had mentioned the idea of "functional core, imperative shell." That is a pattern most Clojurists would agree with, and it is what people really do in a code base, or at least attempt to design. Lastly, while Clojure allows one very naturally to mock up a finite state machine with a single map and writing a few functions to describe transitions given the current state of the machine, I want to emphasize that not much Clojure codebases I've seen ever used FSMs explicitly :-) Overall, the article is okay, but my brain registers it as another "Clojure portrayed with rose-tinted glasses and contrived examples" article. Not that this is bad. It's just not the kind of sales pitch I'd want to show to non-Clojure programmers, because they will likely want to ask questions about maintainability, testing, instrumentation, etc. And it is possible to show Clojure being nice for each of these things. At a previous job I got to see ClojureScript code dating back to 2014, untouched, and surviving 8 years worth of language and library updates.