6 ms·

The recent focus on adapting functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadr

The recent focus on adapting functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadrant" chart. Before multi-core, multi-threading, it didn't matter how or when you decided to update your local variables, they were YOUR local variables... unshared. With threading, and multiple cores, we all just blindly moved right to the mutable/shared quadrant... the one in bright red... which effectively changes the laws of physics of your programs. Effectively every piece of code now runs in it's own time and space, and if you don't coordinate things correctly, it's like killing your own grandparents. It took me a long time to understand why you'd want to refactor your code until everything was immutable[1], but now that I get the lesson, its something I won't forget. Functional programming avoids mutable data, and thus, intentionally or not, works well in a world of shared data. Because pure functions have no side effects, they are timeless - they need no synchronization. Spend a few nights watching everything Kevlin Henney has said in the last few years, and you'll have a much better handle on things. [1] - https://youtu.be/APUCMSPiNh4 [Edit] Incorporate wording suggestion from DonaldPShimoda

DonaldPShimoda 4y ago
Just a small nitpick about your comment: > Functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadrant" chart. I might rather say "The recent focus on adapting functional programming is a reaction...". Functional programming itself is nearly as old as the modern computer (McCarthy's first LISP paper was published in 1960), but your phrasing kind of suggests that the entirety of FP is a recent innovation.
- frostwarrior 4y ago
  Functional programming itself is really old. Practical functional programming for scale is the new trend. What's new is the scale of software being developed right now. Developers simply can't afford to create mutable and shared state as if their program was just another desktop program, executed on a computer disconnected from the internet. Sooner or later we learn that focusing on the human value of code (I mean, making the code clear, concise and simple to read) pays off way better on the long run rather than optimizing for CPU cycles or memory footprint.
nine_k 4y ago
Your global variables were / are shared with yourself, even in no-concurrency case. With typical control flow being far from linear, it's easy to surprise yourself with unexpected state mutation.
- Scarbutt 4y ago
  Hence the value of immutability even for single-threaded code.
- Jtsummers 4y ago
  Giving me flashbacks to inherited codebases from old-school FORTRAN coders... // 500 variables shared by everything main loop "task" 1 "task" 2 "task" 3 ... "task" n So t121 is now supposed to only be used by task 3, but "t" as a prefix means it is temp. However, we don't want to change all the references. So just don't use it everyone else, please?
- MaxBarraclough 4y ago
  Right. Pure functions make it considerably easier to reason about code, even in the absence of concurrency. It's a shame D's pure modifier (to statically guarantee purity) isn't available in many other languages. [0] SPARK Ada has something similar in that functions must always be pure, forcing procedures to be used for impure functionality. [1] edit I think that's mistaken, SPARK appears to permit functions to read globals but not to modify them. edit John Carmack has written about this kind of thing. From [2]: > if a function only references a piece or two of global state, it is probably wise to consider passing it in as a variable. It would be kind of nice if C had a "functional" keyword to enforce no global references. [0] https://dlang.org/spec/function.html#pure-functions https://dlang.org/spec/function.html#pure-functions [1] https://learn.adacore.com/courses/intro-to-spark/chapters/01_Overview.html https://learn.adacore.com/courses/intro-to-spark/chapters/01... [2] https://cbarrete.com/carmack.html https://cbarrete.com/carmack.html
jahewson 4y ago
> Functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadrant" chart. This makes it sound like you’re saying that’s why functional programming was created, which isn’t the case. I assume you mean that this is why it’s gained popularity but I don’t really agree with that - it’s certainly valuable but most functional programmers will never touch threads. I’d argue that programs without state are just easier to reason about and debug and for most software this is where the value comes from.
- klipt 4y ago
  > I’d argue that programs without state are just easier to reason about and debug Right, isn't the fundamental trade-off between programmer efficieny and runtime efficiency? Immutable structures are easier to reason about, but almost always run slower than their mutable counterparts. The holy grail would be immutable structures that are equally performant, but I'm not sure that's even mathematically possible in general.
  - kephasp 4y ago
    Is there data supporting the notion that immutable structures are slower? There are many cases where there won't be a single difference in performance and there are a bunch of cases where immutable structures will be way faster than mutable ones.
    - philipkglass 4y ago
      The most obvious case I'm aware of is with numerical simulations in scientific computing. I worked on molecular simulations in graduate school. The core data structures are large (many-gigabytes) arrays, and almost every element of the array is updated on every iteration. Everything that requires more memory or even more memory accesses is going to be slower. In business applications you're usually not updating everything in a data structure all the time, so I can see how clever immutability may improve performance.
      - 4y ago
at_compile_time 4y ago
What are the advantages of complete immutability when we can write compilers that specifically prevent shared mutability? You're avoiding the dangerous quadrant but also excluding a valid one. I'd definitely prefer FP's solution over Python's GIL, but it still feels like an unnecessary compromise.
- assbuttbuttass 4y ago
  Most functional languages allow mutability when you need it, they just default to immutable data. Haskell might be the exception, where everything has to be immutable all the time
  - marcosdumay 4y ago
    Everything does not have to be immutable in Haskell. It's only that mutability can only exist in IO or some other context that deals with it. The language enforces purity, not immutability.
  - poorlyknit 4y ago
    This is not true, Haskell of course allows you to mutate state. It just forces you to declare it: For example, there's a MutableByteArray type whose operations only work in a specific Monad (because Haskell...) [1]. There's also more basic stuff like IORef which represents an assignable variable [2]. Again, you're constrained to use this in IO contexts. Etc. Haskell basically just forces you to do that thing someone else in this thread mentioned: Write functional APIs, allow side effects but make them explicit and confined to very specific places. [1]: https://hackage.haskell.org/package/primitive-0.7.4.0/docs/src/Data.Primitive.ByteArray.html#newByteArray https://hackage.haskell.org/package/primitive-0.7.4.0/docs/s... [2]: https://hackage.haskell.org/package/base-4.16.3.0/docs/Data-IORef.html https://hackage.haskell.org/package/base-4.16.3.0/docs/Data-...
badpun 4y ago
I've worked in two functional codebases so far and I'd say that in both cases the mutation of shared state was just moved to database and its transaction semantics. The FP code itself was about mostly stateless server-side processing. So, FP wasn't doing anything for us on that front. The main selling point for it was that it makes program more composable, more concise and easier to refactor (as you're effectively working with higher-level abstractions).
- lostcolony 4y ago
  >> in both cases the mutation of shared state was just moved to database and its transaction semantics. The FP code itself was about mostly stateless server-side processing. So, FP wasn't doing anything for us on that front. Sure it was. While it didn't force devs to stop using mutable state, you just called out that it -did- force them to move that state out of the service and into something with ACID guarantees, and write their server code in a way that was far more stateless. That sounds like a win to me?
  - yowlingcat 4y ago
    You can also do something like that by using a vanilla ORM and/or batteries include web framework (which includes that) in any popular language of your choice (Spring, Rails, Django, etc).
    - lostcolony 4y ago
      Sure, if you ensure that every bit of state is going to the ORM, and the ORM caches nothing. Of course, that tends to ensure that what is and what isn't side effect free is massively obfuscated, which seems like you're including some new negatives, but I will totally agree there are other ways to ensure state goes to the state store; I already did.
      - yowlingcat 4y ago
        For most standard web applications, it is true that the RDBMS stores all durable application state; it's also the case that you avoid ORM caching because it introduces more problems than it solves. Saying "that tends to ensure what is and isn't side effect free is massively obfuscated" really depends on the ORM; if it's an ORM that tries to make an RDMBS "quick" like an object, then I agree (this is something I hate about Django's ORM), but if it's a query builder style ORM then I disagree, as it basically creates a DSL that wraps around the SQL which is already purely functional. What I'm getting at is that we shouldn't conflate what should take credit for creating sanity in how applications persist state. I think that credit should go to the RDBMS because it exposes the power of the benefits you ascribe to FP, to any programming language that can bind to an RDBMS. If that is the case, what do you really need FP for in general purpose line of business software engineering?
bitL 4y ago
This resembles a situation when I got a new boss who once tried to write a 2-threaded app and got burnt, so from then onward he mandated to never use multiple threads (despite me using it safely on 65000+ threads before). A better answer would be "most devs aren't skilled enough to use many threads/mutable shared objects, so lets mandate immutable for basic devs so that they don't get burnt". But why would you prevent proven experts from operating in the dangerous quadrant? Does everything have to become dumb to make it safe?
- jerf 4y ago
  That dangerous quadrant has a track record of chewing up and spitting out "proven experts". I'd put your question in the same class of question as "Look, I know memory management in C is just dangerous, but do we have to make everything dumb to make it safe?" False dichotomy; the solution to everything isn't to "make it dumb", but the proven experts have a long track record of being chewed up and spit out and the answer isn't to tell them to just Git Gud either. We tried that. We tried that a lot. It doesn't work. The solutions to concurrency pretty much all involve staying out that quadrant as much as possible. Immutability isn't the only solution. Another is to confine all variables to one thread. Erlang, for instance, superficially uses immutability to achieve this, but a deeper reason it achieves it is that there is simply no way for a "thread" (what Erlang calls "process") to modify a variable that another thread can see; this is sufficient. (See what Elixir does.) Rust implements a super-rich system of variable ownership that allows you to implement the constraint that no variable can be unexpectedly just modified by an arbitrary thread in a far richer fashion than just hard-locking all memory to a particular owner thread. This is also sufficient. I program a lot in Go, and while it lacks compiler support for forcing you to stay out of that quadrant, my code runs at high concurrency rates precisely because I've learned how to stay out of there, and my code looks very Erlang-ish when you look at the concurrency patterns in play.
- Chris_Newton 4y ago
  new boss who once tried to write a 2-threaded app and got burnt, so from then onward he mandated to never use multiple threads An unfortunate overreaction, I agree. Does everything have to become dumb to make it safe? I don’t think trying to avoid shared mutable state is dumb. It’s deliberately choosing a safer default, just like not allowing all references to be null or not accepting the string "123" where an integer like 123 is needed. Sometimes it can be useful to have shared mutable state, or to make a reference nullable, or to accept different types of data and quietly reinterpret them the same way internally. But these are inherently more dangerous styles of programming — they create opportunities for mistakes that are otherwise systematically prevented — so it seems reasonable to encourage the safer alternatives unless there is a good reason not to use them, and to make the more dangerous choices the ones you have to make explicitly.
heavenlyblue 4y ago
Functional programming is representing the program state as a single expression, so that the compiler could do funky maths stuff in reordering things in it. It has little to do with immutability or functions per se.
dkersten 4y ago
I’ve been bitten by mutable data changing behind my back even without concurrency. It can happen in code that is too magical, eg by having various listeners trigger that aren’t explicit in code (for example “pre_save” hooks in an ORM I have to use in work). It can also happen in deeply heated call chains (my code calls code that calls code that calls code… that mutates data it was given access to at some prior point). The change is technically in your call stack, but it’s indirect and not necessarily related to your code. If it were immutable data, then some distant code that holds a reference can’t mutate it unless that data is explicitly marked as mutable. To me, immutable data is a guarantee that a variable keeps its value, unless I explicitly say it can change (eg by wrapping it in a mutable “box” whose content can be changed). This way, I always know whether I’m giving something write access or only read access. Technically this can be achieved without immutable types (eg constant parameters, pass by value or just documented API promises) but having immutable data as per of the language gives an extra layer of guarantee that’s harder to violate and therefore more predictable. That is why I like immutable by default, even in single threaded code.

DonaldPShimoda 4y ago

Just a small nitpick about your comment: > Functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadrant" chart. I might rather say "The recent focus on adapting functional programming is a reaction...". Functional programming itself is nearly as old as the modern computer (McCarthy's first LISP paper was published in 1960), but your phrasing kind of suggests that the entirety of FP is a recent innovation.

frostwarrior 4y ago
Functional programming itself is really old. Practical functional programming for scale is the new trend. What's new is the scale of software being developed right now. Developers simply can't afford to create mutable and shared state as if their program was just another desktop program, executed on a computer disconnected from the internet. Sooner or later we learn that focusing on the human value of code (I mean, making the code clear, concise and simple to read) pays off way better on the long run rather than optimizing for CPU cycles or memory footprint.

nine_k 4y ago

Your global variables were / are shared with yourself, even in no-concurrency case. With typical control flow being far from linear, it's easy to surprise yourself with unexpected state mutation.

Scarbutt 4y ago
Hence the value of immutability even for single-threaded code.
Jtsummers 4y ago
Giving me flashbacks to inherited codebases from old-school FORTRAN coders... // 500 variables shared by everything main loop "task" 1 "task" 2 "task" 3 ... "task" n So t121 is now supposed to only be used by task 3, but "t" as a prefix means it is temp. However, we don't want to change all the references. So just don't use it everyone else, please?
MaxBarraclough 4y ago
Right. Pure functions make it considerably easier to reason about code, even in the absence of concurrency. It's a shame D's pure modifier (to statically guarantee purity) isn't available in many other languages. [0] SPARK Ada has something similar in that functions must always be pure, forcing procedures to be used for impure functionality. [1] edit I think that's mistaken, SPARK appears to permit functions to read globals but not to modify them. edit John Carmack has written about this kind of thing. From [2]: > if a function only references a piece or two of global state, it is probably wise to consider passing it in as a variable. It would be kind of nice if C had a "functional" keyword to enforce no global references. [0] https://dlang.org/spec/function.html#pure-functions https://dlang.org/spec/function.html#pure-functions [1] https://learn.adacore.com/courses/intro-to-spark/chapters/01_Overview.html https://learn.adacore.com/courses/intro-to-spark/chapters/01... [2] https://cbarrete.com/carmack.html https://cbarrete.com/carmack.html

jahewson 4y ago

> Functional programming is a reaction from the shift to the dangerous quadrant of the immutable-mutable/unshared-shared "Magic Quadrant" chart. This makes it sound like you’re saying that’s why functional programming was created, which isn’t the case. I assume you mean that this is why it’s gained popularity but I don’t really agree with that - it’s certainly valuable but most functional programmers will never touch threads. I’d argue that programs without state are just easier to reason about and debug and for most software this is where the value comes from.

klipt 4y ago
> I’d argue that programs without state are just easier to reason about and debug Right, isn't the fundamental trade-off between programmer efficieny and runtime efficiency? Immutable structures are easier to reason about, but almost always run slower than their mutable counterparts. The holy grail would be immutable structures that are equally performant, but I'm not sure that's even mathematically possible in general.
- kephasp 4y ago
  Is there data supporting the notion that immutable structures are slower? There are many cases where there won't be a single difference in performance and there are a bunch of cases where immutable structures will be way faster than mutable ones.
  - philipkglass 4y ago
    The most obvious case I'm aware of is with numerical simulations in scientific computing. I worked on molecular simulations in graduate school. The core data structures are large (many-gigabytes) arrays, and almost every element of the array is updated on every iteration. Everything that requires more memory or even more memory accesses is going to be slower. In business applications you're usually not updating everything in a data structure all the time, so I can see how clever immutability may improve performance.
    - 4y ago

at_compile_time 4y ago

What are the advantages of complete immutability when we can write compilers that specifically prevent shared mutability? You're avoiding the dangerous quadrant but also excluding a valid one. I'd definitely prefer FP's solution over Python's GIL, but it still feels like an unnecessary compromise.

assbuttbuttass 4y ago
Most functional languages allow mutability when you need it, they just default to immutable data. Haskell might be the exception, where everything has to be immutable all the time
- marcosdumay 4y ago
  Everything does not have to be immutable in Haskell. It's only that mutability can only exist in IO or some other context that deals with it. The language enforces purity, not immutability.
- poorlyknit 4y ago
  This is not true, Haskell of course allows you to mutate state. It just forces you to declare it: For example, there's a MutableByteArray type whose operations only work in a specific Monad (because Haskell...) [1]. There's also more basic stuff like IORef which represents an assignable variable [2]. Again, you're constrained to use this in IO contexts. Etc. Haskell basically just forces you to do that thing someone else in this thread mentioned: Write functional APIs, allow side effects but make them explicit and confined to very specific places. [1]: https://hackage.haskell.org/package/primitive-0.7.4.0/docs/src/Data.Primitive.ByteArray.html#newByteArray https://hackage.haskell.org/package/primitive-0.7.4.0/docs/s... [2]: https://hackage.haskell.org/package/base-4.16.3.0/docs/Data-IORef.html https://hackage.haskell.org/package/base-4.16.3.0/docs/Data-...

badpun 4y ago

I've worked in two functional codebases so far and I'd say that in both cases the mutation of shared state was just moved to database and its transaction semantics. The FP code itself was about mostly stateless server-side processing. So, FP wasn't doing anything for us on that front. The main selling point for it was that it makes program more composable, more concise and easier to refactor (as you're effectively working with higher-level abstractions).

lostcolony 4y ago
>> in both cases the mutation of shared state was just moved to database and its transaction semantics. The FP code itself was about mostly stateless server-side processing. So, FP wasn't doing anything for us on that front. Sure it was. While it didn't force devs to stop using mutable state, you just called out that it -did- force them to move that state out of the service and into something with ACID guarantees, and write their server code in a way that was far more stateless. That sounds like a win to me?
- yowlingcat 4y ago
  You can also do something like that by using a vanilla ORM and/or batteries include web framework (which includes that) in any popular language of your choice (Spring, Rails, Django, etc).
  - lostcolony 4y ago
    Sure, if you ensure that every bit of state is going to the ORM, and the ORM caches nothing. Of course, that tends to ensure that what is and what isn't side effect free is massively obfuscated, which seems like you're including some new negatives, but I will totally agree there are other ways to ensure state goes to the state store; I already did.
    - yowlingcat 4y ago
      For most standard web applications, it is true that the RDBMS stores all durable application state; it's also the case that you avoid ORM caching because it introduces more problems than it solves. Saying "that tends to ensure what is and isn't side effect free is massively obfuscated" really depends on the ORM; if it's an ORM that tries to make an RDMBS "quick" like an object, then I agree (this is something I hate about Django's ORM), but if it's a query builder style ORM then I disagree, as it basically creates a DSL that wraps around the SQL which is already purely functional. What I'm getting at is that we shouldn't conflate what should take credit for creating sanity in how applications persist state. I think that credit should go to the RDBMS because it exposes the power of the benefits you ascribe to FP, to any programming language that can bind to an RDBMS. If that is the case, what do you really need FP for in general purpose line of business software engineering?

bitL 4y ago

This resembles a situation when I got a new boss who once tried to write a 2-threaded app and got burnt, so from then onward he mandated to never use multiple threads (despite me using it safely on 65000+ threads before). A better answer would be "most devs aren't skilled enough to use many threads/mutable shared objects, so lets mandate immutable for basic devs so that they don't get burnt". But why would you prevent proven experts from operating in the dangerous quadrant? Does everything have to become dumb to make it safe?

jerf 4y ago
That dangerous quadrant has a track record of chewing up and spitting out "proven experts". I'd put your question in the same class of question as "Look, I know memory management in C is just dangerous, but do we have to make everything dumb to make it safe?" False dichotomy; the solution to everything isn't to "make it dumb", but the proven experts have a long track record of being chewed up and spit out and the answer isn't to tell them to just Git Gud either. We tried that. We tried that a lot. It doesn't work. The solutions to concurrency pretty much all involve staying out that quadrant as much as possible. Immutability isn't the only solution. Another is to confine all variables to one thread. Erlang, for instance, superficially uses immutability to achieve this, but a deeper reason it achieves it is that there is simply no way for a "thread" (what Erlang calls "process") to modify a variable that another thread can see; this is sufficient. (See what Elixir does.) Rust implements a super-rich system of variable ownership that allows you to implement the constraint that no variable can be unexpectedly just modified by an arbitrary thread in a far richer fashion than just hard-locking all memory to a particular owner thread. This is also sufficient. I program a lot in Go, and while it lacks compiler support for forcing you to stay out of that quadrant, my code runs at high concurrency rates precisely because I've learned how to stay out of there, and my code looks very Erlang-ish when you look at the concurrency patterns in play.
Chris_Newton 4y ago
new boss who once tried to write a 2-threaded app and got burnt, so from then onward he mandated to never use multiple threads An unfortunate overreaction, I agree. Does everything have to become dumb to make it safe? I don’t think trying to avoid shared mutable state is dumb. It’s deliberately choosing a safer default, just like not allowing all references to be null or not accepting the string "123" where an integer like 123 is needed. Sometimes it can be useful to have shared mutable state, or to make a reference nullable, or to accept different types of data and quietly reinterpret them the same way internally. But these are inherently more dangerous styles of programming — they create opportunities for mistakes that are otherwise systematically prevented — so it seems reasonable to encourage the safer alternatives unless there is a good reason not to use them, and to make the more dangerous choices the ones you have to make explicitly.

heavenlyblue 4y ago

Functional programming is representing the program state as a single expression, so that the compiler could do funky maths stuff in reordering things in it. It has little to do with immutability or functions per se.

dkersten 4y ago

I’ve been bitten by mutable data changing behind my back even without concurrency. It can happen in code that is too magical, eg by having various listeners trigger that aren’t explicit in code (for example “pre_save” hooks in an ORM I have to use in work). It can also happen in deeply heated call chains (my code calls code that calls code that calls code… that mutates data it was given access to at some prior point). The change is technically in your call stack, but it’s indirect and not necessarily related to your code. If it were immutable data, then some distant code that holds a reference can’t mutate it unless that data is explicitly marked as mutable. To me, immutable data is a guarantee that a variable keeps its value, unless I explicitly say it can change (eg by wrapping it in a mutable “box” whose content can be changed). This way, I always know whether I’m giving something write access or only read access. Technically this can be achieved without immutable types (eg constant parameters, pass by value or just documented API promises) but having immutable data as per of the language gives an extra layer of guarantee that’s harder to violate and therefore more predictable. That is why I like immutable by default, even in single threaded code.