10 ms·

Functional core, imperative shell (2012)

389 points by fagnerbrack 8y ago

savethefuture 8y ago
This is one of my favorite videos. I highly recommend DAS videos, definitely worth the subscription.
- mercer 8y ago
  Seconded! This one and a few others gave me 'aha moments' that I don't get much without hunting for the right videos (recommendations much appreciated! In the vein of Simple Made Easy).
  - squirrelicus 8y ago
    Simple Made Easy should be mandatory watching for your developer's license. Seriously though, this talk, Boundaries, and Simple Made Easy are the trifecta that forms the foundation of the software I design.
Twisol 8y ago
This basic idea has deeply infected how I approach software architecture and design. When someone asks me why I wrote something a particular way, it's usually because I'm trying to keep the imperative/environmental concerns separate from the functional/algorithmic concerns. I end up recommending this video at least once every year. (EDIT: Another one from Gary that pairs well is "Boundaries": https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries . The core idea is that values -- data, not code -- should form the boundaries between distinct systems.)
- monocularvision 8y ago
  The entire architecture of our iOS app is built around this principle. We have tried to give it a name but keep coming back to “Functional Core, Imperative Shell”. You can find some of the ideas that were implemented here: https://m.youtube.com/watch?v=7AGQ9dhWCX0 https://m.youtube.com/watch?v=7AGQ9dhWCX0
  - tylerc230 8y ago
    Yeah swift lends itself to this style well. Value types, functional aspects etc.
  - yoava 8y ago
    What you seek is actually something a bit different. It is the observation that all software is built of 3 layers - Inbound IO, what you call imperative shell Business logic core, what you call functional core Outbound IO, what you call again imperative shell. The problem with the terms in functional programming is that when they say side effects, in most cases they mean IO. The functional core does not have to be functional for you to get the benefits - easy testing, easy to reason about, easy to develop. In fact, in some cases, functional programing is the wrong tool while having a business logic core that is separated from IO is still a very valid architecture.
    - nicoburns 8y ago
      Generally speaking, I find that it is making the "business core" stateless that has the benefits.
    - tasuki 8y ago
      In which cases is functional programming the wrong tool for the business logic core?
- cellularmitosis 8y ago
  This is another talk which pairs well with it (values vs. objects, and how to split your code up accordingly): https://academy.realm.io/posts/andy-matuschak-controlling-complexity/ https://academy.realm.io/posts/andy-matuschak-controlling-co...
- ttflee 8y ago
  And his visionary talk on JavaScript, asm.js and others: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
- fpoling 8y ago
  The actor model is prized in the talk. Yet actor model is not functional. The problem is the message queue of each actor. It is an imperative storage and one can get all nasty problems of imperative code from apparently functional code patterns.
  - msangi 8y ago
    What would a functional approach be?
    - yoava 8y ago
      Function composition
    - fpoling 8y ago
      It depends on what is the goal. To take advantage of multicore for computations one can use explicitly parallel functional algorithms. To improve response under load one can do load balancing within application and schedule at the very top (shell level) several independent functional pipelines.
  - yoava 8y ago
    Actor model is even worse, you have no gerentee of a response or error status
  - shrimpx 8y ago
    There aren't that many ways you can mess up a message queue.
  - ramchip 8y ago
    You can build every actor like a miniature program, with its own imperative shell and functional core. In Erlang it's common to have various message handlers that are very imperative, but defer all complicated logic to pure functions that take part of the actor state and return a transformed state.
    - mercer 8y ago
bcbrown 8y ago
One of the things I really like about this approach is how well it lends itself to separating what needs unit testing from what is unsuitable to unit testing and should instead be validated by higher-level end-to-end/functional testing. I think it's also related to the concept of building a DSL in which to implement business requirements. Once you have the right 'primitives' you can then combine them in useful ways that are easy to verify (by reading the code) that the implementation matches the requirements.
- pwm 8y ago
  Exactly. Also once all what you have in core are types and functions then things like mocking becomes obsolete. You can just instantiate those types for real.
- daenz 8y ago
  Bingo. My philosophy has started to become "If it can be functional without sub-optimal overhead or confusing state magic, it probably should be." Delaying that "DSL" layer as long as possible keeps your system super straightforward and easily testable.
- MichaelMoser123 8y ago
  How do you deal with performance problems? I mean to add an element to a list you need to create a copy of the list and then add the new element, all in order to remain functional. Isn't that very expensive?
  - adrianN 8y ago
    Purely functional data structures are more efficient than the naive "just copy everything" approach. You can do anything that's possible in a stateful imperative setting with at most a logarithmic slowdown.
    - MichaelMoser123 8y ago
      Let's say the entries of the original list and the new one point to the same objects, still for a list of a thousand entries you need to copy all the link entries to add one on top of it. What am I missing?
ryanmarsh 8y ago
Great to see this. I ended up writing this way after watching Brian Will's Why Object Oriented Programming is Bad [0]. Near the end he states we should be writing "procedural code" but that we should favor "pure functions". What I took away from this was that at a high level I should be able to reason about a problem procedurally but the solution should be composed of mostly pure functions. 0: https://www.youtube.com/watch?v=QM1iUe6IofM https://www.youtube.com/watch?v=QM1iUe6IofM
T-R 8y ago
This concept is a big part of what the "big deal" around monads is - using monads to model effectful code conveniently puts the information of "this should be shell code" into the type, in a way that ensures that code that calls it also gets annotated as "shell" code. Monads are of course also a much more broadly applicable abstraction, but their application to effectful code, enforcing this design, is usually the first and most apparent place people run into them in the ML family of languages.
- dwohnitmok 8y ago
  I disagree, although it's possible I only disagree with how you've phrased it. Monadic interfaces in the context of non-deterministic effects are a consolation prize. They represent a way to combine effectful code, but ideally your code would have almost no effects at all. As far as I can tell, the idealized version of this talk is a batch interface: one effect to grab all the data you need, transform the data, and then one effect to "flush" the transformed data (where flush could mean to persist the data in a database, send it out as commands to control a robot, etc). Tracking side effects in your types (maybe what you were going for?) is helpful for measuring to what degree your code fails to adhere to this idealized model. If most of your code has an effect type, that's probably a sign to refactor. It also keeps you honest as to the infectious nature of effectful code by propagating the type as necessary.
  - T-R 8y ago
    I don't think we disagree in spirit - I didn't mean to imply that it prevents you from, e.g., writing all of your code in the IO monad, just the points you made in your last paragraph. So, more that they're a useful tool to help you realize these goals, not something that gets you there on its own. It does let you broaden/specify your definition of "effectful" a bit - modelling event streams with monads gives you FRP (as in your robot example), and I vaguely remember reading in some paper somewhere the suggestion of using monads to separate out unconstrained recursion/non-totality/co-data from total code.
- pankajdoharey 8y ago
  Monads is taking it too far. Mutation is a reality, the correct approach is disciplined mutation. Shoving mutation into convenient boxes and convincing yourself to never look inside it does not mean mutation does not exist. The best approach is taken by scheme, and more specifically clojure to have a disciplined and practical approach. Mathematical purity of programs is a myth propagated by Type theorists dont buy into it.
lackbeard 8y ago
Does anyone know of an example of a non-trivial codebase written in this style? How do you do this cleanly when, e.g., you need to make a network call and then based on what the result returned to you is, either do something with a local database, make a different network call, or return a result to your user. Also, error handling... It seems to me like monads must be the logical conclusion to this style of programming, or else you wind up with a mess (or just abandoning this technique.)
- _greim_ 8y ago
  The Elm architecture seems to use pretty much this exact pattern. The imperative shell parts are all buried within the Elm runtime, and you contribute the functional bits. Redux also to a lesser extent.
  - arwhatever 8y ago
    Have been using Elm architecture for a while, and also have pushed my code toward an immutable core/mutable shell architecture since watching Vladimir Khorikov's video on the subject at Pluralsight, but had never considered how the two are interrelated. Thank you for the big light bulb lighting up in my head just now.
- dnautics 8y ago
  Basically anything written in erlang or elixir follows this paradigm. It's really extremely productive to code this way... I wrote a job scheduler from scratch in 3 months and never once had to write or use a mutex or semaphore. Immutability makes you very confident about your code. Similarly, my UI guy wrote a UI in basically functional react. It's amazing. With very little js experience I made code patches that.. just worked because I was guaranteed that no function calls had mysterious side effects...
  - lackbeard 8y ago
    Know of any good open-source examples?
mlthoughts2018 8y ago
I like to imagine that the huge planet-sized villain from the film The Fifth Element is the physical manifestation of giving one’s self over fully to the programming style of indisciminantly mixing side-effectful and I/O operations into arbitrary code whenever it’s locally convenient to do so.
ww520 8y ago
It's similar to what usually do. Use functional style in inner, smaller, library level code. At the global level, it's unavoidable to deal with state when interfacing with outside. State changes just need to be well encapsulated and managed.
grzm 8y ago
(2012)
dwohnitmok 8y ago
Although it means something different and is not the the actual semantic opposite, I've found the syntactically opposite phrase "Functional shell, imperative core" also useful, which the speaker alludes to when he talks about local variable rebinding that is invisible to the outside. That is locally scoped, mutation-heavy, imperative code is fine as long as you can wrap it in a deterministic, immutable interface for general use. This is the premise behind things like Haskell's ST type. More generally it's the usual way FP languages try to recover competitive performance.
- taeric 8y ago
  It is also how your computer does basic addition. It isn't like there is such a thing as immutable numbers when it gets to the silicon. Look at how a barrel shifter is implemented someday, or a comparator.
  - branislav 8y ago
    It's also how something like React.js works. Wrapping stateful DOM manipulation in a functional shell, made performant thanks to the Virtual DOM diffing algorithm.
- JBiserkov 8y ago
  >locally scoped, mutation-heavy, imperative code is fine as long as you can wrap it in a deterministic, immutable interface for general use. Yes! Clojure supports this via transients. What's even better is that you can write the loop using immutable data, make sure it works. Then just 1. add a call to transient in the initializations, 2. add a call to persistent! in the end and 3. add a ! to each function that modifies the transient, e.g. conj becomes conj! The benefit in doing it this way is that the language will catch and flag as errors any "mixed" usage - calling a pure function with a mutable argument or vice-versa. https://clojure.org/reference/transients https://clojure.org/reference/transients http://www.hypirion.com/musings/understanding-clojure-transients http://www.hypirion.com/musings/understanding-clojure-transi...
ridiculous_fish 8y ago
How do you implement progress reporting and cancellation in this model?
- mvc 8y ago
  Make it an explicit part of the data model.
  - ridiculous_fish 8y ago
    A quicksort that supported progress reporting and cancellation might have a function parameter that it calls periodically to report progress, with the function returning a bool indicating cancellation. But this is using code, not data. How might this be implemented within the data model?
    - yen223 8y ago
      How do you measure progress in a quicksort?
      - ridiculous_fish 8y ago
        Glad you asked. The progress should be a bound on the worst case time. If you have N elements, then the initial worst case time is N^2. Say after the first partition, you are left with pieces N/3 and 2N/3; the worst case time is now (N/3)^2 + (2N/3)^2. Your progress after the first partition is the difference between the original worst case and the new worst case. This can make for uneven progress advancement but it’s monotonic: the progress bar will never go backwards.
    - jhomedall 8y ago
      You would implement the sort so that it either: 1: Returns the sorted list (once finished) 2: Returns an intermediate state containing the progress of the sort in addition to the current state of the sort. You would repeatedly pass the intermediate state to the sorting function until finished. You can use the progress component of the intermediate state to track progress. Something like the following (I don't actually write Haskell, so this could probably be represented better): data SomeIntermediateState = ... data Progress = Double data SortState a = Complete [a] | InProgress (SomeIntermediateState, Progress) sortInit :: [a] -> SortState sort :: SortState -> SortState
codetrotter 8y ago
While the purpose of this screencast is to explain a principle, not to focus on the particular piece of software that was implemented, I am left wondering about a couple of things; The timeline as he explains it, is updated by creating a new timeline instance which consists of previous timeline + new tweets. It seems to me then, that his client does not remove from the timeline tweets that were deleted by its author subsequently to having been downloaded. To some it might be a feature to capture as many tweets as possible, but at the same time, if your view of the timeline includes deleted tweets then you might find yourself trying to reply to a deleted tweet, and then you would get an error response from Twitter when you try to post your reply. (Though I don’t know if his client also does posting, or if it’s a read-only client.) Furthermore, what about tweets that are edited by their author subsequently to having been downloaded? Seems that you would not see those edits.
- gary_bernhardt 8y ago
  Tweets aren't editable so no need to handle that. Deleted tweets can be honored by updating the timeline to be `old_timeline - deleted_tweets + new_tweets`.
danidiaz 8y ago
In the Idris language, one best practice is to relegate to the outer shell not only IO effects, but also "partiality"—in the sense of computations that might enter an unproductive infinite loop. (In Haskell, one way of getting into unproductive infinite loops is by mistakenly asking "parse elements until the first failure" to a parser combinator that can always succeed without consuming input.) The ideal is that the core of the application should be both pure and total, purity and totality being tracked by the type system. In fact, one can often relegate partiality to a single function of the outer shell.
draw_down 8y ago
Love this one! I think Rich Hickey’s “The Value of Values” pairs nicely with it.
squirrelicus 8y ago
It's difficult to overstate the importance of the principles taught in this talk. Almost as difficult as it is to describe why they are so important. The fundamentals of quality software are found here, not in some Martin Fowler dissertation on DDD. Edit: correction from commenter, thanks!
- yawaramin 8y ago
  overstate
- stevebmark 8y ago
  I think the general concept of using immutability, queues, and isolating side effects, is lost and muddied by using Ruby, and it's a misuse of the term "functional."
squirrelicus 8y ago
Often people will ask how to follow these principles in the world of I/O. The answer is incredibly simple: with values. Return a DTO that encapsulates all the business information about the result state of the I/O call. Do not throw exceptions. Log them if you like, but you must return all the data necessary for the business logic to react to failure conditions, encoded in your own use case specific structure. If you don't care about whether you failed due to timeouts or refused connections or query parse failures what have you, don't return that data, just return a hasError kind of property on the return structure (bonus points if your language supports discriminated unions, but this is not necessary). If the parent logic needs to react to timeouts and failures to connect differently, then catch those exceptions or state separately and return didTimeout or cantConnect flags separately. Values values values
- the_gipsy 8y ago
  I would prefer a ‘Result<ValueType, ErrorType>’ over stuffing weird ‘hasError’, ‘didTimeout’, etc flags everywhere. It’s much safer to type-check at compile time that you have discriminated between result and error, than to hope that you are ‘if’-checking some flags at the right time at runtime.
  - squirrelicus 8y ago
    That is another way to accomplish the same goal. This trade-off is more aesthetic than architectural and is just a matter of language support
GolDDranks 8y ago
This is the pattern I'm striving to code in, but when developing with a team, I keep struggling convincing other people (with OOP, mutability-happy mindsets) to do so...
jypepin 8y ago
And I'd strongly recommend paying for DAS and looking at every.single.video in there. Gary is brilliant and each video explains stuff in a very simple, enjoyable way. Reminds me a bit of Ryan Bate's ruby/rails videos.
abenedic 8y ago
So, I am a person who wishes there was a proper article I could read for this. I don't know how much this matters to people, but I can understand written English far far better than I can understand spoken English. I know auto-translate works for some. I am trying with applet and get nothing. I like core idea. I feel often people outside English land get second class experience.
- godot 8y ago
  You're not alone. I immigrated to the US from Asia in the 90s. For many years I didn't understand spoken English fully. Arguably I still miss some words now and then, even nowadays, after over 20 years living in an English environment. I had great grades throughout high school and college here mostly thanks to having excellent reading and writing skills in English, since I came from a place in Asia that emphasized learning English since childhood. Listening to spoken English is not easy for non native speakers.
  - abenedic 8y ago
    > Listening to spoken English is not easy for non native speakers Thank you for saying this. You clearly have a good understanding of English(far better than mine at least). I feel like this is an area that is glossed over. For every article that is written in english there are at least 10+ non-native speaker struggling to understand the work, who could extract something useful or help explicate the work.
    - theoh 8y ago
      In the case of this video, you aren't missing much. It's deliberately 'dense', according to the author, but the concept is simple. Almost any app will involve some imperative code, but using pure functions within that, where possible, makes it easier to reason about what is going on. That's all there is to it. https://en.wikipedia.org/wiki/Referential_transparency https://en.wikipedia.org/wiki/Referential_transparency
- pfranz 8y ago
  I've followed Gary's stuff for a bunch of years now. His preferred medium seems to be screencasts and the occasional conference talk. His blog dating back to 2011 has 10 posts (he does seem to tweet often, though). His screencasts are very thoughtfully done and dense. So I imagine you'd have to find that article from someone else. As a native English speaker, I generally have the same preference you do. Articles I can skim, they're easier to refer back to, and I can search. I hope you can take solace that I will sometimes search for an error message and the only result is a forum posting in a foreign language that Google Translate barfs on.
blt 8y ago
I'm struggling to apply this model to code for machine learning experiments. The functional core takes an extremely long time to compute, e.g. "train the model". Thus, you want to save lots of intermediate values to disk so you can reload them over and over while you are experimenting the downstream part of the program. I end up with a lot of code writing values to the disk, which is currently mixed in with the computations. I'm wondering if there's some way to automate this "save intermediate values to the disk" so I can write the code in a more functional style without having to constantly go in and out of the imperative shell portion. The process of ML experimentation is extremely painful compared to normal software development where nothing really takes that long to compute.
- waluigi 8y ago
  The way I would approach this is to write what essentially amounts to a declarative specification of what computation needs to be run, and then define an interpreter that handles the caching of intermediate values. The "Embedded DSL + Interpreter" pattern is incredibly powerful, and it's nice to see it catching on more.
ceronman 8y ago
This is the same principle of the Clean Architecture. I like how Brandon Rhodes explain how to apply it in Python: https://www.youtube.com/watch?v=DJtef410XaM https://www.youtube.com/watch?v=DJtef410XaM
fpoling 8y ago
One does not need a pure functional core to have benefits of such design. The key idea is that the core code should not change or even read the global state. It should only be allowed to access or change what is explicitly passed to it. For example, to avoid excessive memory consumption to sum matrixes one wants to code it like A += B, not like C = A + B. Yet one still benefits from all the testing, design etc. benefits of pure functional code. At the end one call always get a functional version just by turning A += B into C = A; C += B.
Flenser 8y ago
After watching this and his Boundaries talk, and then reading the Out of the Tar Pit paper, I started watching Pete Hunt's first talks on why they built React and had an epiphany why React was going to end up being the most popular front end framework.
jcyw 8y ago
This is the github repo featured in the video: https://github.com/bf4/toot https://github.com/bf4/toot
jcyw 8y ago
The idea of functional core is cool. I find domain-driven design a better guide for implementation. In a nut shell, there are three types of objects: value, entity, and service. Both value and entity are functional and stateless objects. A service object maps between stateless objects and may cross the domain boundary (such as network, IO, etc). When I do this in java, a service object would have interfaces returning CompletableFuture of Value or Entity objects. InfoQ has a very nice summary of DDD book: https://www.infoq.com/minibooks/download/domain-driven-design-quickly https://www.infoq.com/minibooks/download/domain-driven-desig...
Apaec 8y ago
I learned this through Haskell, it enforces the "functional core, imperative shell" approach through the IO monad. Now I find myself repeating the same pattern in other languages(js, java).
stevebmark 8y ago
"Functional core" littered with classes, attr_readers, data that's not data (Curosr.new) and += imperative loops. Yikes.
- gary_bernhardt 8y ago
  The classes serve to close some functions over variables for convenience; the attr_readers serve as destructuring functions for convenience; the += replaces a recursive function for convenience and familiarity. The function with the += doesn't mutate any value and it remains pure from all callers' perspectives. You're complaining about syntax, but this screencast is about semantics.
  - stevebmark 8y ago
    No, something with "functional" in the title that doesn't use functions isn't related to syntax.
    - gary_bernhardt 8y ago
      What is a function?

389 points by fagnerbrack 8y ago

savethefuture 8y ago

This is one of my favorite videos. I highly recommend DAS videos, definitely worth the subscription.

mercer 8y ago
Seconded! This one and a few others gave me 'aha moments' that I don't get much without hunting for the right videos (recommendations much appreciated! In the vein of Simple Made Easy).
- squirrelicus 8y ago
  Simple Made Easy should be mandatory watching for your developer's license. Seriously though, this talk, Boundaries, and Simple Made Easy are the trifecta that forms the foundation of the software I design.

Twisol 8y ago

This basic idea has deeply infected how I approach software architecture and design. When someone asks me why I wrote something a particular way, it's usually because I'm trying to keep the imperative/environmental concerns separate from the functional/algorithmic concerns. I end up recommending this video at least once every year. (EDIT: Another one from Gary that pairs well is "Boundaries": https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries . The core idea is that values -- data, not code -- should form the boundaries between distinct systems.)

monocularvision 8y ago
The entire architecture of our iOS app is built around this principle. We have tried to give it a name but keep coming back to “Functional Core, Imperative Shell”. You can find some of the ideas that were implemented here: https://m.youtube.com/watch?v=7AGQ9dhWCX0 https://m.youtube.com/watch?v=7AGQ9dhWCX0
- tylerc230 8y ago
  Yeah swift lends itself to this style well. Value types, functional aspects etc.
- yoava 8y ago
  What you seek is actually something a bit different. It is the observation that all software is built of 3 layers - Inbound IO, what you call imperative shell Business logic core, what you call functional core Outbound IO, what you call again imperative shell. The problem with the terms in functional programming is that when they say side effects, in most cases they mean IO. The functional core does not have to be functional for you to get the benefits - easy testing, easy to reason about, easy to develop. In fact, in some cases, functional programing is the wrong tool while having a business logic core that is separated from IO is still a very valid architecture.
  - nicoburns 8y ago
    Generally speaking, I find that it is making the "business core" stateless that has the benefits.
  - tasuki 8y ago
    In which cases is functional programming the wrong tool for the business logic core?
cellularmitosis 8y ago
This is another talk which pairs well with it (values vs. objects, and how to split your code up accordingly): https://academy.realm.io/posts/andy-matuschak-controlling-complexity/ https://academy.realm.io/posts/andy-matuschak-controlling-co...
ttflee 8y ago
And his visionary talk on JavaScript, asm.js and others: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
fpoling 8y ago
The actor model is prized in the talk. Yet actor model is not functional. The problem is the message queue of each actor. It is an imperative storage and one can get all nasty problems of imperative code from apparently functional code patterns.
- msangi 8y ago
  What would a functional approach be?
  - yoava 8y ago
    Function composition
  - fpoling 8y ago
    It depends on what is the goal. To take advantage of multicore for computations one can use explicitly parallel functional algorithms. To improve response under load one can do load balancing within application and schedule at the very top (shell level) several independent functional pipelines.
- yoava 8y ago
  Actor model is even worse, you have no gerentee of a response or error status
- shrimpx 8y ago
  There aren't that many ways you can mess up a message queue.
- ramchip 8y ago
  You can build every actor like a miniature program, with its own imperative shell and functional core. In Erlang it's common to have various message handlers that are very imperative, but defer all complicated logic to pure functions that take part of the actor state and return a transformed state.
  - mercer 8y ago

bcbrown 8y ago

One of the things I really like about this approach is how well it lends itself to separating what needs unit testing from what is unsuitable to unit testing and should instead be validated by higher-level end-to-end/functional testing. I think it's also related to the concept of building a DSL in which to implement business requirements. Once you have the right 'primitives' you can then combine them in useful ways that are easy to verify (by reading the code) that the implementation matches the requirements.

pwm 8y ago
Exactly. Also once all what you have in core are types and functions then things like mocking becomes obsolete. You can just instantiate those types for real.
daenz 8y ago
Bingo. My philosophy has started to become "If it can be functional without sub-optimal overhead or confusing state magic, it probably should be." Delaying that "DSL" layer as long as possible keeps your system super straightforward and easily testable.
MichaelMoser123 8y ago
How do you deal with performance problems? I mean to add an element to a list you need to create a copy of the list and then add the new element, all in order to remain functional. Isn't that very expensive?
- adrianN 8y ago
  Purely functional data structures are more efficient than the naive "just copy everything" approach. You can do anything that's possible in a stateful imperative setting with at most a logarithmic slowdown.
  - MichaelMoser123 8y ago
    Let's say the entries of the original list and the new one point to the same objects, still for a list of a thousand entries you need to copy all the link entries to add one on top of it. What am I missing?

ryanmarsh 8y ago

Great to see this. I ended up writing this way after watching Brian Will's Why Object Oriented Programming is Bad [0]. Near the end he states we should be writing "procedural code" but that we should favor "pure functions". What I took away from this was that at a high level I should be able to reason about a problem procedurally but the solution should be composed of mostly pure functions. 0: https://www.youtube.com/watch?v=QM1iUe6IofM https://www.youtube.com/watch?v=QM1iUe6IofM

T-R 8y ago

This concept is a big part of what the "big deal" around monads is - using monads to model effectful code conveniently puts the information of "this should be shell code" into the type, in a way that ensures that code that calls it also gets annotated as "shell" code. Monads are of course also a much more broadly applicable abstraction, but their application to effectful code, enforcing this design, is usually the first and most apparent place people run into them in the ML family of languages.

dwohnitmok 8y ago
I disagree, although it's possible I only disagree with how you've phrased it. Monadic interfaces in the context of non-deterministic effects are a consolation prize. They represent a way to combine effectful code, but ideally your code would have almost no effects at all. As far as I can tell, the idealized version of this talk is a batch interface: one effect to grab all the data you need, transform the data, and then one effect to "flush" the transformed data (where flush could mean to persist the data in a database, send it out as commands to control a robot, etc). Tracking side effects in your types (maybe what you were going for?) is helpful for measuring to what degree your code fails to adhere to this idealized model. If most of your code has an effect type, that's probably a sign to refactor. It also keeps you honest as to the infectious nature of effectful code by propagating the type as necessary.
- T-R 8y ago
  I don't think we disagree in spirit - I didn't mean to imply that it prevents you from, e.g., writing all of your code in the IO monad, just the points you made in your last paragraph. So, more that they're a useful tool to help you realize these goals, not something that gets you there on its own. It does let you broaden/specify your definition of "effectful" a bit - modelling event streams with monads gives you FRP (as in your robot example), and I vaguely remember reading in some paper somewhere the suggestion of using monads to separate out unconstrained recursion/non-totality/co-data from total code.
pankajdoharey 8y ago
Monads is taking it too far. Mutation is a reality, the correct approach is disciplined mutation. Shoving mutation into convenient boxes and convincing yourself to never look inside it does not mean mutation does not exist. The best approach is taken by scheme, and more specifically clojure to have a disciplined and practical approach. Mathematical purity of programs is a myth propagated by Type theorists dont buy into it.

lackbeard 8y ago

Does anyone know of an example of a non-trivial codebase written in this style? How do you do this cleanly when, e.g., you need to make a network call and then based on what the result returned to you is, either do something with a local database, make a different network call, or return a result to your user. Also, error handling... It seems to me like monads must be the logical conclusion to this style of programming, or else you wind up with a mess (or just abandoning this technique.)

_greim_ 8y ago
The Elm architecture seems to use pretty much this exact pattern. The imperative shell parts are all buried within the Elm runtime, and you contribute the functional bits. Redux also to a lesser extent.
- arwhatever 8y ago
  Have been using Elm architecture for a while, and also have pushed my code toward an immutable core/mutable shell architecture since watching Vladimir Khorikov's video on the subject at Pluralsight, but had never considered how the two are interrelated. Thank you for the big light bulb lighting up in my head just now.
dnautics 8y ago
Basically anything written in erlang or elixir follows this paradigm. It's really extremely productive to code this way... I wrote a job scheduler from scratch in 3 months and never once had to write or use a mutex or semaphore. Immutability makes you very confident about your code. Similarly, my UI guy wrote a UI in basically functional react. It's amazing. With very little js experience I made code patches that.. just worked because I was guaranteed that no function calls had mysterious side effects...
- lackbeard 8y ago
  Know of any good open-source examples?

mlthoughts2018 8y ago

I like to imagine that the huge planet-sized villain from the film The Fifth Element is the physical manifestation of giving one’s self over fully to the programming style of indisciminantly mixing side-effectful and I/O operations into arbitrary code whenever it’s locally convenient to do so.

ww520 8y ago

It's similar to what usually do. Use functional style in inner, smaller, library level code. At the global level, it's unavoidable to deal with state when interfacing with outside. State changes just need to be well encapsulated and managed.

grzm 8y ago

(2012)

dwohnitmok 8y ago

Although it means something different and is not the the actual semantic opposite, I've found the syntactically opposite phrase "Functional shell, imperative core" also useful, which the speaker alludes to when he talks about local variable rebinding that is invisible to the outside. That is locally scoped, mutation-heavy, imperative code is fine as long as you can wrap it in a deterministic, immutable interface for general use. This is the premise behind things like Haskell's ST type. More generally it's the usual way FP languages try to recover competitive performance.

taeric 8y ago
It is also how your computer does basic addition. It isn't like there is such a thing as immutable numbers when it gets to the silicon. Look at how a barrel shifter is implemented someday, or a comparator.
- branislav 8y ago
  It's also how something like React.js works. Wrapping stateful DOM manipulation in a functional shell, made performant thanks to the Virtual DOM diffing algorithm.
JBiserkov 8y ago
>locally scoped, mutation-heavy, imperative code is fine as long as you can wrap it in a deterministic, immutable interface for general use. Yes! Clojure supports this via transients. What's even better is that you can write the loop using immutable data, make sure it works. Then just 1. add a call to transient in the initializations, 2. add a call to persistent! in the end and 3. add a ! to each function that modifies the transient, e.g. conj becomes conj! The benefit in doing it this way is that the language will catch and flag as errors any "mixed" usage - calling a pure function with a mutable argument or vice-versa. https://clojure.org/reference/transients https://clojure.org/reference/transients http://www.hypirion.com/musings/understanding-clojure-transients http://www.hypirion.com/musings/understanding-clojure-transi...

ridiculous_fish 8y ago

How do you implement progress reporting and cancellation in this model?

mvc 8y ago
Make it an explicit part of the data model.
- ridiculous_fish 8y ago
  A quicksort that supported progress reporting and cancellation might have a function parameter that it calls periodically to report progress, with the function returning a bool indicating cancellation. But this is using code, not data. How might this be implemented within the data model?
  - yen223 8y ago
    How do you measure progress in a quicksort?
    - ridiculous_fish 8y ago
      Glad you asked. The progress should be a bound on the worst case time. If you have N elements, then the initial worst case time is N^2. Say after the first partition, you are left with pieces N/3 and 2N/3; the worst case time is now (N/3)^2 + (2N/3)^2. Your progress after the first partition is the difference between the original worst case and the new worst case. This can make for uneven progress advancement but it’s monotonic: the progress bar will never go backwards.
  - jhomedall 8y ago
    You would implement the sort so that it either: 1: Returns the sorted list (once finished) 2: Returns an intermediate state containing the progress of the sort in addition to the current state of the sort. You would repeatedly pass the intermediate state to the sorting function until finished. You can use the progress component of the intermediate state to track progress. Something like the following (I don't actually write Haskell, so this could probably be represented better): data SomeIntermediateState = ... data Progress = Double data SortState a = Complete [a] | InProgress (SomeIntermediateState, Progress) sortInit :: [a] -> SortState sort :: SortState -> SortState

codetrotter 8y ago

While the purpose of this screencast is to explain a principle, not to focus on the particular piece of software that was implemented, I am left wondering about a couple of things; The timeline as he explains it, is updated by creating a new timeline instance which consists of previous timeline + new tweets. It seems to me then, that his client does not remove from the timeline tweets that were deleted by its author subsequently to having been downloaded. To some it might be a feature to capture as many tweets as possible, but at the same time, if your view of the timeline includes deleted tweets then you might find yourself trying to reply to a deleted tweet, and then you would get an error response from Twitter when you try to post your reply. (Though I don’t know if his client also does posting, or if it’s a read-only client.) Furthermore, what about tweets that are edited by their author subsequently to having been downloaded? Seems that you would not see those edits.

gary_bernhardt 8y ago
Tweets aren't editable so no need to handle that. Deleted tweets can be honored by updating the timeline to be `old_timeline - deleted_tweets + new_tweets`.

danidiaz 8y ago

In the Idris language, one best practice is to relegate to the outer shell not only IO effects, but also "partiality"—in the sense of computations that might enter an unproductive infinite loop. (In Haskell, one way of getting into unproductive infinite loops is by mistakenly asking "parse elements until the first failure" to a parser combinator that can always succeed without consuming input.) The ideal is that the core of the application should be both pure and total, purity and totality being tracked by the type system. In fact, one can often relegate partiality to a single function of the outer shell.

draw_down 8y ago

Love this one! I think Rich Hickey’s “The Value of Values” pairs nicely with it.

squirrelicus 8y ago

It's difficult to overstate the importance of the principles taught in this talk. Almost as difficult as it is to describe why they are so important. The fundamentals of quality software are found here, not in some Martin Fowler dissertation on DDD. Edit: correction from commenter, thanks!

yawaramin 8y ago
overstate
stevebmark 8y ago
I think the general concept of using immutability, queues, and isolating side effects, is lost and muddied by using Ruby, and it's a misuse of the term "functional."

squirrelicus 8y ago

Often people will ask how to follow these principles in the world of I/O. The answer is incredibly simple: with values. Return a DTO that encapsulates all the business information about the result state of the I/O call. Do not throw exceptions. Log them if you like, but you must return all the data necessary for the business logic to react to failure conditions, encoded in your own use case specific structure. If you don't care about whether you failed due to timeouts or refused connections or query parse failures what have you, don't return that data, just return a hasError kind of property on the return structure (bonus points if your language supports discriminated unions, but this is not necessary). If the parent logic needs to react to timeouts and failures to connect differently, then catch those exceptions or state separately and return didTimeout or cantConnect flags separately. Values values values

the_gipsy 8y ago
I would prefer a ‘Result<ValueType, ErrorType>’ over stuffing weird ‘hasError’, ‘didTimeout’, etc flags everywhere. It’s much safer to type-check at compile time that you have discriminated between result and error, than to hope that you are ‘if’-checking some flags at the right time at runtime.
- squirrelicus 8y ago
  That is another way to accomplish the same goal. This trade-off is more aesthetic than architectural and is just a matter of language support

GolDDranks 8y ago

This is the pattern I'm striving to code in, but when developing with a team, I keep struggling convincing other people (with OOP, mutability-happy mindsets) to do so...

jypepin 8y ago

And I'd strongly recommend paying for DAS and looking at every.single.video in there. Gary is brilliant and each video explains stuff in a very simple, enjoyable way. Reminds me a bit of Ryan Bate's ruby/rails videos.

abenedic 8y ago

So, I am a person who wishes there was a proper article I could read for this. I don't know how much this matters to people, but I can understand written English far far better than I can understand spoken English. I know auto-translate works for some. I am trying with applet and get nothing. I like core idea. I feel often people outside English land get second class experience.

godot 8y ago
You're not alone. I immigrated to the US from Asia in the 90s. For many years I didn't understand spoken English fully. Arguably I still miss some words now and then, even nowadays, after over 20 years living in an English environment. I had great grades throughout high school and college here mostly thanks to having excellent reading and writing skills in English, since I came from a place in Asia that emphasized learning English since childhood. Listening to spoken English is not easy for non native speakers.
- abenedic 8y ago
  > Listening to spoken English is not easy for non native speakers Thank you for saying this. You clearly have a good understanding of English(far better than mine at least). I feel like this is an area that is glossed over. For every article that is written in english there are at least 10+ non-native speaker struggling to understand the work, who could extract something useful or help explicate the work.
  - theoh 8y ago
    In the case of this video, you aren't missing much. It's deliberately 'dense', according to the author, but the concept is simple. Almost any app will involve some imperative code, but using pure functions within that, where possible, makes it easier to reason about what is going on. That's all there is to it. https://en.wikipedia.org/wiki/Referential_transparency https://en.wikipedia.org/wiki/Referential_transparency
pfranz 8y ago
I've followed Gary's stuff for a bunch of years now. His preferred medium seems to be screencasts and the occasional conference talk. His blog dating back to 2011 has 10 posts (he does seem to tweet often, though). His screencasts are very thoughtfully done and dense. So I imagine you'd have to find that article from someone else. As a native English speaker, I generally have the same preference you do. Articles I can skim, they're easier to refer back to, and I can search. I hope you can take solace that I will sometimes search for an error message and the only result is a forum posting in a foreign language that Google Translate barfs on.

blt 8y ago

I'm struggling to apply this model to code for machine learning experiments. The functional core takes an extremely long time to compute, e.g. "train the model". Thus, you want to save lots of intermediate values to disk so you can reload them over and over while you are experimenting the downstream part of the program. I end up with a lot of code writing values to the disk, which is currently mixed in with the computations. I'm wondering if there's some way to automate this "save intermediate values to the disk" so I can write the code in a more functional style without having to constantly go in and out of the imperative shell portion. The process of ML experimentation is extremely painful compared to normal software development where nothing really takes that long to compute.

waluigi 8y ago
The way I would approach this is to write what essentially amounts to a declarative specification of what computation needs to be run, and then define an interpreter that handles the caching of intermediate values. The "Embedded DSL + Interpreter" pattern is incredibly powerful, and it's nice to see it catching on more.

ceronman 8y ago

This is the same principle of the Clean Architecture. I like how Brandon Rhodes explain how to apply it in Python: https://www.youtube.com/watch?v=DJtef410XaM https://www.youtube.com/watch?v=DJtef410XaM

fpoling 8y ago

One does not need a pure functional core to have benefits of such design. The key idea is that the core code should not change or even read the global state. It should only be allowed to access or change what is explicitly passed to it. For example, to avoid excessive memory consumption to sum matrixes one wants to code it like A += B, not like C = A + B. Yet one still benefits from all the testing, design etc. benefits of pure functional code. At the end one call always get a functional version just by turning A += B into C = A; C += B.

Flenser 8y ago

After watching this and his Boundaries talk, and then reading the Out of the Tar Pit paper, I started watching Pete Hunt's first talks on why they built React and had an epiphany why React was going to end up being the most popular front end framework.

jcyw 8y ago

This is the github repo featured in the video: https://github.com/bf4/toot https://github.com/bf4/toot

jcyw 8y ago

The idea of functional core is cool. I find domain-driven design a better guide for implementation. In a nut shell, there are three types of objects: value, entity, and service. Both value and entity are functional and stateless objects. A service object maps between stateless objects and may cross the domain boundary (such as network, IO, etc). When I do this in java, a service object would have interfaces returning CompletableFuture of Value or Entity objects. InfoQ has a very nice summary of DDD book: https://www.infoq.com/minibooks/download/domain-driven-design-quickly https://www.infoq.com/minibooks/download/domain-driven-desig...

Apaec 8y ago

I learned this through Haskell, it enforces the "functional core, imperative shell" approach through the IO monad. Now I find myself repeating the same pattern in other languages(js, java).

stevebmark 8y ago

"Functional core" littered with classes, attr_readers, data that's not data (Curosr.new) and += imperative loops. Yikes.

gary_bernhardt 8y ago
The classes serve to close some functions over variables for convenience; the attr_readers serve as destructuring functions for convenience; the += replaces a recursive function for convenience and familiarity. The function with the += doesn't mutate any value and it remains pure from all callers' perspectives. You're complaining about syntax, but this screencast is about semantics.
- stevebmark 8y ago
  No, something with "functional" in the title that doesn't use functions isn't related to syntax.
  - gary_bernhardt 8y ago
    What is a function?