5 ms·

Clojure's Solutions to the Expression Problem

158 points by adityaathalye 1y ago

ares623 1y ago
I miss the days when productivity booster posts were all about REPL, functional programming, composability, immutability.
- gleenn 1y ago
  I just wish more people appreciated those things at all. What's the name of the rule where everyone with a sufficiently large codebase has a partially implemented, buggy time oriented data log somewhere and it's too bad people didn't realize that sooner. Datomic et all are so cool and solve so many interesting and pervasive problems but people would rather just mangle those bits themselves. The glory of refactoring a large, purely immutable piece of software is so glorious and easy. The wrapper/adapter hell that comes out of the Expression Problem is just one more thing that people just assume is unavoidable and the amount of gross code that falls out of that is so painful.
  - adityaathalye 1y ago
    "Henderson's Tenth Law" :D *But beware. Once you see, you cannot un-see the fact that…* Any sufficiently complicated data system contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a bitemporal database. — Henderson's Tenth Law. (= henderson https://github.com/jarohen) I blogged about it here: https://www.evalapply.org/posts/poor-mans-time-oriented-data-system/index.html#reading-guide-thinky-thoughts-alert-same-thing https://www.evalapply.org/posts/poor-mans-time-oriented-data... (discussed here recently: https://news.ycombinator.com/item?id=44583790 https://news.ycombinator.com/item?id=44583790 ). (edit: add links)
    - olau 1y ago
      I enjoyed reading most of that. One thing to watch out for with immutability is that if you're dealing with personal information about people, immutability is probably illegal. You must be able to forget information, and not just simulate you've forgotten it. I don't how universal this is at the moment, but I think it's likely to be more universal in the future.
      - adityaathalye 1y ago
        Thank you for the kind words, and feedback! Yes, controlled excision is a (necessary) capability of all event-sourced systems and/or immutable object stores, i.e. it's quite universal already. All such systems have to build in some sort of special-case mechanism to excise facts, not merely redact (~tombstone) them. But data-destruction superpower must be used sparingly, with care. See: git's documentation for --force-push, for example (and bemoan what popular git forges force us to do on a normal basis). Back to DB-land... Datomic does it this way: https://docs.datomic.com/operation/excision.html https://docs.datomic.com/operation/excision.html Excision is the complete removal of a set of datoms matching a predicate. Excision should be a very infrequent operation and is designed to support the following two scenarios: Removing data for privacy reasons Removing data older than some domain-defined retention period Excision should never be used to correct erroneous data and is unsuitable for that task as it does not restore any previous view of the facts. Consider using ordinary retraction to correct errors without removing history. XTDB calls it ERASE: https://docs.xtdb.com/reference/main/sql/txs.html#_erase https://docs.xtdb.com/reference/main/sql/txs.html#_erase Irrevocably erases documents from a table, for all valid-time, for all system-time. While XTDB is immutable, in some cases it is legally necessary to irretrievably delete data (e.g. for a GDPR request). This operation removes data such that even queries as of a previous system-time no longer return the erased data.
  - adityaathalye 1y ago
    > The wrapper/adapter hell that comes out of the Expression Problem Yeah, I got lucky in my work (see sibling comment https://news.ycombinator.com/item?id=45207880 https://news.ycombinator.com/item?id=45207880). For me, it was PageObjects all the way down, which "just composed". Something like this: (defprotocol IPageObject "Each PageObject MUST implement the IPageObject protocol." (page-object [this]) (exists? [this]) (visible? [this])) And then an implementation like this: (defrecord Checkbox [target-css] IPageObject (page-object [this] this) (exists? [this] ;; webdriver check if target-css exists ) (visible? [this] ;; webdriver check if target-css is visible ) Selectable (select [this] ;; webdriver select the target ) (deselect [this] ;; webdriver undo selection ) (selected? [this] ;; webdriver return true if target selected ) Value (get-value [this] ;; webdriver return current selection state (selected? this))) Thanks to Records (stateless, type-identified, hash-map semantics), the PageObject model composes arbitrarily. i.e. A Page is itself a PageObject containing a tree of arbitrarily nested PageObjects... i.e. PageObjects all the way down. Solid gold! (edit: x-link to sibling comment)
    - geokon 1y ago
      This example just scratches the surface. I don't think it's showing any Clojure-magic per se, b/c in effect what you're showing is just "implementing an interface". You can do that in most languages. The magic of protocol/records is that they work across library boundaries. A library may provide a record - and then you can extend the record with new protocols. Key is that it's all without needing to explicitly creating new agglomeration types. You can take some Dog record from some pet-simulation library and then `extend-type` it with the IPageObject protocol and make the Dog record now something that can be displayed on a webpage
      - roenxi 1y ago
        The magic of Clojure is that if a problem can be solved with an interface Clojure lets you solve it with an interface. It doesn't have any magic to show off, it is just a relentless implementation of a lot of basic good ideas.
  - merlincorey 1y ago
    Greenspun's tenth rule: > Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. > -- Philip Greenspun,
- adityaathalye 1y ago
  So. Much. This! Circa 2013-15, I was a QA having to write some Selenium / WebDriver test suites for the web application side of our product. At the time, when our team was tiny, Clojure was one of our "company languages" (besides JavaScript/YUI for web, and iOS/Objective-C, Android/Java for our mobile SDKs). Our extant web testing framework was written in Clojure, so I continued refactoring that to support our people through "hypergrowth" feature churn including a web tech migration from YUI to React (took about four years for YUI to completely factor out, IIRC). I did not fully grok just how much Clojure's multiple dispatch facility helped me then. I knew that feature was useful, of course, but the magnitude became clearer as the years progressed. It allowed a single person---yours truly---to address the real needs of a ridiculously complex moving target (because B2B web products be like that --- full of special cases and customisation options and spooky effects at a distance). I would not want to try supporting a test suite in that sort of situation, without at least these Clojurish programming facilities, that I have come to expect as a given these days. Talk: https://www.youtube.com/watch?v=hwoLON80ZzA&list=PLG4-zNACPCsNVwz3ohXKeoeQjRLoFsc7F&index=5 https://www.youtube.com/watch?v=hwoLON80ZzA&list=PLG4-zNACPC... PDF Deck (contains reading references): designing_object_functional_system_IN-Clojure_2016.pdf, here https://github.com/adityaathalye/slideware/ https://github.com/adityaathalye/slideware/ Incidentally, I implemented the refactored test suite API using the "PageObject" abstraction. An apt OO abstraction is worth its weight in gold, provided the implementation composes naturally. Thanks, Martin Fowler! This very post helped me back then: https://martinfowler.com/bliki/PageObject.html https://martinfowler.com/bliki/PageObject.html (edit: formatting, small bit of context)
stefcoetzee 1y ago
Enjoyed this demonstration as well: https://max.computer/blog/solving-the-expression-problem-in-clojure/ https://max.computer/blog/solving-the-expression-problem-in-...
emccue 1y ago
One of the drawbacks about protocols mentioned in the talk is no longer correct. Protocols can now dispatch on metadata, not just type. (defprotocol Dog :extend-via-metadata true (bark [_]))
- panick21_ 1y ago
  Oh. Interesting. I had to look it up: When :extend-via-metadata is true, values can extend protocols by adding metadata where keys are fully-qualified protocol function symbols and values are function implementations. Protocol implementations are checked first for direct definitions (defrecord, deftype, reify), then metadata definitions, then external extensions (extend, extend-type, extend-protocol). (def component (with-meta {:name "db"} {`start (constantly "started")})) (start component) ;;=> "started" Kind of a crazy feature.
jbhn 1y ago
Note that "full" solutions are possible in standard OO languages as well (without the usual visitor vs subclasses trade-off): https://news.ycombinator.com/item?id=45210114 https://news.ycombinator.com/item?id=45210114
- adityaathalye 1y ago
  I agree, given that (JVM) Clojure is literally just a Java Library. From a programmer's point of view I don't want to have to write all the machinery that makes for a "full" solution (that is also correct and performant enough)... In fact I can't! Which is why I hope more programming languages make that machinery available, because the "X/Y" problem (Type / Method) manifests itself (pretty easily) in every single web app out there (and likely most software ever written).

158 points by adityaathalye 1y ago

I miss the days when productivity booster posts were all about REPL, functional programming, composability, immutability.

gleenn 1y ago
I just wish more people appreciated those things at all. What's the name of the rule where everyone with a sufficiently large codebase has a partially implemented, buggy time oriented data log somewhere and it's too bad people didn't realize that sooner. Datomic et all are so cool and solve so many interesting and pervasive problems but people would rather just mangle those bits themselves. The glory of refactoring a large, purely immutable piece of software is so glorious and easy. The wrapper/adapter hell that comes out of the Expression Problem is just one more thing that people just assume is unavoidable and the amount of gross code that falls out of that is so painful.
- adityaathalye 1y ago
  "Henderson's Tenth Law" :D *But beware. Once you see, you cannot un-see the fact that…* Any sufficiently complicated data system contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a bitemporal database. — Henderson's Tenth Law. (= henderson https://github.com/jarohen) I blogged about it here: https://www.evalapply.org/posts/poor-mans-time-oriented-data-system/index.html#reading-guide-thinky-thoughts-alert-same-thing https://www.evalapply.org/posts/poor-mans-time-oriented-data... (discussed here recently: https://news.ycombinator.com/item?id=44583790 https://news.ycombinator.com/item?id=44583790 ). (edit: add links)
  - olau 1y ago
    I enjoyed reading most of that. One thing to watch out for with immutability is that if you're dealing with personal information about people, immutability is probably illegal. You must be able to forget information, and not just simulate you've forgotten it. I don't how universal this is at the moment, but I think it's likely to be more universal in the future.
    - adityaathalye 1y ago
      Thank you for the kind words, and feedback! Yes, controlled excision is a (necessary) capability of all event-sourced systems and/or immutable object stores, i.e. it's quite universal already. All such systems have to build in some sort of special-case mechanism to excise facts, not merely redact (~tombstone) them. But data-destruction superpower must be used sparingly, with care. See: git's documentation for --force-push, for example (and bemoan what popular git forges force us to do on a normal basis). Back to DB-land... Datomic does it this way: https://docs.datomic.com/operation/excision.html https://docs.datomic.com/operation/excision.html Excision is the complete removal of a set of datoms matching a predicate. Excision should be a very infrequent operation and is designed to support the following two scenarios: Removing data for privacy reasons Removing data older than some domain-defined retention period Excision should never be used to correct erroneous data and is unsuitable for that task as it does not restore any previous view of the facts. Consider using ordinary retraction to correct errors without removing history. XTDB calls it ERASE: https://docs.xtdb.com/reference/main/sql/txs.html#_erase https://docs.xtdb.com/reference/main/sql/txs.html#_erase Irrevocably erases documents from a table, for all valid-time, for all system-time. While XTDB is immutable, in some cases it is legally necessary to irretrievably delete data (e.g. for a GDPR request). This operation removes data such that even queries as of a previous system-time no longer return the erased data.
- adityaathalye 1y ago
  > The wrapper/adapter hell that comes out of the Expression Problem Yeah, I got lucky in my work (see sibling comment https://news.ycombinator.com/item?id=45207880 https://news.ycombinator.com/item?id=45207880). For me, it was PageObjects all the way down, which "just composed". Something like this: (defprotocol IPageObject "Each PageObject MUST implement the IPageObject protocol." (page-object [this]) (exists? [this]) (visible? [this])) And then an implementation like this: (defrecord Checkbox [target-css] IPageObject (page-object [this] this) (exists? [this] ;; webdriver check if target-css exists ) (visible? [this] ;; webdriver check if target-css is visible ) Selectable (select [this] ;; webdriver select the target ) (deselect [this] ;; webdriver undo selection ) (selected? [this] ;; webdriver return true if target selected ) Value (get-value [this] ;; webdriver return current selection state (selected? this))) Thanks to Records (stateless, type-identified, hash-map semantics), the PageObject model composes arbitrarily. i.e. A Page is itself a PageObject containing a tree of arbitrarily nested PageObjects... i.e. PageObjects all the way down. Solid gold! (edit: x-link to sibling comment)
  - geokon 1y ago
    This example just scratches the surface. I don't think it's showing any Clojure-magic per se, b/c in effect what you're showing is just "implementing an interface". You can do that in most languages. The magic of protocol/records is that they work across library boundaries. A library may provide a record - and then you can extend the record with new protocols. Key is that it's all without needing to explicitly creating new agglomeration types. You can take some Dog record from some pet-simulation library and then `extend-type` it with the IPageObject protocol and make the Dog record now something that can be displayed on a webpage
    - roenxi 1y ago
      The magic of Clojure is that if a problem can be solved with an interface Clojure lets you solve it with an interface. It doesn't have any magic to show off, it is just a relentless implementation of a lot of basic good ideas.
- merlincorey 1y ago
  Greenspun's tenth rule: > Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. > -- Philip Greenspun,
adityaathalye 1y ago
So. Much. This! Circa 2013-15, I was a QA having to write some Selenium / WebDriver test suites for the web application side of our product. At the time, when our team was tiny, Clojure was one of our "company languages" (besides JavaScript/YUI for web, and iOS/Objective-C, Android/Java for our mobile SDKs). Our extant web testing framework was written in Clojure, so I continued refactoring that to support our people through "hypergrowth" feature churn including a web tech migration from YUI to React (took about four years for YUI to completely factor out, IIRC). I did not fully grok just how much Clojure's multiple dispatch facility helped me then. I knew that feature was useful, of course, but the magnitude became clearer as the years progressed. It allowed a single person---yours truly---to address the real needs of a ridiculously complex moving target (because B2B web products be like that --- full of special cases and customisation options and spooky effects at a distance). I would not want to try supporting a test suite in that sort of situation, without at least these Clojurish programming facilities, that I have come to expect as a given these days. Talk: https://www.youtube.com/watch?v=hwoLON80ZzA&list=PLG4-zNACPCsNVwz3ohXKeoeQjRLoFsc7F&index=5 https://www.youtube.com/watch?v=hwoLON80ZzA&list=PLG4-zNACPC... PDF Deck (contains reading references): designing_object_functional_system_IN-Clojure_2016.pdf, here https://github.com/adityaathalye/slideware/ https://github.com/adityaathalye/slideware/ Incidentally, I implemented the refactored test suite API using the "PageObject" abstraction. An apt OO abstraction is worth its weight in gold, provided the implementation composes naturally. Thanks, Martin Fowler! This very post helped me back then: https://martinfowler.com/bliki/PageObject.html https://martinfowler.com/bliki/PageObject.html (edit: formatting, small bit of context)

stefcoetzee 1y ago

Enjoyed this demonstration as well: https://max.computer/blog/solving-the-expression-problem-in-clojure/ https://max.computer/blog/solving-the-expression-problem-in-...

emccue 1y ago

One of the drawbacks about protocols mentioned in the talk is no longer correct. Protocols can now dispatch on metadata, not just type. (defprotocol Dog :extend-via-metadata true (bark [_]))

panick21_ 1y ago
Oh. Interesting. I had to look it up: When :extend-via-metadata is true, values can extend protocols by adding metadata where keys are fully-qualified protocol function symbols and values are function implementations. Protocol implementations are checked first for direct definitions (defrecord, deftype, reify), then metadata definitions, then external extensions (extend, extend-type, extend-protocol). (def component (with-meta {:name "db"} {`start (constantly "started")})) (start component) ;;=> "started" Kind of a crazy feature.

jbhn 1y ago

Note that "full" solutions are possible in standard OO languages as well (without the usual visitor vs subclasses trade-off): https://news.ycombinator.com/item?id=45210114 https://news.ycombinator.com/item?id=45210114

adityaathalye 1y ago
I agree, given that (JVM) Clojure is literally just a Java Library. From a programmer's point of view I don't want to have to write all the machinery that makes for a "full" solution (that is also correct and performant enough)... In fact I can't! Which is why I hope more programming languages make that machinery available, because the "X/Y" problem (Type / Method) manifests itself (pretty easily) in every single web app out there (and likely most software ever written).