20 ms·

Relic: Functional relational programming for Clojure(Script)

224 points by zonotope 4y ago

prettyStandard 4y ago
This looks really interesting! As someone who has worked with relational databases and Clojure in the past, I can definitely see the appeal of a functional relational programming model. I like that relic provides support for declarative data processing and declarative relational constraints. These are areas that can be tricky to handle when working with traditional relational databases, so it's great to see a library that addresses these pain points. The ability to use relic with reactive programming is also a big plus. I'm curious to see how this would work in practice, particularly in the context of an interactive application. Overall, relic seems like a promising library for anyone looking to work with normalized data in Clojure. I'll definitely be giving it a try on my next project!
joelittlejohn 4y ago
Dan recently recorded a session about Relic, if you'd like to hear him speak more about the origins of the library, some design choices, and some examples: https://youtube.com/watch?v=QsEJ5O2e4Es https://youtube.com/watch?v=QsEJ5O2e4Es
Funnyduck99 4y ago
I dont like clojure
- ballpark 4y ago
  Funnyduck99 is probably mocking all the people that go into HN posts about Clojure to complain about how their team of non-Clojure programmers went to work on a Clojure project and it was a bad experience.
  - Funnyduck99 4y ago
    No I genuinely don't like clojure because I just finished my first clojure class, it was also my first functional language and it was online and i am bad at learning online so thats why.
    - dgb23 4y ago
      That’s a much more interesting comment than the top level one. People generally care about learning experiences and explicitly subjective statements.
    - OliverM 4y ago
      Sounds like any functional language would have disappointed in that case?
      - Funnyduck99 4y ago
        yup thats what i said
        eyelidlessness 4y ago
        I hope you get an opportunity to explore it (FP, whether Clojure or otherwise) in a more conducive environment. It sounds like this wasn’t the best learning environment for you, and that’s totally valid, but there’s a lot of good stuff to learn if you’re in an environment that suits you.
    - rawoke083600 4y ago
      LPT: While learning Clojure, the following (almost always true) mental-model help me massively at "getting" Clojure. "It's Maps All the Day Down" Spend a lot of time, just learning how you (CRUD) map contents. There will be enough time to tackle the other cases/tech (atoms, protocols etc) but until you get good at maps don't get bogged down by the other cool stuff.
      - lenkite 4y ago
        "It's Maps All the Day Down" In reality - its actually Trees all the way down. But, because you don't have proper structures in Clojure, one uses Maps when one should actually be using Trees.
        4y ago
wcerfgba 4y ago
How are updates triggered, what's the mechanism? From the examples it seems like you don't need to subscribe and provide a function to be called when changes happen, but then how does it work?
- wotbrew 4y ago
  A relic db is a persistent data-structure [1]. Applying a transaction with rel/transact gives you a new database, rel/track-transact also returns the changes to relations you have opted-in to change tracking (using rel/watch). [1] https://en.wikipedia.org/wiki/Persistent_data_structure https://en.wikipedia.org/wiki/Persistent_data_structure
sesm 4y ago
I like how the typical JS-library-style README is put separately in a 'Pitch' section.
Rodeoclash 4y ago
Clojure(script) always seems to me to be this hotbed of interesting ideas in programming. I.e. you'll see something wild like this start here then eventually the concepts make their way out into regular JavaScript. I'm almost starting to regret not picking Clojurescript for my app
- wotbrew 4y ago
  A big part of the curse of lisp is that anything feels possible. Even for one guy who works at a veg shop.
  - grugagag 4y ago
    There’s nothing wrong with it. It’s wrong not pursuing once get that feeling. Worse outcome is a much better understanding
    - danuker 4y ago
      Worse outcome is wasting time tying your future to a language that will die soon, with few people you could hire to help. But the rewards are in line with the risks.
      - stevepeg 4y ago
        Geez, it's a programming language not a spouse! You can just, you know, use another one if it doesn't work out.
        eyelidlessness 4y ago
        Yep! I’ve been years parted from Clojure(Script) and I’m still glad we had those times we shared together.
  - jsyolo 4y ago
    I don't see any macro usage in this project, what part of clojure made you feel anything is possible for this project?
Scarbutt 4y ago
I guess sqlite + honeysql would be an alternative. Curious to know why the author prefers the relations/table/codd model over graph-map-databases like datomic and thinks something like datomic "feels out" of "out of the tarpit".
- wotbrew 4y ago
  Mainly it is because the tar pit paper proposed the relational algebra in its functional relational programming. I do think for some it might be easier to reason mechanically about dataflow through collection-oriented operators. But I suppose it's subjective. clj-3df [1] to my understanding does something like Relic for datomic datalog using differential dataflow. Disclaimer: I now work on XTDB [2], so datalog is somewhat now my day job. [1] https://github.com/sixthnormal/clj-3df https://github.com/sixthnormal/clj-3df [2] https://xtdb.com/ https://xtdb.com/
  - ithrow 4y ago
    Interesting. What are you thoughts in regards to Hickey's comments about the structural rigidity of relations/tables to represent information on how they impede flexibility, make your program hard to change over time and increase complexity. He mentions it in various videos but these snippets are two quick finds: - https://youtu.be/Pz_NvY1kw6I?t=489 https://youtu.be/Pz_NvY1kw6I?t=489 - https://youtu.be/thpzXjmYyGk?t=255 https://youtu.be/thpzXjmYyGk?t=255
    - refset 4y ago
      I'm not the OP, but thinking well beyond the original topic of in-memory reactive programming (where my answer would be quite different) to the world of long-lived durable databases... one perspective to consider is that any system built around N-ary relations can automatically benefit from the full range of relational algebra for transforming and composing both base data and derived relations. The flexibility of N-ary relations is largely what has kept SQL databases relevant despite the flaws of SQL itself. In contrast, a system that only handles base data in terms of triples assumes that you have a perfect attribute-oriented information model figured out upfront. But given this is rarely the case users will want tools that help them to easily transform/migrate their data and schema over time. Ideally this takes the form of a declarative language that minimises the amount of code that needs to be written. However, without a compelling end-to-end transformation language figured out I think any alternative database systems with their alternative information models (triple-based or otherwise) will struggle to compare favourably with mainstream databases, where declarative data munging with SQL is considered valuable and routine. Triples may well prove to be the best way to handle information in software over the long-term, but I'm not sure that the systems which currently work with triples are good enough or widespread enough to test that theory.
brap 4y ago
What makes people see Lisp-like languages and think to themselves "yep, this is how I want my code to look like" is beyond me
- tmtvl 4y ago
  I like the structure that S-expressions convey. Must be something with how my brain works. It's probably also why I find Python code utterly unreadable.
  - brap 4y ago
    To each their own. There are obviously very smart people choosing Lisp so I’m not trying to knock it down or anything, I just never got the appeal Every now and then I try to check out a Lisp project and this is my reaction the moment I see the code: https://giphy.com/gifs/seinfeld-bye-jerry-106PwpLIIXJnXi https://giphy.com/gifs/seinfeld-bye-jerry-106PwpLIIXJnXi
    - vindarel 4y ago
      This is what we see: https://raw.githubusercontent.com/tarsius/paren-face/master/parentheses.png https://raw.githubusercontent.com/tarsius/paren-face/master/... Don't look at the parens and don't stop at the superficial look :]
    - dgb23 4y ago
      For me much of the appeal of the syntax comes from using it with a editor integrated REPL, or when writing macros, or when generating linter configuration from a domain model etc. However I agree with you on some level. Lisp code can easily be written in a way that’s hard to read. Specifically because of nesting expression way more often than is comfortable for me to read. I much rather prefer let bindings and threading so most of the code reads from left to right.
    - rawoke083600 4y ago
      >To each their own. There are obviously very smart people choosing Lisp so I’m not trying to knock it down or anything, I just never got the appeal Just know that almost no one STARTS off by liking Lisp-Syntax, but oh boy does it grow on you.
crabmusket 4y ago
Using the relational model for app data in memory is really interesting. Martin Fowler wrote about doing that as a way to get around the "object-relational mismatch" issue[1]. Richard Fabian describes "data-oriented design" as having a lot of overlap with the relational model[2]. ECSes becoming very popular in game engines are basically in-memory relational databases where "components" are "tables"[3]. [1]: https://martinfowler.com/bliki/OrmHate.html https://martinfowler.com/bliki/OrmHate.html [2]: https://dataorienteddesign.com/dodbook/ https://dataorienteddesign.com/dodbook/ [3]: https://github.com/SanderMertens/ecs-faq#what-is-ecs https://github.com/SanderMertens/ecs-faq#what-is-ecs
- sitkack 4y ago
  I'd love to see something like Relic able to operate over capnproto/arrow/parquet/avro/hdf5 etc along with a DAG based recomputation engine.
- nextaccountic 4y ago
  > "components" are "tables" No, components are columns. Each entity is a row in a table and an archetype (the set of all entities with the same components) is a table. In ECS it's usual to store components together, this corresponds to columnar storage in database terms (or equivalently, it's as if data were in sixth normal form [0]). However in some systems you can opt to store a more traditional row-based column. [0] https://en.wikipedia.org/wiki/Database_normalization#Satisfying_6NF https://en.wikipedia.org/wiki/Database_normalization#Satisfy...
  - crabmusket 4y ago
    Thanks for the correction. In equating components and tables I was thinking of archetype-based engines, but even in that case they may store a whole set of components together in a single "table", right? Conceptually I think you could equate components with relations. For example a "grid location" component could be imagined as a relation with "entity ID" and "coordinate" columns. I suppose that admits multiple component instances per entity ID though which is typically not what you want.
  - 4y ago
lincpa 4y ago
[dead]
tantaman 4y ago
Beautiful. I've been noodling on exactly this same idea. So far I've explored using SQLite but it would be ideal if there was no SQL between me and my relations. Another direction I've taken is adding STM to JavaScript and implementing a JS api for querying relations.

224 points by zonotope 4y ago

This looks really interesting! As someone who has worked with relational databases and Clojure in the past, I can definitely see the appeal of a functional relational programming model. I like that relic provides support for declarative data processing and declarative relational constraints. These are areas that can be tricky to handle when working with traditional relational databases, so it's great to see a library that addresses these pain points. The ability to use relic with reactive programming is also a big plus. I'm curious to see how this would work in practice, particularly in the context of an interactive application. Overall, relic seems like a promising library for anyone looking to work with normalized data in Clojure. I'll definitely be giving it a try on my next project!

joelittlejohn 4y ago

Dan recently recorded a session about Relic, if you'd like to hear him speak more about the origins of the library, some design choices, and some examples: https://youtube.com/watch?v=QsEJ5O2e4Es https://youtube.com/watch?v=QsEJ5O2e4Es

Funnyduck99 4y ago

I dont like clojure

ballpark 4y ago
Funnyduck99 is probably mocking all the people that go into HN posts about Clojure to complain about how their team of non-Clojure programmers went to work on a Clojure project and it was a bad experience.
- Funnyduck99 4y ago
  No I genuinely don't like clojure because I just finished my first clojure class, it was also my first functional language and it was online and i am bad at learning online so thats why.
  - dgb23 4y ago
    That’s a much more interesting comment than the top level one. People generally care about learning experiences and explicitly subjective statements.
  - OliverM 4y ago
    Sounds like any functional language would have disappointed in that case?
    - Funnyduck99 4y ago
      yup thats what i said
      - eyelidlessness 4y ago
        I hope you get an opportunity to explore it (FP, whether Clojure or otherwise) in a more conducive environment. It sounds like this wasn’t the best learning environment for you, and that’s totally valid, but there’s a lot of good stuff to learn if you’re in an environment that suits you.
  - rawoke083600 4y ago
    LPT: While learning Clojure, the following (almost always true) mental-model help me massively at "getting" Clojure. "It's Maps All the Day Down" Spend a lot of time, just learning how you (CRUD) map contents. There will be enough time to tackle the other cases/tech (atoms, protocols etc) but until you get good at maps don't get bogged down by the other cool stuff.
    - lenkite 4y ago
      "It's Maps All the Day Down" In reality - its actually Trees all the way down. But, because you don't have proper structures in Clojure, one uses Maps when one should actually be using Trees.
      - 4y ago

wcerfgba 4y ago

How are updates triggered, what's the mechanism? From the examples it seems like you don't need to subscribe and provide a function to be called when changes happen, but then how does it work?

wotbrew 4y ago
A relic db is a persistent data-structure [1]. Applying a transaction with rel/transact gives you a new database, rel/track-transact also returns the changes to relations you have opted-in to change tracking (using rel/watch). [1] https://en.wikipedia.org/wiki/Persistent_data_structure https://en.wikipedia.org/wiki/Persistent_data_structure

sesm 4y ago

I like how the typical JS-library-style README is put separately in a 'Pitch' section.

Rodeoclash 4y ago

Clojure(script) always seems to me to be this hotbed of interesting ideas in programming. I.e. you'll see something wild like this start here then eventually the concepts make their way out into regular JavaScript. I'm almost starting to regret not picking Clojurescript for my app

wotbrew 4y ago
A big part of the curse of lisp is that anything feels possible. Even for one guy who works at a veg shop.
- grugagag 4y ago
  There’s nothing wrong with it. It’s wrong not pursuing once get that feeling. Worse outcome is a much better understanding
  - danuker 4y ago
    Worse outcome is wasting time tying your future to a language that will die soon, with few people you could hire to help. But the rewards are in line with the risks.
    - stevepeg 4y ago
      Geez, it's a programming language not a spouse! You can just, you know, use another one if it doesn't work out.
      - eyelidlessness 4y ago
        Yep! I’ve been years parted from Clojure(Script) and I’m still glad we had those times we shared together.
- jsyolo 4y ago
  I don't see any macro usage in this project, what part of clojure made you feel anything is possible for this project?

Scarbutt 4y ago

I guess sqlite + honeysql would be an alternative. Curious to know why the author prefers the relations/table/codd model over graph-map-databases like datomic and thinks something like datomic "feels out" of "out of the tarpit".

wotbrew 4y ago
Mainly it is because the tar pit paper proposed the relational algebra in its functional relational programming. I do think for some it might be easier to reason mechanically about dataflow through collection-oriented operators. But I suppose it's subjective. clj-3df [1] to my understanding does something like Relic for datomic datalog using differential dataflow. Disclaimer: I now work on XTDB [2], so datalog is somewhat now my day job. [1] https://github.com/sixthnormal/clj-3df https://github.com/sixthnormal/clj-3df [2] https://xtdb.com/ https://xtdb.com/
- ithrow 4y ago
  Interesting. What are you thoughts in regards to Hickey's comments about the structural rigidity of relations/tables to represent information on how they impede flexibility, make your program hard to change over time and increase complexity. He mentions it in various videos but these snippets are two quick finds: - https://youtu.be/Pz_NvY1kw6I?t=489 https://youtu.be/Pz_NvY1kw6I?t=489 - https://youtu.be/thpzXjmYyGk?t=255 https://youtu.be/thpzXjmYyGk?t=255
  - refset 4y ago
    I'm not the OP, but thinking well beyond the original topic of in-memory reactive programming (where my answer would be quite different) to the world of long-lived durable databases... one perspective to consider is that any system built around N-ary relations can automatically benefit from the full range of relational algebra for transforming and composing both base data and derived relations. The flexibility of N-ary relations is largely what has kept SQL databases relevant despite the flaws of SQL itself. In contrast, a system that only handles base data in terms of triples assumes that you have a perfect attribute-oriented information model figured out upfront. But given this is rarely the case users will want tools that help them to easily transform/migrate their data and schema over time. Ideally this takes the form of a declarative language that minimises the amount of code that needs to be written. However, without a compelling end-to-end transformation language figured out I think any alternative database systems with their alternative information models (triple-based or otherwise) will struggle to compare favourably with mainstream databases, where declarative data munging with SQL is considered valuable and routine. Triples may well prove to be the best way to handle information in software over the long-term, but I'm not sure that the systems which currently work with triples are good enough or widespread enough to test that theory.

brap 4y ago

What makes people see Lisp-like languages and think to themselves "yep, this is how I want my code to look like" is beyond me

tmtvl 4y ago
I like the structure that S-expressions convey. Must be something with how my brain works. It's probably also why I find Python code utterly unreadable.
- brap 4y ago
  To each their own. There are obviously very smart people choosing Lisp so I’m not trying to knock it down or anything, I just never got the appeal Every now and then I try to check out a Lisp project and this is my reaction the moment I see the code: https://giphy.com/gifs/seinfeld-bye-jerry-106PwpLIIXJnXi https://giphy.com/gifs/seinfeld-bye-jerry-106PwpLIIXJnXi
  - vindarel 4y ago
    This is what we see: https://raw.githubusercontent.com/tarsius/paren-face/master/parentheses.png https://raw.githubusercontent.com/tarsius/paren-face/master/... Don't look at the parens and don't stop at the superficial look :]
  - dgb23 4y ago
    For me much of the appeal of the syntax comes from using it with a editor integrated REPL, or when writing macros, or when generating linter configuration from a domain model etc. However I agree with you on some level. Lisp code can easily be written in a way that’s hard to read. Specifically because of nesting expression way more often than is comfortable for me to read. I much rather prefer let bindings and threading so most of the code reads from left to right.
  - rawoke083600 4y ago
    >To each their own. There are obviously very smart people choosing Lisp so I’m not trying to knock it down or anything, I just never got the appeal Just know that almost no one STARTS off by liking Lisp-Syntax, but oh boy does it grow on you.

crabmusket 4y ago

Using the relational model for app data in memory is really interesting. Martin Fowler wrote about doing that as a way to get around the "object-relational mismatch" issue[1]. Richard Fabian describes "data-oriented design" as having a lot of overlap with the relational model[2]. ECSes becoming very popular in game engines are basically in-memory relational databases where "components" are "tables"[3]. [1]: https://martinfowler.com/bliki/OrmHate.html https://martinfowler.com/bliki/OrmHate.html [2]: https://dataorienteddesign.com/dodbook/ https://dataorienteddesign.com/dodbook/ [3]: https://github.com/SanderMertens/ecs-faq#what-is-ecs https://github.com/SanderMertens/ecs-faq#what-is-ecs

sitkack 4y ago
I'd love to see something like Relic able to operate over capnproto/arrow/parquet/avro/hdf5 etc along with a DAG based recomputation engine.
nextaccountic 4y ago
> "components" are "tables" No, components are columns. Each entity is a row in a table and an archetype (the set of all entities with the same components) is a table. In ECS it's usual to store components together, this corresponds to columnar storage in database terms (or equivalently, it's as if data were in sixth normal form [0]). However in some systems you can opt to store a more traditional row-based column. [0] https://en.wikipedia.org/wiki/Database_normalization#Satisfying_6NF https://en.wikipedia.org/wiki/Database_normalization#Satisfy...
- crabmusket 4y ago
  Thanks for the correction. In equating components and tables I was thinking of archetype-based engines, but even in that case they may store a whole set of components together in a single "table", right? Conceptually I think you could equate components with relations. For example a "grid location" component could be imagined as a relation with "entity ID" and "coordinate" columns. I suppose that admits multiple component instances per entity ID though which is typically not what you want.
- 4y ago

lincpa 4y ago

[dead]

tantaman 4y ago

Beautiful. I've been noodling on exactly this same idea. So far I've explored using SQLite but it would be ideal if there was no SQL between me and my relations. Another direction I've taken is adding STM to JavaScript and implementing a JS api for querying relations.