6 ms·

Biff.graph: structure your Clojure codebase as a queryable graph

147 points by jacobobryant 3mo ago

tosti 3mo ago
The name suggests it plots graphs related to incoming email.
adamfeldman 3mo ago
There are a few great talks on YouTube by the creator of Pathom explaining the value of the graph: https://m.youtube.com/watch?v=IS3i3DTUnAI https://m.youtube.com/watch?v=IS3i3DTUnAI
- geokon 3mo ago
  I've been rewriting a codebase using Pathom resolvers and it has been extremely fun and has made me really reexamine how I organize code. Without being hyperbolic, it's really a new coding paradigm. You get some extreme decoupling and it allows the engine to automatically maximizes concurrency.
  - jacobobryant 3mo ago
    Yeah, it's a big eye-opener. I'd like to see if I can figure out an ergonomic way to do it in Python since I do a fair amount of work in that, and passing ORM objects around isn't great.
    - brianritz 3mo ago
      I have an old repo that explores the concept at bmritz/datajet. I’ve also toyed around with the idea of using type hints and type aliases in python as the “data key” (equivalent to :user/id). Would love to have something equivalent in python.
      - jacobobryant 3mo ago
        Thanks for mentioning datajet, I'll be taking a look at that for sure...
chamomeal 3mo ago
> biff.graph is basically a lightweight version of Pathom. It implements only a subset of Pathom's functionality with the intention of being easier to understand. I’ve heard of pathom but I’ve never actually dove in and tried it out. It sounds super neato though. I have a bunch of microservice DB’s (that should really just be on DB, but I think we’re created in the peak of microservices hype). I def need a better way to explore the data. “Easier to understand” sounds sick and I think I’ll check it out this week!
Capricorn2481 3mo ago
My experience with Pathom, and other graph query libraries, is it feels like a deliberately confusing way to reason about a program. I'd like to know your thoughts on it. From what I hear, the main draw is separating what you want from how you get it, so your calling code can just focus on what it needs. But you can use regular functions to do that. What libraries like Pathom do is leave it open to the caller what shape of data they need. But I think letting the caller do subtle query changes that can completely change which resolvers are triggered and how something is fetched is kinda leaky. How do you write the perfect resolver for all situations? How do you keep them from accidentally exploding their fetches? Is it not better to have things be explicit through function calls instead of chasing down disjointed call graphs?
- jacobobryant 3mo ago
  > From what I hear, the main draw is separating what you want from how you get it, so your calling code can just focus on what it needs. But you can use regular functions to do that. What libraries like Pathom do is leave it open to the caller what shape of data they need. hmmm... it would be interesting to try an approach where you make heavy use of memoization and then write your functions to take the the minimal set of inputs (e.g. just the primary key for a record). I'm not sure if that's exactly what you had in mind, but here's a strawman example: ;; instead of having a resolver with this input {:input [:person/age :person/name {:person/pet [:pet/species :pet/n-legs]}]} ;; you could have this plain function which calls regular functions to get its ;; input, each of which only need a single entity ID for their input (defn get-person-stuff [db person-id] (let [age (get-person-age db person-id) name (get-person-name db person-id) pet-id (get-person-pet db person-id) pet-species (get-pet-species db pet-id) pet-n-legs (get-pet-n-legs db pet-id)] ...)) And you know, I think that would be workable, even though it feels more boilerplatey to me. It would still get you the main benefit of not having to keep track of all the data shapes that are needed by the functions you're calling etc. Some off-the-cuff thoughts: - with this approach you have a single function for each attribute, so you don't have the situation with pathom/biff.graph where there are multiple resolvers that could be called to get a particular attribute. However note that you could always put an assertion in your codebase that ensures no two resolvers share the same output key, which would then also give you the ability to know exactly what resolvers are being called. - my example above doesn't include optional inputs, so that's logic you'd also need to write into all your functions: don't fetch the pet data if the pet ID is nil, don't return anything if the person name is nil, etc. - if you do all that with regular code instead of dependency injection, that does mean you have more code to test, and you have to either supply a test DB (and populate it with everything the functions you're calling need) or mock out the functions. With the dependency injection approach you get plain-old-pure-functions which helps keep your unit tests nice and dumb. - I like the readability of being able to look at the input / output queries and know exactly what shape of data I'm dealing with. - There might be performance issues with the memoized functions approach. Pathom and biff.graph both support batch resolvers for example, and I'm not sure if you could do the equivalent as cleanly with the functions approach. And Pathom of course has its additional query planning step which does... stuff. Going back to your comment, some thoughts: > But I think letting the caller do subtle query changes that can completely change which resolvers are triggered and how something is fetched is kinda leaky. This is an area where you might like biff.graph more than Pathom. Since there's no query planning step, the way that biff.graph executes your queries should be fairly predictable. It's basically just doing a depth-first traversal of your query. (My first bullet point above is relevant too--you can always restrict yourself to having only one resolver per attribute so there's no question of what resolver is getting used.) > How do you write the perfect resolver for all situations? How do you keep them from accidentally exploding their fetches? Typically you write resolvers with only one level of joins/nesting and then let the query engine do the rest. so e.g. instead of writing a resolver that returns something like `{:person/pet {:pet/id 1, :pet/toys [{:toy/id 2, ...}, ...]}}`, you would have one resolver that returns `{:person/pet {:pet/id 1}}` and then another resolver that takes a pet ID and returns `{:pet/toys [{:toy/id 2}, ...]}` etc. So there is a trade-off here in that e.g. you may end up running multiple database queries even though you could've stuffed everything you need into a single database query. That is mitigated by batch resolvers at least so you don't get N+1 query problems. I've never needed to do this myself yet, but if you do run into any places where the performance isn't good enough, you can always write those bits the regular way (e.g. have a resolver that does a more complex query and returns nested data and/or don't even use pathom/biff.graph for this one bit). i.e. optimize where needed but stick with the default in most places. > Is it not better to have things be explicit through function calls instead of chasing down disjointed call graphs? There are pros and cons I think. Sometimes you want to know how an input is being computed and sometimes you want to be able to understand some logic in isolation. In practice I've acclimated quite a bit to the graph structure; I feel like it does a nice job of helping you split your code into the right "chunks".
geokon 3mo ago
It's very cool you managed to make a mini Pathom - esp in so few lines of code :)) But the end result looks almost identical? Resolver declarations are a bit reorganized and look a bit cleaner - though you could do that with a wrapper around Pathom. Why not fork Pathom and just make some QOL adjustments? If you are okay only having a subset, then I think you could streamline Pathom quite a bit more.. Off the top of my head: - adding and "registering" resolvers in to an environment is always annoying. To me this feels like it should be abstracted away and you should never has to manage this stuff. Each time you add a resolver you have to copy the new resolver name, scroll down to the bottom of the file and paste it in to the resolver list. Stale resolvers floating around in the ns and forgetting to register resolvers regularly leads to weird scenarios where you're wondering why something isn't resolving. I think the user shouldn't really have to think about any of this.. the environment should be constructed automatically by the engine. Scan the namespace and register ever resolver. I get that it'd be not as flexible this way.. but I've never had to make multiple environments in one namespace. - You should be able to safely reregister resolvers. This happens when you try to decouple sub-systems that depend on common resolvers. Ex: I have some utility resolvers that do some format conversions. If I add them to the environments of two namespaces, then those two namespaces can't be registered in a parent namespace. It's not a dealbreaker, you just register all your ns environments all the top-level and do all your queries there. But you can't add inline test queries in these lower level namespaces and it sort of breaks the decoupling (esp if it's across library boundaries). - The Pathom errors are actually pretty good once you know how to read them (but there is a ton of visual noise). My guess is it's going to be a challenge to get to the same level in a rewrite. It feels like there is room for improvement here, but I don't have concrete ideas. Maybe a ASCII diagram of the chain of missing keys? There is Pathom-Viz, but from what I understand it doesn't handle nested queries (which in a not-toy program will be basically all your queries)
- jacobobryant 3mo ago
  > It's very cool you managed to make a mini Pathom - esp in so few lines of code :)) Thanks! The possibility of doing this had been on my mind for a while... and then I finally got around to trying it since all I had to do to get started was say "try making something like pathom but without [...]". I actually have all the prompts and feedback for the initial POC over here[1] since at the time I was using github issues/comments for my LLM-driven-development workflow. Over the past few weeks as prep for release I went over all the code manually (especially since the whole point of this thing is for the implementation to be easy to understand) and basically rewrote the whole thing, or at least that's what it felt like. > But the end result looks almost identical? Resolver declarations are a bit reorganized and look a bit cleaner - though you could do that with a wrapper around Pathom. Why not fork Pathom and just make some QOL adjustments? The main thing I was going for was just to reduce the implementation size; the tweaks I made to e.g. `defresolver` were really just a side thing. To give some more background on the motivations, an issue I've had sometimes with Pathom is figuring out what's going wrong when my queries don't give me the results I'd expect. A few times as part of that I've gone spelunking through the Pathom codebase but still had never built up a complete understanding of how the query planning and execution works, which has meant that my debugging has always been more trial-and-error / black-box than I'd prefer. So I wanted to see "what is the least complex way that I could take an EQL query and figure out the results, even if the way I do it is dumber than the way Pathom does it?" i.e. I'm trying to minimize the amount of time it takes for someone to read the code and understand exactly what's going on under the hood. Hence layering more code on top of Pathom would only hinder that goal. [1] https://github.com/jacobobryant/biff.graph/issues?q=is%3Aissue%20state%3Aclosed https://github.com/jacobobryant/biff.graph/issues?q=is%3Aiss...
winterissnowing 3mo ago
[flagged]
arikrahman 3mo ago
Amazing project, getting into Clojure and will no doubt have use for this as a solo dev

147 points by jacobobryant 3mo ago

tosti 3mo ago

The name suggests it plots graphs related to incoming email.

There are a few great talks on YouTube by the creator of Pathom explaining the value of the graph: https://m.youtube.com/watch?v=IS3i3DTUnAI https://m.youtube.com/watch?v=IS3i3DTUnAI

geokon 3mo ago
I've been rewriting a codebase using Pathom resolvers and it has been extremely fun and has made me really reexamine how I organize code. Without being hyperbolic, it's really a new coding paradigm. You get some extreme decoupling and it allows the engine to automatically maximizes concurrency.
- jacobobryant 3mo ago
  Yeah, it's a big eye-opener. I'd like to see if I can figure out an ergonomic way to do it in Python since I do a fair amount of work in that, and passing ORM objects around isn't great.
  - brianritz 3mo ago
    I have an old repo that explores the concept at bmritz/datajet. I’ve also toyed around with the idea of using type hints and type aliases in python as the “data key” (equivalent to :user/id). Would love to have something equivalent in python.
    - jacobobryant 3mo ago
      Thanks for mentioning datajet, I'll be taking a look at that for sure...

chamomeal 3mo ago

> biff.graph is basically a lightweight version of Pathom. It implements only a subset of Pathom's functionality with the intention of being easier to understand. I’ve heard of pathom but I’ve never actually dove in and tried it out. It sounds super neato though. I have a bunch of microservice DB’s (that should really just be on DB, but I think we’re created in the peak of microservices hype). I def need a better way to explore the data. “Easier to understand” sounds sick and I think I’ll check it out this week!

Capricorn2481 3mo ago

My experience with Pathom, and other graph query libraries, is it feels like a deliberately confusing way to reason about a program. I'd like to know your thoughts on it. From what I hear, the main draw is separating what you want from how you get it, so your calling code can just focus on what it needs. But you can use regular functions to do that. What libraries like Pathom do is leave it open to the caller what shape of data they need. But I think letting the caller do subtle query changes that can completely change which resolvers are triggered and how something is fetched is kinda leaky. How do you write the perfect resolver for all situations? How do you keep them from accidentally exploding their fetches? Is it not better to have things be explicit through function calls instead of chasing down disjointed call graphs?

jacobobryant 3mo ago
> From what I hear, the main draw is separating what you want from how you get it, so your calling code can just focus on what it needs. But you can use regular functions to do that. What libraries like Pathom do is leave it open to the caller what shape of data they need. hmmm... it would be interesting to try an approach where you make heavy use of memoization and then write your functions to take the the minimal set of inputs (e.g. just the primary key for a record). I'm not sure if that's exactly what you had in mind, but here's a strawman example: ;; instead of having a resolver with this input {:input [:person/age :person/name {:person/pet [:pet/species :pet/n-legs]}]} ;; you could have this plain function which calls regular functions to get its ;; input, each of which only need a single entity ID for their input (defn get-person-stuff [db person-id] (let [age (get-person-age db person-id) name (get-person-name db person-id) pet-id (get-person-pet db person-id) pet-species (get-pet-species db pet-id) pet-n-legs (get-pet-n-legs db pet-id)] ...)) And you know, I think that would be workable, even though it feels more boilerplatey to me. It would still get you the main benefit of not having to keep track of all the data shapes that are needed by the functions you're calling etc. Some off-the-cuff thoughts: - with this approach you have a single function for each attribute, so you don't have the situation with pathom/biff.graph where there are multiple resolvers that could be called to get a particular attribute. However note that you could always put an assertion in your codebase that ensures no two resolvers share the same output key, which would then also give you the ability to know exactly what resolvers are being called. - my example above doesn't include optional inputs, so that's logic you'd also need to write into all your functions: don't fetch the pet data if the pet ID is nil, don't return anything if the person name is nil, etc. - if you do all that with regular code instead of dependency injection, that does mean you have more code to test, and you have to either supply a test DB (and populate it with everything the functions you're calling need) or mock out the functions. With the dependency injection approach you get plain-old-pure-functions which helps keep your unit tests nice and dumb. - I like the readability of being able to look at the input / output queries and know exactly what shape of data I'm dealing with. - There might be performance issues with the memoized functions approach. Pathom and biff.graph both support batch resolvers for example, and I'm not sure if you could do the equivalent as cleanly with the functions approach. And Pathom of course has its additional query planning step which does... stuff. Going back to your comment, some thoughts: > But I think letting the caller do subtle query changes that can completely change which resolvers are triggered and how something is fetched is kinda leaky. This is an area where you might like biff.graph more than Pathom. Since there's no query planning step, the way that biff.graph executes your queries should be fairly predictable. It's basically just doing a depth-first traversal of your query. (My first bullet point above is relevant too--you can always restrict yourself to having only one resolver per attribute so there's no question of what resolver is getting used.) > How do you write the perfect resolver for all situations? How do you keep them from accidentally exploding their fetches? Typically you write resolvers with only one level of joins/nesting and then let the query engine do the rest. so e.g. instead of writing a resolver that returns something like `{:person/pet {:pet/id 1, :pet/toys [{:toy/id 2, ...}, ...]}}`, you would have one resolver that returns `{:person/pet {:pet/id 1}}` and then another resolver that takes a pet ID and returns `{:pet/toys [{:toy/id 2}, ...]}` etc. So there is a trade-off here in that e.g. you may end up running multiple database queries even though you could've stuffed everything you need into a single database query. That is mitigated by batch resolvers at least so you don't get N+1 query problems. I've never needed to do this myself yet, but if you do run into any places where the performance isn't good enough, you can always write those bits the regular way (e.g. have a resolver that does a more complex query and returns nested data and/or don't even use pathom/biff.graph for this one bit). i.e. optimize where needed but stick with the default in most places. > Is it not better to have things be explicit through function calls instead of chasing down disjointed call graphs? There are pros and cons I think. Sometimes you want to know how an input is being computed and sometimes you want to be able to understand some logic in isolation. In practice I've acclimated quite a bit to the graph structure; I feel like it does a nice job of helping you split your code into the right "chunks".

geokon 3mo ago

It's very cool you managed to make a mini Pathom - esp in so few lines of code :)) But the end result looks almost identical? Resolver declarations are a bit reorganized and look a bit cleaner - though you could do that with a wrapper around Pathom. Why not fork Pathom and just make some QOL adjustments? If you are okay only having a subset, then I think you could streamline Pathom quite a bit more.. Off the top of my head: - adding and "registering" resolvers in to an environment is always annoying. To me this feels like it should be abstracted away and you should never has to manage this stuff. Each time you add a resolver you have to copy the new resolver name, scroll down to the bottom of the file and paste it in to the resolver list. Stale resolvers floating around in the ns and forgetting to register resolvers regularly leads to weird scenarios where you're wondering why something isn't resolving. I think the user shouldn't really have to think about any of this.. the environment should be constructed automatically by the engine. Scan the namespace and register ever resolver. I get that it'd be not as flexible this way.. but I've never had to make multiple environments in one namespace. - You should be able to safely reregister resolvers. This happens when you try to decouple sub-systems that depend on common resolvers. Ex: I have some utility resolvers that do some format conversions. If I add them to the environments of two namespaces, then those two namespaces can't be registered in a parent namespace. It's not a dealbreaker, you just register all your ns environments all the top-level and do all your queries there. But you can't add inline test queries in these lower level namespaces and it sort of breaks the decoupling (esp if it's across library boundaries). - The Pathom errors are actually pretty good once you know how to read them (but there is a ton of visual noise). My guess is it's going to be a challenge to get to the same level in a rewrite. It feels like there is room for improvement here, but I don't have concrete ideas. Maybe a ASCII diagram of the chain of missing keys? There is Pathom-Viz, but from what I understand it doesn't handle nested queries (which in a not-toy program will be basically all your queries)

jacobobryant 3mo ago
> It's very cool you managed to make a mini Pathom - esp in so few lines of code :)) Thanks! The possibility of doing this had been on my mind for a while... and then I finally got around to trying it since all I had to do to get started was say "try making something like pathom but without [...]". I actually have all the prompts and feedback for the initial POC over here[1] since at the time I was using github issues/comments for my LLM-driven-development workflow. Over the past few weeks as prep for release I went over all the code manually (especially since the whole point of this thing is for the implementation to be easy to understand) and basically rewrote the whole thing, or at least that's what it felt like. > But the end result looks almost identical? Resolver declarations are a bit reorganized and look a bit cleaner - though you could do that with a wrapper around Pathom. Why not fork Pathom and just make some QOL adjustments? The main thing I was going for was just to reduce the implementation size; the tweaks I made to e.g. `defresolver` were really just a side thing. To give some more background on the motivations, an issue I've had sometimes with Pathom is figuring out what's going wrong when my queries don't give me the results I'd expect. A few times as part of that I've gone spelunking through the Pathom codebase but still had never built up a complete understanding of how the query planning and execution works, which has meant that my debugging has always been more trial-and-error / black-box than I'd prefer. So I wanted to see "what is the least complex way that I could take an EQL query and figure out the results, even if the way I do it is dumber than the way Pathom does it?" i.e. I'm trying to minimize the amount of time it takes for someone to read the code and understand exactly what's going on under the hood. Hence layering more code on top of Pathom would only hinder that goal. [1] https://github.com/jacobobryant/biff.graph/issues?q=is%3Aissue%20state%3Aclosed https://github.com/jacobobryant/biff.graph/issues?q=is%3Aiss...

winterissnowing 3mo ago

[flagged]

arikrahman 3mo ago

Amazing project, getting into Clojure and will no doubt have use for this as a solo dev