16 ms·

Coauthor of Graph here, I'm happy to answer any questions.

by w01fe 13y ago

sigil 13y ago
How does the parallel "execution strategy" work? Do you specify a maximum concurrency level? Can you distribute the computation onto other machines?
- w01fe 13y ago
  Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used. Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.
  - jared314 13y ago
    It would be very interesting if you could easily convert a graph into a Storm[0] topology. An example of the Storm Clojure DSL is available in the storm-starter project[1]. [0] http://storm-project.net/ http://storm-project.net/ [1] https://github.com/nathanmarz/storm-starter/blob/master/src/clj/storm/starter/clj/word_count.clj https://github.com/nathanmarz/storm-starter/blob/master/src/...
- batbomb 13y ago
  I'm not sure how it's implemented in Graph, but the best thing you can do is to abstract out an execution to a general object. From there, you can create all sorts of adapters and the "graph" engine just farms out the execution to the appropriate adapter. This could be anything from a job daemon on a batch farm, or, say, a typed ThreadPoolExecutor on a machine (which can either execute native clojure/java, or any sort of other script language.
  - thelittlelisper 13y ago
    This would be brilliant. Does any other framework support this? (outside Clojure)
cliffbean 13y ago
It's not clear to me how Graph is better than just regular pure functional programming. In theory, you could implement graph/par-compile for regular functions by examining the dependence graph in their code, right? Is the special-purpose declarative syntax useful in its own right?
- w01fe 13y ago
  Short answer: because Graphs are data, it's easy to do tons of things with them that are difficult to do with code. In principle tooling may eventually bridge the gap, but for now it's hard to take a function and automatically monitor it's sub-functions, or run up to a particular intermediate result, or substitute one step for another in a test, etc. Our previous blog post gives some more detailed examples: http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph-at-strange-loop.html http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...
  - cliffbean 13y ago
    You're using LISP, so code is data, right? And if all your functions are pure, the code should have a nice dependence-graph structure. I can believe that the actual LISP data structure of a function may not be the most convenient to work with for what you're doing, but it seems like you ought to be able to translate from LISP code into whatever graph structure you want, as long as all the function calls are pure. Or are there reasons this isn't feasible?
    - w01fe 13y ago
      Sure, analyzing code into an AST is easier in LISP (trivial, even). But you don't necessarily want to monitor every sub-function call within your function, because of performance overhead, and to limit noise. And if you want to sub out a step, the AST is not the most natural data structure to work with. Graph forces you to make the steps that you care about explicit, and in exchange you get a nice way to observe, reason about, and change your code in terms of these steps. The goal is to make the overall process as clear and non-magical as possible, while incurring as little programmer overhead as possible. I think it's a really interesting project to attempt to provide similar tools over ordinary functions, but that seems like a much loftier goal -- Graph is pragmatic, simple, and it works now :).
taliesinb 13y ago
Beyond using graphviz on your topology, can you interactively visualize the results flowing along each edge? What does a "graph backtrace" look like? How do you do logging -- does something like Zipkin help? I feel like we still don't know what more traditional tools and workflows look like from the graph point-of-view. Edit: cool project, btw.
- w01fe 13y ago
  Re: interactive visualizations, we haven't gotten there yet, but it sounds like a really cool (and feasible) idea. On this front, libraries like 'lamina' look like a nice place to start. https://github.com/ztellman/lamina https://github.com/ztellman/lamina Graph backtraces look like ordinary stacktraces -- the compiled output is basically the same if you wrote the function by hand. For logging, we wrap each node in an 'observer' with the path through the graph injected, which automatically records execution time and exceptions from each node, and lets you spit stuff out to the dashboard that will appear in the graph structure. There's an example of this in the graph_examples_test.clj, I believe.

Coauthor of Graph here, I'm happy to answer any questions.

by w01fe 13y ago

Coauthor of Graph here, I'm happy to answer any questions.

sigil 13y ago

How does the parallel "execution strategy" work? Do you specify a maximum concurrency level? Can you distribute the computation onto other machines?

w01fe 13y ago
Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used. Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.
- jared314 13y ago
  It would be very interesting if you could easily convert a graph into a Storm[0] topology. An example of the Storm Clojure DSL is available in the storm-starter project[1]. [0] http://storm-project.net/ http://storm-project.net/ [1] https://github.com/nathanmarz/storm-starter/blob/master/src/clj/storm/starter/clj/word_count.clj https://github.com/nathanmarz/storm-starter/blob/master/src/...
batbomb 13y ago
I'm not sure how it's implemented in Graph, but the best thing you can do is to abstract out an execution to a general object. From there, you can create all sorts of adapters and the "graph" engine just farms out the execution to the appropriate adapter. This could be anything from a job daemon on a batch farm, or, say, a typed ThreadPoolExecutor on a machine (which can either execute native clojure/java, or any sort of other script language.
- thelittlelisper 13y ago
  This would be brilliant. Does any other framework support this? (outside Clojure)

cliffbean 13y ago

It's not clear to me how Graph is better than just regular pure functional programming. In theory, you could implement graph/par-compile for regular functions by examining the dependence graph in their code, right? Is the special-purpose declarative syntax useful in its own right?

w01fe 13y ago
Short answer: because Graphs are data, it's easy to do tons of things with them that are difficult to do with code. In principle tooling may eventually bridge the gap, but for now it's hard to take a function and automatically monitor it's sub-functions, or run up to a particular intermediate result, or substitute one step for another in a test, etc. Our previous blog post gives some more detailed examples: http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph-at-strange-loop.html http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...
- cliffbean 13y ago
  You're using LISP, so code is data, right? And if all your functions are pure, the code should have a nice dependence-graph structure. I can believe that the actual LISP data structure of a function may not be the most convenient to work with for what you're doing, but it seems like you ought to be able to translate from LISP code into whatever graph structure you want, as long as all the function calls are pure. Or are there reasons this isn't feasible?
  - w01fe 13y ago
    Sure, analyzing code into an AST is easier in LISP (trivial, even). But you don't necessarily want to monitor every sub-function call within your function, because of performance overhead, and to limit noise. And if you want to sub out a step, the AST is not the most natural data structure to work with. Graph forces you to make the steps that you care about explicit, and in exchange you get a nice way to observe, reason about, and change your code in terms of these steps. The goal is to make the overall process as clear and non-magical as possible, while incurring as little programmer overhead as possible. I think it's a really interesting project to attempt to provide similar tools over ordinary functions, but that seems like a much loftier goal -- Graph is pragmatic, simple, and it works now :).

taliesinb 13y ago

Beyond using graphviz on your topology, can you interactively visualize the results flowing along each edge? What does a "graph backtrace" look like? How do you do logging -- does something like Zipkin help? I feel like we still don't know what more traditional tools and workflows look like from the graph point-of-view. Edit: cool project, btw.

w01fe 13y ago
Re: interactive visualizations, we haven't gotten there yet, but it sounds like a really cool (and feasible) idea. On this front, libraries like 'lamina' look like a nice place to start. https://github.com/ztellman/lamina https://github.com/ztellman/lamina Graph backtraces look like ordinary stacktraces -- the compiled output is basically the same if you wrote the function by hand. For logging, we wrap each node in an 'observer' with the path through the graph injected, which automatically records execution time and exceptions from each node, and lets you spit stuff out to the dashboard that will appear in the graph structure. There's an example of this in the graph_examples_test.clj, I believe.