3 ms·

This sounds suboptimal. Just write code. If you need a DAG, parse the code using an abstract syntax tree or whatever.

alangpierce 9y ago
Experimental design definitely has some parallels to code structure (the control flow graph, not the AST), but I think most real-world experiments would be awkward to model as code in a typical programming language. Some examples of things that I don't think translate super-well: arbitrary-fan-out forks and joins, complex decisions on when a process is "done", ad-hoc changes in the middle of the experiment, and experimental condition metadata on each branch. Currently, we have a visual UI for defining all of this, which is particularly nice for scientists without a programming background, but it's certainly possible that we'd have some sort of description language (maybe something like Makefile) in the future. And, of course, you could always write custom code to generate the graph and define it using our API (which isn't the same as the graph being derived from your code's control flow). Also, like I mentioned with configuration vs execution graphs, more interesting than the "code" is the "execution trace". One of the main goals here is to keep a team of scientists in the loop on how a complex experiment is going. When you have hundreds or thousands of samples undergoing different treatments and analyses in parallel, it's not at all obvious how to visually represent what's going on, especially if you need to account for errors and ad-hoc changes that happen while the experiment is going on. In a certain sense, the "code" to define the high-level ideal structure is the easy part.
- elcritch 9y ago
  Spot on! Had these same thoughts trying to write my own mini data processing workflow. I think every scientist faces it at some point if they do any more Complex experiments. Most like my previous advisor solved it largely through simple excel sheets and discipline. Solving it with code becomes about as comprehensible as a adhoc build system. Out of curiosity, does your UI allow any form of "simulation", particularly a random fuzzing/monte carlo trials? Having the ability to simulate how a complex rule set plays out can help weed out errors before starting real experiments. P.S. I admire the work y'all are doing at benchling! Bio-sciences are surprisingly anachronistic when it comes to performing research.
  - alangpierce 9y ago
    > Out of curiosity, does your UI allow any form of "simulation", particularly a random fuzzing/monte carlo trials? No, at the moment we don't really have enough structured knowledge of the underlying steps to be able to do predictions/simulations like that. We're taking existing scientific workflows and adding structure to them, but there are still plenty of details that are unstructured and performed manually by scientists, and there's a lot of variability across use cases. Monte carlo trials certainly seems interesting, though, and hopefully something we can explore more as the procedures get more structured. A similar idea that's a bit closer in reach is to do an analysis on past variations of the same experiment to predict how a newly-designed experiment will go. Possibly we could do a simulation based on historical data like that, although modeling it correctly certainly sounds tricky.
    - elcritch 9y ago
      > We're taking existing scientific workflows and adding structure to them, but there are still plenty of details that are unstructured and performed manually by scientists, and there's a lot of variability across use cases. That makes sense. It’s difficult to start adding structure to biological sciences. And to the fields credit I don’t think it’s out of lack of ability or effort but just the true complexity of dealing with biological systems. In contrast, even quantum mechanics is seemingly simple! > Possibly we could do a simulation based approach on historical data like that, although modeling it would correctly certainly sounds tricky. True, seems like you will have a very valuable historical dataset though. I’ve been pondering this type of problem for a few years since my (abortive) attempt at a PhD in material sciences. For my project I made some decent progress using various Bayesian and bootstrapping models to work with uncertainty in my models and tissue samples. Bayesian approaches, especially combined with MCMC type analysis can yield a lot of fruit dealing with biological systems with minimal need to understand the underlying models. But as you mention modeling the parameters correctly is challenging and there’s a lacunae in research broaching the topic. But at the least a system like yours could indicate items like conditional probability of failure given certain sequence sets or combinations of experimental procedures (e.g. cell lines A tend to do better when cultured in X media and tested with Y +/- Z hours). The article seems to indicate you’d have most of the data described in a format that’d be amendable to such data mining.
- tedmiston 9y ago
  Just curious if you guys have experimented with anything like Apache Airflow where the core component is DAGs structured as simple code files using components for each node?

Experimental design definitely has some parallels to code structure (the control flow graph, not the AST), but I think most real-world experiments would be awkward to model as code in a typical programming language. Some examples of things that I don't think translate super-well: arbitrary-fan-out forks and joins, complex decisions on when a process is "done", ad-hoc changes in the middle of the experiment, and experimental condition metadata on each branch. Currently, we have a visual UI for defining all of this, which is particularly nice for scientists without a programming background, but it's certainly possible that we'd have some sort of description language (maybe something like Makefile) in the future. And, of course, you could always write custom code to generate the graph and define it using our API (which isn't the same as the graph being derived from your code's control flow). Also, like I mentioned with configuration vs execution graphs, more interesting than the "code" is the "execution trace". One of the main goals here is to keep a team of scientists in the loop on how a complex experiment is going. When you have hundreds or thousands of samples undergoing different treatments and analyses in parallel, it's not at all obvious how to visually represent what's going on, especially if you need to account for errors and ad-hoc changes that happen while the experiment is going on. In a certain sense, the "code" to define the high-level ideal structure is the easy part.

elcritch 9y ago
Spot on! Had these same thoughts trying to write my own mini data processing workflow. I think every scientist faces it at some point if they do any more Complex experiments. Most like my previous advisor solved it largely through simple excel sheets and discipline. Solving it with code becomes about as comprehensible as a adhoc build system. Out of curiosity, does your UI allow any form of "simulation", particularly a random fuzzing/monte carlo trials? Having the ability to simulate how a complex rule set plays out can help weed out errors before starting real experiments. P.S. I admire the work y'all are doing at benchling! Bio-sciences are surprisingly anachronistic when it comes to performing research.
- alangpierce 9y ago
  > Out of curiosity, does your UI allow any form of "simulation", particularly a random fuzzing/monte carlo trials? No, at the moment we don't really have enough structured knowledge of the underlying steps to be able to do predictions/simulations like that. We're taking existing scientific workflows and adding structure to them, but there are still plenty of details that are unstructured and performed manually by scientists, and there's a lot of variability across use cases. Monte carlo trials certainly seems interesting, though, and hopefully something we can explore more as the procedures get more structured. A similar idea that's a bit closer in reach is to do an analysis on past variations of the same experiment to predict how a newly-designed experiment will go. Possibly we could do a simulation based on historical data like that, although modeling it correctly certainly sounds tricky.
  - elcritch 9y ago
    > We're taking existing scientific workflows and adding structure to them, but there are still plenty of details that are unstructured and performed manually by scientists, and there's a lot of variability across use cases. That makes sense. It’s difficult to start adding structure to biological sciences. And to the fields credit I don’t think it’s out of lack of ability or effort but just the true complexity of dealing with biological systems. In contrast, even quantum mechanics is seemingly simple! > Possibly we could do a simulation based approach on historical data like that, although modeling it would correctly certainly sounds tricky. True, seems like you will have a very valuable historical dataset though. I’ve been pondering this type of problem for a few years since my (abortive) attempt at a PhD in material sciences. For my project I made some decent progress using various Bayesian and bootstrapping models to work with uncertainty in my models and tissue samples. Bayesian approaches, especially combined with MCMC type analysis can yield a lot of fruit dealing with biological systems with minimal need to understand the underlying models. But as you mention modeling the parameters correctly is challenging and there’s a lacunae in research broaching the topic. But at the least a system like yours could indicate items like conditional probability of failure given certain sequence sets or combinations of experimental procedures (e.g. cell lines A tend to do better when cultured in X media and tested with Y +/- Z hours). The article seems to indicate you’d have most of the data described in a format that’d be amendable to such data mining.
tedmiston 9y ago
Just curious if you guys have experimented with anything like Apache Airflow where the core component is DAGs structured as simple code files using components for each node?