3 ms·

We are well aware of the Beaker Notebook and BeakerX and appreciate the great work they are doing. We however decided to develop our own tool because of a few s

by bpeng2000 9y ago

We are well aware of the Beaker Notebook and BeakerX and appreciate the great work they are doing. We however decided to develop our own tool because of a few specific reasons such as we needed support for MATLAB and SAS, we needed a more powerful data exchange model for (almost) arbitrary data types, and most importantly, we were creating an interactive data analysis environment backed by a powerful workflow engine, which is well beyond the scope of Beaker. The combination of SoS (workflow engine) and SoS Notebook is what makes SoS Notebook a powerful environment for (bioinformatics) data analysis. I agree that all other notebooks (e.g. BeakerX, Zeppelin, R Notebook) have nicer interface than Jupyter but Jupyter excels at its simplicity and JupyterLab (to which SoS Notebook will be ported eventually) is making great progress there. With the frontend enhancement that SoS Notebook provides, especially the line-by-line execution feature, we are pretty satisfied with the frontend and do not really miss the fancy frontend of other notebooks.

zmmmmm 9y ago
That's a really interesting idea about backing it with a workflow engine. I can't recall something that did that before - though there are obviously plenty of workflow engines for python and other languages. Definitely interesting to look at. Good luck!
- bpeng2000 9y ago
  Yes, multi-language notebooks solve the "multi-language" but not the "large-scale" problems with bioinformatics (or data science) data analysis. However powerful other notebook environments can be, they are rather limited if they can only execute the notebooks on a single machine. However powerful other workflow systems can be, they are counterproductive if they require you to develop workflows in another environment and in another language. Backing up SoS Notebook with the SoS workflow engine provides a single environment for both interactive data analysis and the development and execution of workflows. This topic definitely worths a separate blog post so I will just list a few features that SoS enables here: 1. Extended from Python 3.6 to make SoS an easy and yet powerful workflow language. 2. Embedding workflows in SoS Notebooks allows you to annotate the workflows with detailed descriptions (markdown cells) and results (of demo runs). 3. Supports both forward (sequentially numbered) and makefile style (patten matching) workflows. 4. Execution signatures to avoid re-execution of long steps. 5. Magics to execute workflows in SoS Notebooks so you can, for example, execute cells of a notebook conditionally and repeatedly. 6. A task system that sends parts of workflows to remote host (a more powerful workstation), cluster (PBS/Torque/LFS/Slurm systems), or RQ task queue, even if the remote systems are on different file systems (SoS automatically maps paths and synchronize files).

zmmmmm 9y ago

That's a really interesting idea about backing it with a workflow engine. I can't recall something that did that before - though there are obviously plenty of workflow engines for python and other languages. Definitely interesting to look at. Good luck!

bpeng2000 9y ago
Yes, multi-language notebooks solve the "multi-language" but not the "large-scale" problems with bioinformatics (or data science) data analysis. However powerful other notebook environments can be, they are rather limited if they can only execute the notebooks on a single machine. However powerful other workflow systems can be, they are counterproductive if they require you to develop workflows in another environment and in another language. Backing up SoS Notebook with the SoS workflow engine provides a single environment for both interactive data analysis and the development and execution of workflows. This topic definitely worths a separate blog post so I will just list a few features that SoS enables here: 1. Extended from Python 3.6 to make SoS an easy and yet powerful workflow language. 2. Embedding workflows in SoS Notebooks allows you to annotate the workflows with detailed descriptions (markdown cells) and results (of demo runs). 3. Supports both forward (sequentially numbered) and makefile style (patten matching) workflows. 4. Execution signatures to avoid re-execution of long steps. 5. Magics to execute workflows in SoS Notebooks so you can, for example, execute cells of a notebook conditionally and repeatedly. 6. A task system that sends parts of workflows to remote host (a more powerful workstation), cluster (PBS/Torque/LFS/Slurm systems), or RQ task queue, even if the remote systems are on different file systems (SoS automatically maps paths and synchronize files).