3 ms·
I hadn't seen Plynx before, and indeed it does look quite similar. Something that I believe we do that I don't see with Plynx is our support for "debugging" pi
by ricklamers 6y ago
I hadn't seen Plynx before, and indeed it does look quite similar.
Something that I believe we do that I don't see with Plynx is our support for "debugging" pipeline in realtime through extensive Jupyter integration.
When a pipeline step is a Jupyter notebook, you can execute previous steps in the pipeline (partially execute the pipeline graph). It passes data to the running notebook kernel and which allows you to explore it interactively. This makes it a lot easier to guarantee your incoming data matches your expectations.
We also put each step in a pipeline in a separate container (with its own image), which will simplify a lot of things when taking the effort to productionize pipelines to make them suitable for horizontal scale-out.
Edit: running locally can be useful if you want to use your own resources (i.e. your own GPU-rack) to run compute heavy data science pipelines. But we envision for most real-world team usage they'll want to run in the cloud. We're working on making it run directly on top of cloud provided Kubernetes engines.