4 ms·

This looks like an excellent start but the problem remains: who pays for and does maintenance (can’t publish it)? Who handles the gui (can’t publish that eithe

This looks like an excellent start but the problem remains: who pays for and does maintenance (can’t publish it)? Who handles the gui (can’t publish that either)? Without something resembling the money stream of a commercial company, it’s going to remain dependent on a very small number of people. This kind of work goes back to at least the 90’s when O’Reilly published several books on it. Top-level code in things like Python increase the flexibility of high performance code while writing entirely in an interpretive language has advantages for procedural code. It would also help if the professional organizations in the pertinent areas agree on file and data structures and api to avoid looking like chemistry with regards to the raft of file formats and standards.

maximsmol 5y ago
Full disclosure: I am a founding engineer at this company. We are charging corporate clients for white glove service and minor contracting (helping them upload and adjust pipelines). We also take a small margin over AWS compute prices for regular users. This lets us support the academic and small volume users for free (we give $100 in compute "credits" to all users). So, basically, the money stream of a commercial company is there, as well as the goal to use it for the good of the many. The GUI is generated from the Python docstring (which optionally includes metadata in an inline YAML) and typing information (PEP 484). We handle all the development and the maintenance on this but theoretically somebody could write an entirely separate frontend by just using our GraphQL endpoint and a few HTTP POST requests and interpret custom GUI metadata (we would need to standardize the APIs and commit to not breaking them). The backend lets you safely run arbitrary code already so I doubt there are many applications for which it would not be flexible enough. We are aware that there are dozens of previous systems for generating interfaces for pipelines. Our novel contributions primarily come from the reliance on cloud tech (with primary benefit being the ability to scale to extremely compute-intensive workflows), the integration with a cloud-native data system (everything is in S3, with a wrapper to make it considerably more user friendly; primary benefit is, again, scale—you can transparently store terabytes of data), and the use of static typing throughout (primary benefit is that even unannotated workflows get a "good-enough" GUI if they have types).

maximsmol 5y ago

Full disclosure: I am a founding engineer at this company. We are charging corporate clients for white glove service and minor contracting (helping them upload and adjust pipelines). We also take a small margin over AWS compute prices for regular users. This lets us support the academic and small volume users for free (we give $100 in compute "credits" to all users). So, basically, the money stream of a commercial company is there, as well as the goal to use it for the good of the many. The GUI is generated from the Python docstring (which optionally includes metadata in an inline YAML) and typing information (PEP 484). We handle all the development and the maintenance on this but theoretically somebody could write an entirely separate frontend by just using our GraphQL endpoint and a few HTTP POST requests and interpret custom GUI metadata (we would need to standardize the APIs and commit to not breaking them). The backend lets you safely run arbitrary code already so I doubt there are many applications for which it would not be flexible enough. We are aware that there are dozens of previous systems for generating interfaces for pipelines. Our novel contributions primarily come from the reliance on cloud tech (with primary benefit being the ability to scale to extremely compute-intensive workflows), the integration with a cloud-native data system (everything is in S3, with a wrapper to make it considerably more user friendly; primary benefit is, again, scale—you can transparently store terabytes of data), and the use of static typing throughout (primary benefit is that even unannotated workflows get a "good-enough" GUI if they have types).