10 ms·

For people in this thread interested in what this tool is an alternative to: Airflow, Luigi, Kubeflow, Kedro, Flyte, Metaflow, Sagemaker Pipelines, GCP Vertex W

For people in this thread interested in what this tool is an alternative to: Airflow, Luigi, Kubeflow, Kedro, Flyte, Metaflow, Sagemaker Pipelines, GCP Vertex Workbench, Azure Data Factory, Azure ML, Dagster, DVC, ClearML, Prefect, Pachyderm, and Orchest. Disclaimer: author of Orchest https://github.com/orchest/orchest https://github.com/orchest/orchest

josh-sematic 4y ago
Yup, most of these tools fall into the definition of "orchestration" in one way shape or form, though not all of them are targeted at Machine Learning pipelines, and not all of them focus as much on the low barrier to entry space we're aiming at. In general they also don't optimize for local workflows while still enabling cloud access. We understand there are a lot of tools that may superficially look similar, so we've put together a page to describe things we do differently from other tools: https://docs.sematic.dev/sematic-vs https://docs.sematic.dev/sematic-vs Btw, orchest looks like a cool way to orchestrate notebooks! Also, hi! I'm founding engineer here at Sematic. Happy to answer any questions I can!
- ricklamers 4y ago
  Nice comparison section! And thank you kindly for the compliment. Our approach has been supporting both Python scripts (R, Julia, JS and Bash too for that matter) and notebooks as it would give users the ability to choose the right tool for the job and in case of migrations from notebooks to scripts make the process more incremental. Welcome to the thread :wave:
- troiskaer 4y ago
  How does Sematic compare to Metaflow? it optimizes for many of the same goals of Sematic - local workflows, cloud access, lineage tracking, state transfer etc?
  - josh-sematic 4y ago
    There are several differences, but I'd say these are some of the main ones: UI: whereas Metaflow provides the ability to build your own result visualizations explicitly in your workflow (via their "cards" feature), Sematic makes it so that your outputs (and inputs) get automatic rich visualizations based on the data type of the data being passed around. API: Instead of being based around explicitly building up a graph, where you have to explicitly specify the I/O connections between steps, Sematic makes defining your steps look like writing/calling python functions. Packaging: Whereas metaflow requires you to include packaging information in the code defining your steps (the @conda decorator, etc.), Sematic plugs into your existing dependency management to bundle up dependencies for execution in the cloud.
    - troiskaer 4y ago
      Interesting approach on passing data between steps and constructing the overall graph - it will be interesting to see what the take rate is between the two approaches (of sematic and metaflow). On the UI front, Metaflow generates viz for all objects by default in @card; but how does Sematic package up PyTorch referenced in the example (https://docs.sematic.dev/real-example https://docs.sematic.dev/real-example) for execution on the cloud? IIRC, Metaflow packages the cwd (in addition to @conda, @pip etc.) and relies on existing packages for local execution? Edit: Digging deeper, Sematic relies on Bazel (https://docs.sematic.dev/execution-modes#dependency-packaging https://docs.sematic.dev/execution-modes#dependency-packagin...) and needs a BUILD file to specify all the dependencies for cloud execution. It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment? I am quite interested in trying out Sematic. Any guidelines on what kind of scale Sematic can support today (and the near future)?
      - josh-sematic 4y ago
        The way packaging is designed to work in Sematic is for us to hook into your existing dependency management solution to determine what your dependencies are, then build a docker image for you based on those. As you point out, right now we only integrate with bazel for this purpose, but we hope to add more. A simple plugin for requirements.txt -> Docker image is probably next on the TODO list. > It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment? single docker image, but multiple pods (when you are using the full cloud mode). This was an intentional decision to avoid confusion around what things could be imported in what places (mimicking more what it would be like in one python instance), and also avoid weird version inconsistencies across the pipeline. > Any guidelines on what kind of scale Sematic can support today (and the near future)? Based on some prior tooling experiences, the main bottleneck should be what your Kubernetes cluster can handle. > I am quite interested in trying out Sematic Glad to hear it! We'd love to hear about your experiences. You can join our discord if you want help while you're trying it out: https://discord.com/invite/4KZJ6kYVax https://discord.com/invite/4KZJ6kYVax
learndeeply 4y ago
Why do people keep on re-inventing pipelines? There's so many already, all with almost identical syntax. This one is similar to Flyte.

josh-sematic 4y ago

Yup, most of these tools fall into the definition of "orchestration" in one way shape or form, though not all of them are targeted at Machine Learning pipelines, and not all of them focus as much on the low barrier to entry space we're aiming at. In general they also don't optimize for local workflows while still enabling cloud access. We understand there are a lot of tools that may superficially look similar, so we've put together a page to describe things we do differently from other tools: https://docs.sematic.dev/sematic-vs https://docs.sematic.dev/sematic-vs Btw, orchest looks like a cool way to orchestrate notebooks! Also, hi! I'm founding engineer here at Sematic. Happy to answer any questions I can!

ricklamers 4y ago
Nice comparison section! And thank you kindly for the compliment. Our approach has been supporting both Python scripts (R, Julia, JS and Bash too for that matter) and notebooks as it would give users the ability to choose the right tool for the job and in case of migrations from notebooks to scripts make the process more incremental. Welcome to the thread :wave:
troiskaer 4y ago
How does Sematic compare to Metaflow? it optimizes for many of the same goals of Sematic - local workflows, cloud access, lineage tracking, state transfer etc?
- josh-sematic 4y ago
  There are several differences, but I'd say these are some of the main ones: UI: whereas Metaflow provides the ability to build your own result visualizations explicitly in your workflow (via their "cards" feature), Sematic makes it so that your outputs (and inputs) get automatic rich visualizations based on the data type of the data being passed around. API: Instead of being based around explicitly building up a graph, where you have to explicitly specify the I/O connections between steps, Sematic makes defining your steps look like writing/calling python functions. Packaging: Whereas metaflow requires you to include packaging information in the code defining your steps (the @conda decorator, etc.), Sematic plugs into your existing dependency management to bundle up dependencies for execution in the cloud.
  - troiskaer 4y ago
    Interesting approach on passing data between steps and constructing the overall graph - it will be interesting to see what the take rate is between the two approaches (of sematic and metaflow). On the UI front, Metaflow generates viz for all objects by default in @card; but how does Sematic package up PyTorch referenced in the example (https://docs.sematic.dev/real-example https://docs.sematic.dev/real-example) for execution on the cloud? IIRC, Metaflow packages the cwd (in addition to @conda, @pip etc.) and relies on existing packages for local execution? Edit: Digging deeper, Sematic relies on Bazel (https://docs.sematic.dev/execution-modes#dependency-packaging https://docs.sematic.dev/execution-modes#dependency-packagin...) and needs a BUILD file to specify all the dependencies for cloud execution. It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment? I am quite interested in trying out Sematic. Any guidelines on what kind of scale Sematic can support today (and the near future)?
    - josh-sematic 4y ago
      The way packaging is designed to work in Sematic is for us to hook into your existing dependency management solution to determine what your dependencies are, then build a docker image for you based on those. As you point out, right now we only integrate with bazel for this purpose, but we hope to add more. A simple plugin for requirements.txt -> Docker image is probably next on the TODO list. > It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment? single docker image, but multiple pods (when you are using the full cloud mode). This was an intentional decision to avoid confusion around what things could be imported in what places (mimicking more what it would be like in one python instance), and also avoid weird version inconsistencies across the pipeline. > Any guidelines on what kind of scale Sematic can support today (and the near future)? Based on some prior tooling experiences, the main bottleneck should be what your Kubernetes cluster can handle. > I am quite interested in trying out Sematic Glad to hear it! We'd love to hear about your experiences. You can join our discord if you want help while you're trying it out: https://discord.com/invite/4KZJ6kYVax https://discord.com/invite/4KZJ6kYVax

learndeeply 4y ago

Why do people keep on re-inventing pipelines? There's so many already, all with almost identical syntax. This one is similar to Flyte.