6 ms·

Pachyderm Hub: data science without the hassle of managing infrastructure

107 points by jdoliner 6y ago

hobofan 6y ago
After following Pachyderm for a long time, the docs and everything finally look to be at a point where I really want to try it out! There is no point though in hiding the pricing behind the signup, especially if it's the direct first step you see after you signed up. All you are doing is inflate your signup numbers (and maybe even deter a few people). PS: The link under "Resources -> What's the typical developer workflow?" is broken.
- deleted 6y ago
  [deleted]
- jdoliner 6y ago
  Thanks for staying involved with the project, we're so glad to hear that the improvements are working. You bring up some good points about the pricing page. We'll be making the pricing visible without login soon. If you have some time we'd love to get some more feedback via our slack channel (pachyderm.com/slack). With you being involved for so long your feedback would be super insightful. (Fixed that link. email us and we'll send you some Pachyderm swag if you like)
  - sauwan 6y ago
    I agree. I left the page immediately when it wasn't clear what the pricing would look like. No use investing any of my time into something I don't know if I can afford or get into my work budget.
    - sethammons 6y ago
      yup. I've seen "hidden" prices be as low as $100/yr to as much as $10k/mo. Now I don't dig in deeper and just move along unless I'm desperate.
- deleted 6y ago
  [deleted]
ryan_j_naughton 6y ago
Awesome! We switched from airflow to pachyderm recently, and we have loved it (we still have some legacy on airflow, but all new data engineering dev is in pachyderm).
- jdoliner 6y ago
  Thanks for the kind words, Airflow -> Pachyderm has been an increasingly common migration path for us.
anilshanbhag 6y ago
Looking at this - it seems very similar to Verta.AI Both are doing data lineage/workflow management ~ git for data science workflows. Verta recently raised their series A and launched their platform too.
- jdoliner 6y ago
  I'm not super familiar with Verta.AI but reading about their product it seems pretty different. I read Verta.AI as a model deployment and tracking solution. Seldon would be the opensource comparison that immediately comes to mind. Pachyderm is more of a general pipeline and data version control system. You can do some very basic model deployment and tracking in it but we general recommend people use an external system such as Seldon and even have some published examples of how to integrate the two. Disclosure: I founded Pachyderm.
tspann 6y ago
I often need Dataflow, Spark Engineering, NoSql and Sql datastores, enterprise security and hybrid cloud with my datascience https://docs.cloudera.com/machine-learning/cloud/product/topics/ml-product-overview.html https://docs.cloudera.com/machine-learning/cloud/product/top...
data_ders 6y ago
I'm quite excited for the emerging Python-, Git-, and DAG-driven data engineering workflows/orchestrators that are code-first, GUI-second. But is anyone daunted at the prospect of investing in one of these platforms, then having to pivot to the paradigm that emerges as the winner 3-5 years from now? I've been trying to keep up with all the tools, but am pretty overwhelmed at this point. Obviouly there's differences, but off the top of my head I can think of: Airflow, Luigi, Pachyderm, DAGster, Google Cloud Compose, Kuberflow, MLFlow, Azure ML Pipelines, Ascend.io, and so so many more.
- pm90 6y ago
  There isn’t a way around it. The best thing to do is to try and build your data pipelines in a way that makes it somewhat easier to swap tools if required, but I’m not sure if there’s a general way to do that.
- jdoliner 6y ago
  This is unfortunately always going to be a problem with a developing space. Pachyderm was designed with this issue in mind though. While I don't think we've totally eliminated, or can totally eliminate it, we have done some things to mitigate it. Pachyderm pipelines run your code in a docker container that you define which reads data from the local filesystem. It's designed to be simple and similar to the interface you'd use when playing around with data on your laptop. User's generally have a very easy time migrating existing code into Pachyderm, since often code is already reading from the local filesystem and just needs to be pointed at a new path. Migrating to a different system is normally a little more complicated because it requires integrating whatever that new systems data interface is. Although in another system where you just read from local disk the migration should be quite simple.
- gumby 6y ago
  > But is anyone daunted at the prospect of investing in one of these platforms, then having to pivot to the paradigm that emerges as the winner 3-5 years from now? This will always be an issue if you’re an early adopter. The three mitigating calculations every case are: 1 - somebody* will probably be the “winner”, and could be it. 2 - between now and 3 years from now will you get enough value that it will be worth migrating to the eventual winner rather than doing without and 3 - three years from now you can just start new projects on the winning tech; most or all of the current projects will have withered and/or died anyway.
racl101 6y ago
I wanted to install R to try it and test it to see if it was something I could work with on MacOS (Mojave). It was nothing short of an f-ing nightmare. Wonder if this service will help.
- jaz46 6y ago
  We have lots of users who develop pipelines in R. Here's the standard iris example in R (example also include Python and Julia) if you're interested: https://github.com/pachyderm/pachyderm/tree/master/examples/ml/iris https://github.com/pachyderm/pachyderm/tree/master/examples/...
- claytonjy 6y ago
  I don't think this solves that problem, or intends to. You can run R in pachyderm via Docker, but that's unlikely to be a good way to test R out. If you like Docker, try the images from Rocker: https://www.rocker-project.org/ https://www.rocker-project.org/. The one with RStudio server is especially useful for local experimentation.
  - ellisv 6y ago
    :wave:
  - racl101 6y ago
    Thanks, I'll take a look a this.
- ellisv 6y ago
  > It was nothing short of an f-ing nightmare. Honestly I think I've only ever heard of installation issues from Windows users. What was the problem?
  - racl101 6y ago
    The problem was with C dependencies. I'd install Xcode CLI tools, that didn't work then I'd try installing them with brew and that didn't work. I was trying to get Jupyter Notebook and an R engine to work as advertised in a Packt ebook. Only way I got it to work was not by installing it on MacOS but a Vagrantbox and then doing port fowarding. But I'll have to look into the Docker way. Maybe that way is better.
claytonjy 6y ago
Can this run in a peered VPC connection or will I have to pay for data transferred from my public cloud infrastructure to this service?
- jdoliner 6y ago
  It can't right now, it all runs in GCP. You can still spin up Pachyderm yourself in whatever environment you like. We're working on enabling custom deploys in more environments than the one we just have but it's still a little ways out.
adamsvystun 6y ago
Couldn't find any info on pricing. How much more expensive is this than running your own cluster? P.S. Couldn't login. Stuck on https://hub.pachyderm.com/pricing https://hub.pachyderm.com/pricing. I am just seeing a white screen. (Login with Google, Linux, Firefox, Ukraine)
jamesblonde 6y ago
The data science platform looks great, but I have a fundamental issue with building a data versioning platform using git-like semantics - Pachyderm and DVC follow this approach. Git-like approaches just track immutable files - they do not store the differences between files. That's fine when you're working with small amounts of data, but doesn't scale to handle large volumes of data and prevents you from making some times of time-travel queries - give me the data that changed in this time range. The emerging enterprise approach to handle this is to use a framework like Delta.io, Apache Hudi, or Apache Iceberg. These frameworks add metadata over Parquet, and allow you to do point point-in-time time-travel queries as well as interval-based time-travel queries (give me data that changed in this time interval). Parquet is becoming a dominant file format for data lakes and is cheap to store on s3. Disclosure: I am a co-founder of Logical Clocks.
culturestate 6y ago
Quick note for @jdoliner: there's a typo on the homepage in the Useful Links section - "Technicial Slack Channel"

107 points by jdoliner 6y ago

After following Pachyderm for a long time, the docs and everything finally look to be at a point where I really want to try it out! There is no point though in hiding the pricing behind the signup, especially if it's the direct first step you see after you signed up. All you are doing is inflate your signup numbers (and maybe even deter a few people). PS: The link under "Resources -> What's the typical developer workflow?" is broken.

deleted 6y ago
[deleted]
jdoliner 6y ago
Thanks for staying involved with the project, we're so glad to hear that the improvements are working. You bring up some good points about the pricing page. We'll be making the pricing visible without login soon. If you have some time we'd love to get some more feedback via our slack channel (pachyderm.com/slack). With you being involved for so long your feedback would be super insightful. (Fixed that link. email us and we'll send you some Pachyderm swag if you like)
- sauwan 6y ago
  I agree. I left the page immediately when it wasn't clear what the pricing would look like. No use investing any of my time into something I don't know if I can afford or get into my work budget.
  - sethammons 6y ago
    yup. I've seen "hidden" prices be as low as $100/yr to as much as $10k/mo. Now I don't dig in deeper and just move along unless I'm desperate.
deleted 6y ago
[deleted]

ryan_j_naughton 6y ago

Awesome! We switched from airflow to pachyderm recently, and we have loved it (we still have some legacy on airflow, but all new data engineering dev is in pachyderm).

jdoliner 6y ago
Thanks for the kind words, Airflow -> Pachyderm has been an increasingly common migration path for us.

anilshanbhag 6y ago

Looking at this - it seems very similar to Verta.AI Both are doing data lineage/workflow management ~ git for data science workflows. Verta recently raised their series A and launched their platform too.

jdoliner 6y ago
I'm not super familiar with Verta.AI but reading about their product it seems pretty different. I read Verta.AI as a model deployment and tracking solution. Seldon would be the opensource comparison that immediately comes to mind. Pachyderm is more of a general pipeline and data version control system. You can do some very basic model deployment and tracking in it but we general recommend people use an external system such as Seldon and even have some published examples of how to integrate the two. Disclosure: I founded Pachyderm.

tspann 6y ago

I often need Dataflow, Spark Engineering, NoSql and Sql datastores, enterprise security and hybrid cloud with my datascience https://docs.cloudera.com/machine-learning/cloud/product/topics/ml-product-overview.html https://docs.cloudera.com/machine-learning/cloud/product/top...

data_ders 6y ago

I'm quite excited for the emerging Python-, Git-, and DAG-driven data engineering workflows/orchestrators that are code-first, GUI-second. But is anyone daunted at the prospect of investing in one of these platforms, then having to pivot to the paradigm that emerges as the winner 3-5 years from now? I've been trying to keep up with all the tools, but am pretty overwhelmed at this point. Obviouly there's differences, but off the top of my head I can think of: Airflow, Luigi, Pachyderm, DAGster, Google Cloud Compose, Kuberflow, MLFlow, Azure ML Pipelines, Ascend.io, and so so many more.

pm90 6y ago
There isn’t a way around it. The best thing to do is to try and build your data pipelines in a way that makes it somewhat easier to swap tools if required, but I’m not sure if there’s a general way to do that.
jdoliner 6y ago
This is unfortunately always going to be a problem with a developing space. Pachyderm was designed with this issue in mind though. While I don't think we've totally eliminated, or can totally eliminate it, we have done some things to mitigate it. Pachyderm pipelines run your code in a docker container that you define which reads data from the local filesystem. It's designed to be simple and similar to the interface you'd use when playing around with data on your laptop. User's generally have a very easy time migrating existing code into Pachyderm, since often code is already reading from the local filesystem and just needs to be pointed at a new path. Migrating to a different system is normally a little more complicated because it requires integrating whatever that new systems data interface is. Although in another system where you just read from local disk the migration should be quite simple.
gumby 6y ago
> But is anyone daunted at the prospect of investing in one of these platforms, then having to pivot to the paradigm that emerges as the winner 3-5 years from now? This will always be an issue if you’re an early adopter. The three mitigating calculations every case are: 1 - somebody* will probably be the “winner”, and could be it. 2 - between now and 3 years from now will you get enough value that it will be worth migrating to the eventual winner rather than doing without and 3 - three years from now you can just start new projects on the winning tech; most or all of the current projects will have withered and/or died anyway.

racl101 6y ago

I wanted to install R to try it and test it to see if it was something I could work with on MacOS (Mojave). It was nothing short of an f-ing nightmare. Wonder if this service will help.

jaz46 6y ago
We have lots of users who develop pipelines in R. Here's the standard iris example in R (example also include Python and Julia) if you're interested: https://github.com/pachyderm/pachyderm/tree/master/examples/ml/iris https://github.com/pachyderm/pachyderm/tree/master/examples/...
claytonjy 6y ago
I don't think this solves that problem, or intends to. You can run R in pachyderm via Docker, but that's unlikely to be a good way to test R out. If you like Docker, try the images from Rocker: https://www.rocker-project.org/ https://www.rocker-project.org/. The one with RStudio server is especially useful for local experimentation.
- ellisv 6y ago
  :wave:
- racl101 6y ago
  Thanks, I'll take a look a this.
ellisv 6y ago
> It was nothing short of an f-ing nightmare. Honestly I think I've only ever heard of installation issues from Windows users. What was the problem?
- racl101 6y ago
  The problem was with C dependencies. I'd install Xcode CLI tools, that didn't work then I'd try installing them with brew and that didn't work. I was trying to get Jupyter Notebook and an R engine to work as advertised in a Packt ebook. Only way I got it to work was not by installing it on MacOS but a Vagrantbox and then doing port fowarding. But I'll have to look into the Docker way. Maybe that way is better.

claytonjy 6y ago

Can this run in a peered VPC connection or will I have to pay for data transferred from my public cloud infrastructure to this service?

jdoliner 6y ago
It can't right now, it all runs in GCP. You can still spin up Pachyderm yourself in whatever environment you like. We're working on enabling custom deploys in more environments than the one we just have but it's still a little ways out.

adamsvystun 6y ago

Couldn't find any info on pricing. How much more expensive is this than running your own cluster? P.S. Couldn't login. Stuck on https://hub.pachyderm.com/pricing https://hub.pachyderm.com/pricing. I am just seeing a white screen. (Login with Google, Linux, Firefox, Ukraine)

jamesblonde 6y ago

The data science platform looks great, but I have a fundamental issue with building a data versioning platform using git-like semantics - Pachyderm and DVC follow this approach. Git-like approaches just track immutable files - they do not store the differences between files. That's fine when you're working with small amounts of data, but doesn't scale to handle large volumes of data and prevents you from making some times of time-travel queries - give me the data that changed in this time range. The emerging enterprise approach to handle this is to use a framework like Delta.io, Apache Hudi, or Apache Iceberg. These frameworks add metadata over Parquet, and allow you to do point point-in-time time-travel queries as well as interval-based time-travel queries (give me data that changed in this time interval). Parquet is becoming a dominant file format for data lakes and is cheap to store on s3. Disclosure: I am a co-founder of Logical Clocks.

culturestate 6y ago

Quick note for @jdoliner: there's a typo on the homepage in the Useful Links section - "Technicial Slack Channel"