3 ms·

I really find it difficult to put into words just how little I care to pay for a web ui so I can "manage" my data. Data pipelines are a real problem though, an

by smeeth 6y ago

I really find it difficult to put into words just how little I care to pay for a web ui so I can "manage" my data. Data pipelines are a real problem though, and I'm very interested in what startups do with this space.

factorialboy 6y ago
> Data pipelines are a real problem though Can you please elaborate more, thanks.
- prions 6y ago
  Its not trivial to create and manage Data Pipelines if you care about scale, serving a wide range of inputs and outputs, or making this data easy to surface and spread throughout your org (i.e. making it actually useful to regular people). "Static ETL" like running the same database load every day at 1:00am isn't a super challenging problem. Doing it across many tables with complex transformations and multiple steps easily can be. You really have to consider reliability, processing speed, failure methods and other problems that dont really arise until you hit a certain scale. There's also the issue of what people want out of a Pipeline that's changing. If you want people to be ""data driven"", then that means they need easy access to potentially all of your company's data on an ad hoc basis. So now your boring ETL 1 am pipeline isnt really serving any of these new usecases. How do you create flexible pipelines that can be created from any dataset on an ad hoc basis? This is where tools like Airflow or Prefect come in. Creating a platform that can create these types of Pipelines is a real problem. And before you even ask yourself _how_ to process this data, you need to also ask _where_? If you want to do what I outlined above - making your data more accessible and easy to use - then you probably need to rework how you're storing your data. But Data Lakes (and others) are a whole topic in and of itself.
dttos 6y ago
Suggest you check out https://composable.ai https://composable.ai for building out robust data pipelines
seddonm1 6y ago
We have been thinking about these problems for a few years now and have built Arc https://arc.tripl.ai https://arc.tripl.ai (fully open source.) which is an abstraction layer on top of Apache Spark to help end-users rapidly build and deploy data pipelines without having to know about #dataops. Ultimately we decided that giving users a decent interface https://github.com/tripl-ai/arc-starter https://github.com/tripl-ai/arc-starter (based on Jupyter Notebooks) and encouraging a 'SQL first' approach means we can give users flexibility but also have a standardised way of deploying jobs with many of the devops attributes (like logging and reliability). You can run Arc as a standard docker run command or using Argo Workflows https://argoproj.github.io/ https://argoproj.github.io/ on Kubernetes as the orchestrator as it plays nicely with Arc and is easy to build resilient pipelines (retries etc.)

by smeeth 6y ago

factorialboy 6y ago

> Data pipelines are a real problem though Can you please elaborate more, thanks.

prions 6y ago
Its not trivial to create and manage Data Pipelines if you care about scale, serving a wide range of inputs and outputs, or making this data easy to surface and spread throughout your org (i.e. making it actually useful to regular people). "Static ETL" like running the same database load every day at 1:00am isn't a super challenging problem. Doing it across many tables with complex transformations and multiple steps easily can be. You really have to consider reliability, processing speed, failure methods and other problems that dont really arise until you hit a certain scale. There's also the issue of what people want out of a Pipeline that's changing. If you want people to be ""data driven"", then that means they need easy access to potentially all of your company's data on an ad hoc basis. So now your boring ETL 1 am pipeline isnt really serving any of these new usecases. How do you create flexible pipelines that can be created from any dataset on an ad hoc basis? This is where tools like Airflow or Prefect come in. Creating a platform that can create these types of Pipelines is a real problem. And before you even ask yourself _how_ to process this data, you need to also ask _where_? If you want to do what I outlined above - making your data more accessible and easy to use - then you probably need to rework how you're storing your data. But Data Lakes (and others) are a whole topic in and of itself.

dttos 6y ago

Suggest you check out https://composable.ai https://composable.ai for building out robust data pipelines

seddonm1 6y ago

We have been thinking about these problems for a few years now and have built Arc https://arc.tripl.ai https://arc.tripl.ai (fully open source.) which is an abstraction layer on top of Apache Spark to help end-users rapidly build and deploy data pipelines without having to know about #dataops. Ultimately we decided that giving users a decent interface https://github.com/tripl-ai/arc-starter https://github.com/tripl-ai/arc-starter (based on Jupyter Notebooks) and encouraging a 'SQL first' approach means we can give users flexibility but also have a standardised way of deploying jobs with many of the devops attributes (like logging and reliability). You can run Arc as a standard docker run command or using Argo Workflows https://argoproj.github.io/ https://argoproj.github.io/ on Kubernetes as the orchestrator as it plays nicely with Arc and is easy to build resilient pipelines (retries etc.)