5 ms·

Building a Distributed Data Warehouse Without Data Lakes

26 points by TheIronYuppie 3y ago

TheIronYuppie 3y ago
Heya all! As always, we'd love to hear more if you have any questions/comments/concerns. aronchick (at) expanso (dot) io Disclosure: I co-founded Bacalhau
slt2021 3y ago
I didnt get what this does, is it just running a containerized workloads on a cluster? like a docker swarm? what makes it better than docker swarm or kubernetes or nomad or any other container orchestration tool ?
- TheIronYuppie 3y ago
  Heya! The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 different regions/zones/etc Kubernetes is great, but requires using a central control plane. Swarm is also great but is more focused on job execution, rather than targeting data, results, etc We love both ecosystems (our cofounders and team are made up of very early contributors to both), but we think there’s room for a compliment of something that is more resilient to poor network conditions and allows for data first scheduling and in-place execution of any arbitrary job (containerized or not). Does that help? Disclosure: I co-founded Bacalhau
  - FridgeSeal 3y ago
    Very interesting. Kind of sounds a bit closer to something like Spark, where you want to distribute the compute over several machines, and with a few more executions modes/capabilities, is that correct? > Kubernetes is great, but requires using a central control plane…something that is more resilient to poor network conditions and allows for data first scheduling This sounds very interesting. I’m curious about your consistency model here, how are you preventing cluster members from getting out of sync, losing work, etc? CRDT’s? Make jobs idempotent by controlling side effects? Also, love the name, the pun on compute-over-data is excellent.
    - TheIronYuppie 3y ago
      thank you very much! Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment). You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command. Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly) We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more! [1] https://docs.bacalhau.org/getting-started/architecture https://docs.bacalhau.org/getting-started/architecture [2] https://bit.ly/bacalhau-project-slack https://bit.ly/bacalhau-project-slack [3] https://github.com/bacalhau-project https://github.com/bacalhau-project
- FridgeSeal 3y ago
  It doesn’t appear to be a container-orchestration tool at all. It’s a distributed OLAP database of sorts. More like a snowflake/ ClickHouse/ Trino (nee Presto).
  - chatmasta 3y ago
    I think it's more generalized distributed compute, with OLAP being one of the use cases. There are a lot of videos on YouTube if you search "Bacalhau cluster" or similar, including what appears to be their legacy channel [0] (which I saw a year ago and found very impressive) and a newer one [1], but also conference talks from other channels. I haven't been following the project, but I remember finding the original demo to be very impressive - I don't remember the details off-hand, but IPFS was somehow involved. Seems like a lot of work has taken place since then. Docs are here [2] [0] https://www.youtube.com/@bacalhau3295 https://www.youtube.com/@bacalhau3295 [1] https://www.youtube.com/@bacalhauproject https://www.youtube.com/@bacalhauproject [2] https://docs.bacalhau.org/ https://docs.bacalhau.org/
    - TheIronYuppie 3y ago
      Thank you very much! Yep you nailed it - OLAP is just one of the use cases. We support LOTS of different examples [1]. We also support NON-container workloads like arbitrary exeuctors [2] and WASM [3]. Our goal is to just make it just as easy to build complicated, distributed workloads as it is with Swarm, Kubernetes, Nomad, etc, but adding in cross network reliability and queueing. Oh, one other thing - we do support IPFS [4] natively but it's optional; we also support S3 [5], local disk, etc. [1] https://docs.bacalhau.org/examples https://docs.bacalhau.org/examples [2] https://blog.bacalhau.org/p/distributed-compute-platform-bacalhau https://blog.bacalhau.org/p/distributed-compute-platform-bac... <- look for pluggable executors (it's in beta) [3] https://docs.bacalhau.org/examples/workload-onboarding/rust-wasm/#running-wasm-on-bacalhau https://docs.bacalhau.org/examples/workload-onboarding/rust-... [4] https://docs.bacalhau.org/running-node/storage-providers/#ipfs-1 https://docs.bacalhau.org/running-node/storage-providers/#ip... [5] https://docs.bacalhau.org/references/job-specification/input-source https://docs.bacalhau.org/references/job-specification/input...
up2isomorphism 3y ago
Such a dumb but funny idea. Almost likely a parody for “data lake” hype which does not really make a lot of sense in the first place.
- TheIronYuppie 3y ago
  It's an interesting question! The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake. It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings. Disclosure: I co-founded Bacalhau [1] https://github.com/bacalhau-project/bacalhau https://github.com/bacalhau-project/bacalhau [2] https://github.com/duckdb/duckdb https://github.com/duckdb/duckdb

26 points by TheIronYuppie 3y ago

Heya all! As always, we'd love to hear more if you have any questions/comments/concerns. aronchick (at) expanso (dot) io Disclosure: I co-founded Bacalhau

slt2021 3y ago

I didnt get what this does, is it just running a containerized workloads on a cluster? like a docker swarm? what makes it better than docker swarm or kubernetes or nomad or any other container orchestration tool ?

TheIronYuppie 3y ago
Heya! The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 different regions/zones/etc Kubernetes is great, but requires using a central control plane. Swarm is also great but is more focused on job execution, rather than targeting data, results, etc We love both ecosystems (our cofounders and team are made up of very early contributors to both), but we think there’s room for a compliment of something that is more resilient to poor network conditions and allows for data first scheduling and in-place execution of any arbitrary job (containerized or not). Does that help? Disclosure: I co-founded Bacalhau
- FridgeSeal 3y ago
  Very interesting. Kind of sounds a bit closer to something like Spark, where you want to distribute the compute over several machines, and with a few more executions modes/capabilities, is that correct? > Kubernetes is great, but requires using a central control plane…something that is more resilient to poor network conditions and allows for data first scheduling This sounds very interesting. I’m curious about your consistency model here, how are you preventing cluster members from getting out of sync, losing work, etc? CRDT’s? Make jobs idempotent by controlling side effects? Also, love the name, the pun on compute-over-data is excellent.
  - TheIronYuppie 3y ago
    thank you very much! Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment). You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command. Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly) We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more! [1] https://docs.bacalhau.org/getting-started/architecture https://docs.bacalhau.org/getting-started/architecture [2] https://bit.ly/bacalhau-project-slack https://bit.ly/bacalhau-project-slack [3] https://github.com/bacalhau-project https://github.com/bacalhau-project
FridgeSeal 3y ago
It doesn’t appear to be a container-orchestration tool at all. It’s a distributed OLAP database of sorts. More like a snowflake/ ClickHouse/ Trino (nee Presto).
- chatmasta 3y ago
  I think it's more generalized distributed compute, with OLAP being one of the use cases. There are a lot of videos on YouTube if you search "Bacalhau cluster" or similar, including what appears to be their legacy channel [0] (which I saw a year ago and found very impressive) and a newer one [1], but also conference talks from other channels. I haven't been following the project, but I remember finding the original demo to be very impressive - I don't remember the details off-hand, but IPFS was somehow involved. Seems like a lot of work has taken place since then. Docs are here [2] [0] https://www.youtube.com/@bacalhau3295 https://www.youtube.com/@bacalhau3295 [1] https://www.youtube.com/@bacalhauproject https://www.youtube.com/@bacalhauproject [2] https://docs.bacalhau.org/ https://docs.bacalhau.org/
  - TheIronYuppie 3y ago
    Thank you very much! Yep you nailed it - OLAP is just one of the use cases. We support LOTS of different examples [1]. We also support NON-container workloads like arbitrary exeuctors [2] and WASM [3]. Our goal is to just make it just as easy to build complicated, distributed workloads as it is with Swarm, Kubernetes, Nomad, etc, but adding in cross network reliability and queueing. Oh, one other thing - we do support IPFS [4] natively but it's optional; we also support S3 [5], local disk, etc. [1] https://docs.bacalhau.org/examples https://docs.bacalhau.org/examples [2] https://blog.bacalhau.org/p/distributed-compute-platform-bacalhau https://blog.bacalhau.org/p/distributed-compute-platform-bac... <- look for pluggable executors (it's in beta) [3] https://docs.bacalhau.org/examples/workload-onboarding/rust-wasm/#running-wasm-on-bacalhau https://docs.bacalhau.org/examples/workload-onboarding/rust-... [4] https://docs.bacalhau.org/running-node/storage-providers/#ipfs-1 https://docs.bacalhau.org/running-node/storage-providers/#ip... [5] https://docs.bacalhau.org/references/job-specification/input-source https://docs.bacalhau.org/references/job-specification/input...

up2isomorphism 3y ago

Such a dumb but funny idea. Almost likely a parody for “data lake” hype which does not really make a lot of sense in the first place.

TheIronYuppie 3y ago
It's an interesting question! The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake. It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings. Disclosure: I co-founded Bacalhau [1] https://github.com/bacalhau-project/bacalhau https://github.com/bacalhau-project/bacalhau [2] https://github.com/duckdb/duckdb https://github.com/duckdb/duckdb