3 ms·

Heya! The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 d

Heya! The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 different regions/zones/etc Kubernetes is great, but requires using a central control plane. Swarm is also great but is more focused on job execution, rather than targeting data, results, etc We love both ecosystems (our cofounders and team are made up of very early contributors to both), but we think there’s room for a compliment of something that is more resilient to poor network conditions and allows for data first scheduling and in-place execution of any arbitrary job (containerized or not). Does that help? Disclosure: I co-founded Bacalhau

FridgeSeal 3y ago
Very interesting. Kind of sounds a bit closer to something like Spark, where you want to distribute the compute over several machines, and with a few more executions modes/capabilities, is that correct? > Kubernetes is great, but requires using a central control plane…something that is more resilient to poor network conditions and allows for data first scheduling This sounds very interesting. I’m curious about your consistency model here, how are you preventing cluster members from getting out of sync, losing work, etc? CRDT’s? Make jobs idempotent by controlling side effects? Also, love the name, the pun on compute-over-data is excellent.
- TheIronYuppie 3y ago
  thank you very much! Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment). You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command. Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly) We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more! [1] https://docs.bacalhau.org/getting-started/architecture https://docs.bacalhau.org/getting-started/architecture [2] https://bit.ly/bacalhau-project-slack https://bit.ly/bacalhau-project-slack [3] https://github.com/bacalhau-project https://github.com/bacalhau-project

FridgeSeal 3y ago

Very interesting. Kind of sounds a bit closer to something like Spark, where you want to distribute the compute over several machines, and with a few more executions modes/capabilities, is that correct? > Kubernetes is great, but requires using a central control plane…something that is more resilient to poor network conditions and allows for data first scheduling This sounds very interesting. I’m curious about your consistency model here, how are you preventing cluster members from getting out of sync, losing work, etc? CRDT’s? Make jobs idempotent by controlling side effects? Also, love the name, the pun on compute-over-data is excellent.

TheIronYuppie 3y ago
thank you very much! Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment). You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command. Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly) We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more! [1] https://docs.bacalhau.org/getting-started/architecture https://docs.bacalhau.org/getting-started/architecture [2] https://bit.ly/bacalhau-project-slack https://bit.ly/bacalhau-project-slack [3] https://github.com/bacalhau-project https://github.com/bacalhau-project