4 ms·

This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor https://github.com/vmware/database-stream-proces

This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor https://github.com/vmware/database-stream-processor which builds on a new theoretical foundation and claims to be 9x faster than Flink. It is also written in Rust, and there is a compiler from SQL to Rust executables. Can you comment on the differences?

sorenbs 3y ago
As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0. Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-with-rust-and-dbsp/ https://liveandletlearn.net/post/vmware-take-3-experience-wi...
- jnewhouse 3y ago
  Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box. Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still just planned for the future. (I'm the co-creator of Arroyo, for context)
  - sorenbs 3y ago
    Thank you! I'll try to get something set up to compare performance of the two on the same machine.
    - necubi 3y ago
      That'd be great! We have versions of most of the nexmark queries and some internal benchmark vs Flink, we'd love to help if you're interested in benchmarking against more systems. Reach out at micah@arroyo.systems.
ryzhyk 3y ago
I'm a developer of DBSP. Our repo now lives here: https://github.com/feldera/dbsp/ https://github.com/feldera/dbsp/. And here is some more benchmarking data vs Flink and Beam: https://github.com/feldera/dbsp/tree/main/benchmark https://github.com/feldera/dbsp/tree/main/benchmark. See feldera.com for more info.
- sorenbs 3y ago
  Thank you! Really happy to see all the activity at the new repo. For a while there I thought the project was dying...

sorenbs 3y ago

As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0. Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-with-rust-and-dbsp/ https://liveandletlearn.net/post/vmware-take-3-experience-wi...

jnewhouse 3y ago
Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box. Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still just planned for the future. (I'm the co-creator of Arroyo, for context)
- sorenbs 3y ago
  Thank you! I'll try to get something set up to compare performance of the two on the same machine.
  - necubi 3y ago
    That'd be great! We have versions of most of the nexmark queries and some internal benchmark vs Flink, we'd love to help if you're interested in benchmarking against more systems. Reach out at micah@arroyo.systems.

ryzhyk 3y ago

I'm a developer of DBSP. Our repo now lives here: https://github.com/feldera/dbsp/ https://github.com/feldera/dbsp/. And here is some more benchmarking data vs Flink and Beam: https://github.com/feldera/dbsp/tree/main/benchmark https://github.com/feldera/dbsp/tree/main/benchmark. See feldera.com for more info.

sorenbs 3y ago
Thank you! Really happy to see all the activity at the new repo. For a while there I thought the project was dying...