4 ms·
Connecting Pulsar and Materialize is of interest to me too and something I might try when I find time to do so. Note that Pulsar does have a Kafka compatibility
by chris_overseas 7y ago
Connecting Pulsar and Materialize is of interest to me too and something I might try when I find time to do so. Note that Pulsar does have a Kafka compatibility layer already[0], so it might just work out of the box. If you try this I'd be keen to hear how it goes.
[0] https://pulsar.apache.org/docs/en/adaptors-kafka/ https://pulsar.apache.org/docs/en/adaptors-kafka/
EDIT: I don't think this adaptor will work after all, it works by replacing the Kafka Java client library with its own, so is only applicable to Java applications.
- jamii 7y agoAdding sources is pretty easy. We just need them to provide: * timestamped changes eg "record foo was deleted at time 42" * watermarks eg "I've now sent you all the changes up to time 42" Here's the implementation of the kafka source - https://github.com/MaterializeInc/materialize/blob/master/src/dataflow/source/kafka.rs https://github.com/MaterializeInc/materialize/blob/master/sr.... (There's also a bit of code in the parser/planner to wire up the CREATE SOURCE syntax, but it's fairly trivial.)