3 ms·
Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf
by kod 9y ago
Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields.
If you're equating kafka topics with the idea of a schema, you can add topics.
- l8again 9y agoTo add to this. Having implemented a similar log-based architecture, I would say that it is much simpler data infrastructure than having a central RDBMS considering the use case. Remember we need to deal with several consumer applications with their own respective optimization models. Postgres is a perfect choice for a given application's local datastore, while for a different application they may want to use ElasticSearch as their datastore. However, the "source of truth" remains free of any such optimization requirements. You simply save your "messages", "event", "facts", whatever you want to call it in its purest form preferably immutable, and let the consumer apps create/recreate their local datastores as they deem fit.