3 ms·

> Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible. Sure but backwards and bi-

by mvc 9y ago

> Through its various protocols Kafka topics can be configured to be guaranteed forwards, backwards, or bi-directionally compatible. Sure but backwards and bi-directional compatibility inhibits the evolvability of schemas. Something as simple as adding a new required field for example is not backward compatible in avro. I understand why this is, and that in a huge database, it's not that simple in SQL either. But in 95% of databases it actually is quite simple.

bonesss 9y ago
Ummm sure about that? Something as simple as adding a new required field and maintaining compatability is the explicit use case for Avro. I personally do this in production... I know LinkedIn does too... Seriously: that would be a show stopper beyond show stoppers, and would make Kafka useless. Backwards and bi-directional compatibility are the specific facilities that enable safe schema evolution. From the docs: "Backward compatibility means that data encoded with an older schema can be read with a newer schema... Forward compatibility means that data encoded with a newer schema can be read with an older schema... Full compatibility means schemas are backward and forward compatible[:]...we can evolve the schemas in a fully compatible way: old data can be read with the new schema, and new data can also be read with the old schema." A new required field _with a default value_ (important!), is backwards compatible. Same deal for hardcoded SQL updates: no default value, no backwards compatibility. Default value? Backwards compatible because clients that cannot provide a value get one, by default. And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy... tens and hundreds of parallel databases, active datawarehouse operations, multiple federated service gateways, real-time analytics, and huge batch systems for BigData, all acting in concert 24/7: that's the kind of environment that makes the "95%" simple jobs massively costly due to the logarithmic interaction of complexity when numerous systems are considered at once. Without some kind of dataflow separation these intrinsic complexities trend towards intractable system configurations, hence the ESB boom and bust. Kafka is not for your applications. Kafka is for your Enterprise with many, many, applications.
- mvc 9y ago
  > A new required field _with a default value_ In what sense is this field required when you attempt to produce a value using the new schema, and everything is fine if you omit the "required" field? > And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy We don't have 100s but certainly 10s of databases. But we use a schema evolution policy because we're attempting to use kafka as the system of record for our event log.

by mvc 9y ago

bonesss 9y ago

Ummm sure about that? Something as simple as adding a new required field and maintaining compatability is the explicit use case for Avro. I personally do this in production... I know LinkedIn does too... Seriously: that would be a show stopper beyond show stoppers, and would make Kafka useless. Backwards and bi-directional compatibility are the specific facilities that enable safe schema evolution. From the docs: "Backward compatibility means that data encoded with an older schema can be read with a newer schema... Forward compatibility means that data encoded with a newer schema can be read with an older schema... Full compatibility means schemas are backward and forward compatible[:]...we can evolve the schemas in a fully compatible way: old data can be read with the new schema, and new data can also be read with the old schema." A new required field _with a default value_ (important!), is backwards compatible. Same deal for hardcoded SQL updates: no default value, no backwards compatibility. Default value? Backwards compatible because clients that cannot provide a value get one, by default. And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy... tens and hundreds of parallel databases, active datawarehouse operations, multiple federated service gateways, real-time analytics, and huge batch systems for BigData, all acting in concert 24/7: that's the kind of environment that makes the "95%" simple jobs massively costly due to the logarithmic interaction of complexity when numerous systems are considered at once. Without some kind of dataflow separation these intrinsic complexities trend towards intractable system configurations, hence the ESB boom and bust. Kafka is not for your applications. Kafka is for your Enterprise with many, many, applications.

mvc 9y ago
> A new required field _with a default value_ In what sense is this field required when you attempt to produce a value using the new schema, and everything is fine if you omit the "required" field? > And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy We don't have 100s but certainly 10s of databases. But we use a schema evolution policy because we're attempting to use kafka as the system of record for our event log.