4 ms·

At this point everyone should be using an evolvable (thrift, protocol buffers, avro, etc) schema format when they are storing or transmitting their data if they

At this point everyone should be using an evolvable (thrift, protocol buffers, avro, etc) schema format when they are storing or transmitting their data if they want to run an always on service - there is no downtime for migrations in the real world. Trying to do this ad-hoc with JSON is a lost cause and will eventually lead you to failure at runtime or worse, data loss situations.

mixmastamyk 10y ago
Anyone have a good blog post handy on this?
- spullara 10y ago
  This is a related post that talks about the entity store that I built for my startup that was ultimately acquired by Twitter. They internally had their own similar store called ThriftStore that worked similarly. Google also builds their systems like this using protocol buffers. It is a pretty easy pattern and could theoretically be done with JSON if you provide some kind of schema and evolution strategy on read. https://javarants.com/havrobase-a-searchable-evolvable-entity-store-on-top-of-hbase-and-solr-d305f90a3eaa#.rrdmnf12m https://javarants.com/havrobase-a-searchable-evolvable-entit...
crucini 10y ago
JSON isn't un-evolvable. In fact, thrift can serialize to JSON. What makes thrift evolvable in practice is that we don't remove fields and don't add mandatory fields. The same discipline can be applied to JSON definitions. Well thrift also tags all fields with integers, so a consumer with an older schema can parse a record with a newer schema, skipping the new fields. Of course JSON trivially has this property. Maybe the key here is "ad-hoc"; something like JSON-schema is needed.
- spullara 10y ago
  Yep, I mentioned below that using JSON as a serialization format is fine but you still need to specify a schema and understand what happens when you read data written by newer/older code.

4 ms·

At this point everyone should be using an evolvable (thrift, protocol buffers, avro, etc) schema format when they are storing or transmitting their data if they

by spullara 10y ago

mixmastamyk 10y ago
Anyone have a good blog post handy on this?
- spullara 10y ago
  This is a related post that talks about the entity store that I built for my startup that was ultimately acquired by Twitter. They internally had their own similar store called ThriftStore that worked similarly. Google also builds their systems like this using protocol buffers. It is a pretty easy pattern and could theoretically be done with JSON if you provide some kind of schema and evolution strategy on read. https://javarants.com/havrobase-a-searchable-evolvable-entity-store-on-top-of-hbase-and-solr-d305f90a3eaa#.rrdmnf12m https://javarants.com/havrobase-a-searchable-evolvable-entit...
crucini 10y ago
JSON isn't un-evolvable. In fact, thrift can serialize to JSON. What makes thrift evolvable in practice is that we don't remove fields and don't add mandatory fields. The same discipline can be applied to JSON definitions. Well thrift also tags all fields with integers, so a consumer with an older schema can parse a record with a newer schema, skipping the new fields. Of course JSON trivially has this property. Maybe the key here is "ad-hoc"; something like JSON-schema is needed.
- spullara 10y ago
  Yep, I mentioned below that using JSON as a serialization format is fine but you still need to specify a schema and understand what happens when you read data written by newer/older code.