3 ms·

Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether

Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain. Consider: Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries vs Source Data -> Data Lake -> User Queries vs MonolithDB -> User queries vs MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.

averros 7y ago
This statement is so 20th century. Since then analytical data warehouses (such as Snowflake) got ability to handle schema-less semi-structured data. No more ETL, it's ELT (where transformation, if it is needed, is done in SQL at query time or at intermediate transform time).
satyrnein 7y ago
If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout. Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation. Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.

averros 7y ago

This statement is so 20th century. Since then analytical data warehouses (such as Snowflake) got ability to handle schema-less semi-structured data. No more ETL, it's ELT (where transformation, if it is needed, is done in SQL at query time or at intermediate transform time).

satyrnein 7y ago

If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout. Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation. Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.