4 ms·

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the probl

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data exported from one system to another. If the file sizes, or record counts match, you're in good shape. If it's taking a year to simply copy raw data from one system to another, the enterprise has deeper problems than architecture.

satyrnein 7y ago
If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.
- jlmorton 7y ago
  That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice. Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it. It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.
  - te_chris 7y ago
    The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out
    - lolc 7y ago
      But if I haven't spent the effort to extract it, do I really own it? Let me argue that I don't have it because all my implemented queries turn up none of your data. You wouldn't tax me on gold that hasn't yet been extracted, would you? (End of joke.)
      - ska 7y ago
        But if I haven't spent the effort to extract it, do I really own it? If you collected it, you are responsible for it.
    - yesbabyyes 7y ago
      What I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table} Then you just don't keep red data for longer than 30 days.
- sidlls 7y ago
  One way to avoid this is to have the microservices to publish data changes to some other (monolith) system, like an MQ system with a specified scheme for the payload. On the other hand, the notion that “microservices == completely independent of everything else” is an unrealistic one to hold.
- WoahNoun 7y ago
  Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain. Consider: Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries vs Source Data -> Data Lake -> User Queries vs MonolithDB -> User queries vs MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.
  - averros 7y ago
    This statement is so 20th century. Since then analytical data warehouses (such as Snowflake) got ability to handle schema-less semi-structured data. No more ETL, it's ELT (where transformation, if it is needed, is done in SQL at query time or at intermediate transform time).
  - satyrnein 7y ago
    If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout. Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation. Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.
- hnick 7y ago
  You also lose benefits of microservices if they have to stop and change the data exporting system all the time too, slowing down development. The main benefit of a dumb copy is that the production service is not impacted by reporting, only a copy is. This relates to performance (large queries) but also implementation time.
- mc3 7y ago
  Yeah I think you'd want the microservice to expose an bulk export API point, to an API specification. It might need to transform data most likely and possibly ignore some data. And then you grab data from those and piss them into the lake. The lake now conforms to your published APIs. To me this sounds great. And honestly you should do the same thing with a monolith. Nothing worse than "oh you can't make that schema change because a customer with a read-only view will have broken reports".
- JamesBarney 7y ago
  The key insight and definition of a data lake was pulling data and storing in it's raw form form across the enterprise. https://en.wikipedia.org/wiki/Data_lake https://en.wikipedia.org/wiki/Data_lake The reasons for it are hard to explain succinctly in HN comment but if you look up data lake there will be a lot of explanation. But it basically comes down to "is it better for the data integrators or the data consumers to massage the data?" And data lakes was the insight it's really great when the consumers decide how to massage the data.
- ajxs 7y ago
  This implies that the engineers who lobbied so hard for microservices were even all that concerned with these benefits to begin with, and took this into account when designing the architecture of the system. More often than not, in my experience, the developers involved are more concerned with code ownership than genuine architectural concerns.

satyrnein 7y ago

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

jlmorton 7y ago
That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice. Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it. It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.
- te_chris 7y ago
  The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out
  - lolc 7y ago
    But if I haven't spent the effort to extract it, do I really own it? Let me argue that I don't have it because all my implemented queries turn up none of your data. You wouldn't tax me on gold that hasn't yet been extracted, would you? (End of joke.)
    - ska 7y ago
      But if I haven't spent the effort to extract it, do I really own it? If you collected it, you are responsible for it.
  - yesbabyyes 7y ago
    What I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table} Then you just don't keep red data for longer than 30 days.
sidlls 7y ago
One way to avoid this is to have the microservices to publish data changes to some other (monolith) system, like an MQ system with a specified scheme for the payload. On the other hand, the notion that “microservices == completely independent of everything else” is an unrealistic one to hold.
WoahNoun 7y ago
Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain. Consider: Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries vs Source Data -> Data Lake -> User Queries vs MonolithDB -> User queries vs MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.
- averros 7y ago
  This statement is so 20th century. Since then analytical data warehouses (such as Snowflake) got ability to handle schema-less semi-structured data. No more ETL, it's ELT (where transformation, if it is needed, is done in SQL at query time or at intermediate transform time).
- satyrnein 7y ago
  If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout. Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation. Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.
hnick 7y ago
You also lose benefits of microservices if they have to stop and change the data exporting system all the time too, slowing down development. The main benefit of a dumb copy is that the production service is not impacted by reporting, only a copy is. This relates to performance (large queries) but also implementation time.
mc3 7y ago
Yeah I think you'd want the microservice to expose an bulk export API point, to an API specification. It might need to transform data most likely and possibly ignore some data. And then you grab data from those and piss them into the lake. The lake now conforms to your published APIs. To me this sounds great. And honestly you should do the same thing with a monolith. Nothing worse than "oh you can't make that schema change because a customer with a read-only view will have broken reports".
JamesBarney 7y ago
The key insight and definition of a data lake was pulling data and storing in it's raw form form across the enterprise. https://en.wikipedia.org/wiki/Data_lake https://en.wikipedia.org/wiki/Data_lake The reasons for it are hard to explain succinctly in HN comment but if you look up data lake there will be a lot of explanation. But it basically comes down to "is it better for the data integrators or the data consumers to massage the data?" And data lakes was the insight it's really great when the consumers decide how to massage the data.
ajxs 7y ago
This implies that the engineers who lobbied so hard for microservices were even all that concerned with these benefits to begin with, and took this into account when designing the architecture of the system. More often than not, in my experience, the developers involved are more concerned with code ownership than genuine architectural concerns.