3 ms·

I worry that this promotes the data architecture philosophy that is currently in vogue in tech companies, but is actually a bad fit for many traditional enterpr

by wenc 7y ago

I worry that this promotes the data architecture philosophy that is currently in vogue in tech companies, but is actually a bad fit for many traditional enterprises. Most of the time, data architecture needed depends on use case. Tech/web companies deal with massive amounts of unstructured/semi-structured data ingested at a fast clip, so the architectural thinking here works. However I would argue that many traditional enterprises whose major sources of data are primarily highly structured (SQL databases), a lake is actually not needed. A young data engineer working in a traditional enterprise, enamored with the idea of data lakes say, might try to ETL SQL databases into an object store (add RBAC etc.) only to rebuild it back out into a data warehouse. This will almost always turn out to be the wrong approach. The simpler and more manageable approach is actually to federate existing databases, add cataloging etc. and not even use a data lake at all.

kfk 7y ago
I think by lake they mean s3 and similar alternatives (like hdf is for hadoop). This is not crazy as processing s3 files is quite easy. What you say is true but the problem is that no analyst deals only with one db, they have to deal with an increasing number of data sources, that’s what the lake is for. I also don’t believe you need a warehouse after the lake and that one source of truth is pipe dreams, so I’m with you this architecture is not 100% what companies probably need.
- wenc 7y ago
  > no analyst deals only with one db, they have to deal with an increasing number of data sources Which is why I mentioned federation. Most data sources in an enterprise are SQL native or accessible and it does not make sense to dump a SQL database into S3 just to be able to combine it with other forms of data. Federation means you can operate across multiple databases (eg do Cross database joins etc.)

by wenc 7y ago

kfk 7y ago

I think by lake they mean s3 and similar alternatives (like hdf is for hadoop). This is not crazy as processing s3 files is quite easy. What you say is true but the problem is that no analyst deals only with one db, they have to deal with an increasing number of data sources, that’s what the lake is for. I also don’t believe you need a warehouse after the lake and that one source of truth is pipe dreams, so I’m with you this architecture is not 100% what companies probably need.

wenc 7y ago
> no analyst deals only with one db, they have to deal with an increasing number of data sources Which is why I mentioned federation. Most data sources in an enterprise are SQL native or accessible and it does not make sense to dump a SQL database into S3 just to be able to combine it with other forms of data. Federation means you can operate across multiple databases (eg do Cross database joins etc.)