4 ms·

The elephant (no pun intended) in the room continues to be most data in the enterprise isn't big data. You don't need mapreduce when your data set fits in RAM.

peterthehacker 5y ago
Yes! And if it doesn’t fit in RAM, then most businesses will only need an OLAP data warehouse, like Snowflake or Redshift.
- threeseed 5y ago
  And if you want to join data from different SaaS and internal systems e.g. Google Analytics and a Pega decisioning system. Are you going to spend months upfront carefully modelling the data in order to ingest it making sure to handle schema and DQ issues etc. All to support one use case who only needs a handful of fields. No. Which is why data lakes exist. Because it's cost effective. You simply dump the data and ask the Engineer or Data Scientist building the use case to do the heavy lifting rather than a centralised data team.
  - peterthehacker 5y ago
    There are integration companies that solve this specific use-case. I’ve used Fivetran [0] and highly recommend it. They will extract-load data from your SaaS to your warehouse and your data scientists can run SQL against the tables. Their most popular warehouses are Redshift and Snowflake. So you can still use a centralized data warehouse without dedicating internal resources to the integrations. [0] https://fivetran.com/ https://fivetran.com/
    - ak217 5y ago
      What I find amazing is that Fivetran is a bunch of glue code to forward data between different APIs and database formats, and it's legitimately useful, in part because when the upstream API breaks they go and fix the connectors for you instead of you having to deal with the resulting emergency... but it's only needed because data interchange standards are in such poor shape. If users demanded that SaaS products make data/event streams/replication logs available via robust and standardized APIs, a lot of the use cases for Fivetran would disappear.
kfk 5y ago
Sort of. The issue is coupling data with sql databases doesn’t play well with even basic inferential statisics so you end up replicating data based on analytics use cases. Having parquet files in a cheap storage system that can be queried with sql, python, R, etc. is very convenient. What I don’t get is why so little investments are going in making data lakes safer and easier to govern with proper access controls
- manigandham 5y ago
  You can run those queries straight from relational databases too, since everything ends up in the same dataframe structure for processing in the end. There are tons of data source adaptors now and all the modern OLAP databases support various deeper integrations to run stats/ML directly.
threeseed 5y ago
This is simply nonsense. Big data has never been just about MapReduce. It has always revolved around the concept of a data lake, with data stored as objects, a series of data engineering pipelines moving data around and a query engine on top. And in almost every enterprise company this is the high level architecture you see today. And this model only continues to grow in popularity as the use of siloed SaaS products drives data sprawl and the need for tools like Spark, Fivetran etc to move it all back to a centralised data lake for analysis.
- manigandham 5y ago
  Big data just means big data. A data lake is one way to deal with it. It's convenient but not always the best.
icedchai 5y ago
The real problem is that even when it doesn't fit in ram, you still don't need mapreduce.

peterthehacker 5y ago

Yes! And if it doesn’t fit in RAM, then most businesses will only need an OLAP data warehouse, like Snowflake or Redshift.

threeseed 5y ago
And if you want to join data from different SaaS and internal systems e.g. Google Analytics and a Pega decisioning system. Are you going to spend months upfront carefully modelling the data in order to ingest it making sure to handle schema and DQ issues etc. All to support one use case who only needs a handful of fields. No. Which is why data lakes exist. Because it's cost effective. You simply dump the data and ask the Engineer or Data Scientist building the use case to do the heavy lifting rather than a centralised data team.
- peterthehacker 5y ago
  There are integration companies that solve this specific use-case. I’ve used Fivetran [0] and highly recommend it. They will extract-load data from your SaaS to your warehouse and your data scientists can run SQL against the tables. Their most popular warehouses are Redshift and Snowflake. So you can still use a centralized data warehouse without dedicating internal resources to the integrations. [0] https://fivetran.com/ https://fivetran.com/
  - ak217 5y ago
    What I find amazing is that Fivetran is a bunch of glue code to forward data between different APIs and database formats, and it's legitimately useful, in part because when the upstream API breaks they go and fix the connectors for you instead of you having to deal with the resulting emergency... but it's only needed because data interchange standards are in such poor shape. If users demanded that SaaS products make data/event streams/replication logs available via robust and standardized APIs, a lot of the use cases for Fivetran would disappear.

kfk 5y ago

Sort of. The issue is coupling data with sql databases doesn’t play well with even basic inferential statisics so you end up replicating data based on analytics use cases. Having parquet files in a cheap storage system that can be queried with sql, python, R, etc. is very convenient. What I don’t get is why so little investments are going in making data lakes safer and easier to govern with proper access controls

manigandham 5y ago
You can run those queries straight from relational databases too, since everything ends up in the same dataframe structure for processing in the end. There are tons of data source adaptors now and all the modern OLAP databases support various deeper integrations to run stats/ML directly.

threeseed 5y ago

This is simply nonsense. Big data has never been just about MapReduce. It has always revolved around the concept of a data lake, with data stored as objects, a series of data engineering pipelines moving data around and a query engine on top. And in almost every enterprise company this is the high level architecture you see today. And this model only continues to grow in popularity as the use of siloed SaaS products drives data sprawl and the need for tools like Spark, Fivetran etc to move it all back to a centralised data lake for analysis.

manigandham 5y ago
Big data just means big data. A data lake is one way to deal with it. It's convenient but not always the best.

icedchai 5y ago

The real problem is that even when it doesn't fit in ram, you still don't need mapreduce.