3 ms·
Like other I would say start with a problem which interests you. To mimic the real world try to have extraction from variety of data sources (rdbms, NoSQL, fil
by indogooner 8y ago
Like other I would say start with a problem which interests you.
To mimic the real world try to have extraction from variety of data sources (rdbms, NoSQL, files (different formats csv/json), APIs (salesforce)). Once you have different sources, extract the data to your datalake built on S3/GCS/HDFS. Once the data is present you need to integrate with tools which can extract value. You can use Vendor specific tools like Athena/BigQuery or open source like Presto/Impala/Hive. You can do analytics where you require filtering, cleansing, joining various datasets. You can also look at storing the results in different formats so that other tools like Tableau can use them. To orchestrate all of this you can use Azkaban or Airflow.
My suggestion is slightly biased towards Hadoop ecosystem but the good thing is that most tools here have Open source alternatives.