You're not dreaming, perhaps you might just be underestimating the complexity of building an analytics stack. Here's my suggestion: - Start with picking you d

You're not dreaming, perhaps you might just be underestimating the complexity of building an analytics stack. Here's my suggestion: - Start with picking you data warehouse. 500 GB/mo can exceed regular SQL databases' performance, so the next cheapest option is to choose a cloud based analytical datastore: BigQuery, Snowflake, Redshift, Athena. - Next, work on your ETL. There are some open source tools, they all require a large amount of maintenance: - https://github.com/snowplow/snowplow - Talend - https://www.singer.io - http://www.fluentd.org/ - Kafka & Kafka Connect - Retention, cohorts, and other analysis - you would probably need to write your own queries, and schedule them with a cron like tool: - cron ( :) ) - airflow - luigi - For visualization and charting, I know of two good open source tools: - Redash.io - http://airbnb.io/superset/ One thing you mentioned which might be hard to do is to join across keys from each source. Analytical datastores are not very good at that, as tables are partitioned, and joining tables requires that you plan ahead what you are joining on, and partition your tables accordingly. To support joining records in your ETL process, I'm not sure there are easy ways to implement this (let me know if you find one, we are working on it too :) ). You can look at Kafka Streams or Apache Storm, as a starting point