3 ms·
Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be. By the way, you have to pay for this access or there are some
by qoega 4y ago
Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.
By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves
- srvmshr 4y agoThe access comes with a cost and is pretty expensive unless covered by some academic grants. (Its about 10 grand USD). How expensive would Clickhouse be for data increasing about 1-2 TB per month? We are trying to optimize costs as the Twitter purchase already seems steep
- qoega 4y agoIf you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns. Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/ http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.
- srvmshr 4y agoInteresting. Would you mind talking offline (email in my bio). Would love to take some pointers