3 ms·
I have systems with 100 trillion records and 100 billion events / day using a technology that runs on CentOS, Ubuntu, FreeBSD, Windows Server and Solaris. What
by StreamBright 10y ago
I have systems with 100 trillion records and 100 billion events / day using a technology that runs on CentOS, Ubuntu, FreeBSD, Windows Server and Solaris. What am I doing wrong? Why should I vet a technology that does not run on the platform that our company has chosen, it is approved by security and compliant with all the corporate requirements while the only platform ClickHouse is using is not?
- flr_null 10y agoWhat open source columnar storage do you use for 100 trillion records? It's very impressive.
- StreamBright 10y agoHive/Tez + ORC on HDFS and on S3. I do not think it is very impressive just by the numbers mentioned so far. The performance of data access like how long an average query runs is much more interesting. You can easily store 100T lines but it is much harder to make it queryable. There is a lots of interesting work being done in that area, many in-memory engines are out there to provide access of a subset of the entire dataset at much faster speed.
- flr_null 10y agoOf course I'm asking with context of massive parallel realtime queries from customers as it's going in yandex metrika or google analytics. And yes 100T lines with Hive+ORC it's not an issue, but it's another league. I assume that only Kudu is a competitor for ClickHouse now. Or may be Greenplum. Thank you for your answer.
- pnathan 10y agoFor understanding, for analysis, for education. It's typically a mistake to blow off systems based on compliance checkboxes. Policies can be changed, systems can come into compliance.