4 ms·
Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code
by jabart 3y ago
Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code to run that version. This may be why lightweight work a specific way in that environment, or they need a way to test it at scale that would be hard through a feature flag in the open source product. Lightweight deletes were released to both, and previous roadmaps listed updates as upcoming.
You should be using ReplacingMergeTree if you are doing updates at the current moment.
- unamedrus 3y ago> Lightweight deletes were released to both Because they were contributed by community member, not ClickHouse Inc core team. https://github.com/ClickHouse/ClickHouse/pull/37893 https://github.com/ClickHouse/ClickHouse/pull/37893
- jabart 3y agoCommits show both, yes a community member kicked things over with the initial PR and seems like a team effort to get that feature launched.
- hodgesrm 3y ago> You should be using ReplacingMergeTree if you are doing updates at the current moment. Indeed. Altinity and other community users like ContentSquare made numerous contributions to make it more usable. It's a promising approach to updates at scale and has improved markedly over the last few months. That said you can't currently use RMT very efficiently in S3 because of overall limitations in MergeTree S3 table storage. We need to think about whether the improvements we're proposing will also enhance RMT. Thanks for bringing that up.
- jabart 3y agoYeah, its the don't treat S3 like a disk issue. I'm looking at S3 only for cold storage but need AWS VPC Gateway Endpoint support for S3 access since we are on-premise. *Yes you can use a vpc gateway but need public IPs to waste to setup the BGP/IP routes.
- hodgesrm 3y agoWhat's the "AWS VPC Gateway Endpoint" support you refer to? Do you mean an S3 proxy? If so one of our PRs that just merged may be helpful. [0] [0] https://github.com/ClickHouse/ClickHouse/pull/51749 https://github.com/ClickHouse/ClickHouse/pull/51749
- jabart 3y agoI don't think that will work. Here is the issue logged. This is about a networking feature used for AWS Direct Connect users to access S3 over a dedicated private connection. https://github.com/ClickHouse/ClickHouse/issues/53761#issuecomment-1693760623 https://github.com/ClickHouse/ClickHouse/issues/53761#issuec...
- hodgesrm 3y agoThanks, I had not seen this one.