9 ms·
No, QuestDB is not Faster than ClickHouse
- thegeomaster 4y agoSounds like they didn't re-do the QuestDB benchmark with same change to the indexes, and so their claim is that Clickhouse is 27x faster with a specific index than QuestDB without that index. Which is not a fair comparison. Also, the tone of the post sounds really arrogant. They try to hide it a bit, I feel, but it just seeps through.
- axlee 4y agoI didn't really read it as arrogant, more as annoyed about a mischaracterization that was disparaging their product.
- SOLAR_FIELDS 4y agoIt's also part of a longer trend of saber rattling between these vendors - there's a history of these types of posts also from TimescaleDB: https://news.ycombinator.com/item?id=29096541 https://news.ycombinator.com/item?id=29096541
- qoega 4y agoThere is a small list of vendors that do not forbid to run benchmarks with their systems. https://cube.dev/blog/dewitt-clause-or-can-you-benchmark-a-database https://cube.dev/blog/dewitt-clause-or-can-you-benchmark-a-d... That is why there is a small subset of vendors that are being 'attacked' by this comparisons.
- bombcar 4y agoMore and more we start to see why these forbids are in place.
- Dzugaru 4y agoWell, I don't know how QuestDB works, and I couldn't find anything in the original benchmark, but probably they already have some sort of (geo)index in place? It's really strange to search geo-data by scanning the whole surface of the Earth. The point that Clickhouse outperforms this by just sorting on one axis (and even not using any fancy 2D indices) is reasonable.
- tomhallett 4y agoI was curious to hear more details about this statement - "while QuestDB utilizes its full indexing strategy to read just a tiny fraction of the actual data". Did QuestDB create indexes in their QuestDB benchmark but just not mention it? Are there geoindexes which are automatically enabled which do help (but are of less value in the general sense from Clickhouse' perspective)?
- twoodfin 4y agoI don't know how QuestDB is implemented in any detail, but this statement struck me as confused. My understanding is that for this query, QuestDB is performing a full scan of the relevant columns, and the point of the blog post was how fast their JIT engine for filtering makes this.
- olluk 4y agoNo, there are no indexes in QuestDB in the article. None. Zero. That's bold mistake in the ClickHouse article. Should be named Yes, QuestDb is Faster.
- Dzugaru 4y agoYeah, I've read more carefully and it seems they're doing full scan.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- olluk 4y agoThere were 2 queries in the QuestDB benchmark over the same table. ClickHouse didn't even try to match both of them choosing one as a victim. I guess that's what happens when you optimise the data storage for one query.
- PeterZaitsev 4y agoThis response illustrates important point - if you're expert in technology A and compare it to technology B, you're not expert in, comparison is very likely to be unfair. I very much would like to see vendors at least to follow Journalist ethics and reach out to their competition for optimization comments and suggestions before publishing it, so others are given a chance to suggest optimizations
- klysm 4y agoYeah this happens a lot. I like it when people maintain a repo that accepts changes for the comparison
- snikolaev 4y agoThen you should like https://db-benchmarks.com/ https://db-benchmarks.com/
- PeterZaitsev 4y agoGreat idea.
- hodgesrm 4y agoAgree. Or just load test on your own software, publish how you did it, and let other vendors respond for themselves.
- DeathArrow 4y ago>This response illustrates important point - if you're expert in technology A and compare it to technology B, you're not expert in, comparison is very likely to be unfair. To have fair benchmarks, one organization can manage the test requirements and submissions while expert in a said technology can submit their best efforts to be measured and compared to others. Like Tech Empower Benchmarks used to work.
- noxvilleza 4y agoIs there an existing named adage for something like "if one creates a benchmark in order to rank general performance of some products, some of those products will ultimately sacrifice general performance in order to optimize for that benchmark"?
- nathanwh 4y agohttps://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law You stated it almost directly.
- noxvilleza 4y agoOh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.
- bombcar 4y agoIt's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.
- gfody 4y agothere are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/ https://www.singlestore.com/blog/tpc-benchmarking-results/
- qoega 4y agoIt is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.
- bluestreak 4y agoOur article in question can be found here: https://questdb.io/blog/2022/05/26/query-benchmark-questdb-versus-clickhouse-timescale/ https://questdb.io/blog/2022/05/26/query-benchmark-questdb-v... The intent of the article was to showcase JIT-optimised WHERE clause and we did not use any indexes on QuestDB.
- deleted 4y ago[deleted]
- datalopers 4y ago
- bluestreak 4y agoI am in fact very proud of my team, who worked very hard on both implementation and the article. It is disappointing to read unfounded insults where we made every effort to be fair.
- Dzugaru 4y ago"while QuestDB utilizes its full indexing strategy to read just a tiny fraction of the actual data" Can you please elaborate on this?
- bluestreak 4y agoFull disclosure: I am CTO of QuestDB and I took part in JIT implementation. The quote above is not mine, it was written by Clickhouse staff. "utilizes its full indexing strategy" statement is false and is news to me.
- pepemon 4y agoSo, QuestDB is faster or not? I'm puzzled now!
- deleted 4y ago[deleted]
- gauravphoenix 4y agoI have always felt that DB benchmarks are useless, always use your own dataset https://gauravkumar.blog/performance-benchmarks-are-useless.html https://gauravkumar.blog/performance-benchmarks-are-useless....
- ceejayoz 4y agoThat has the confounding variable of how good you are at configuring each database option. I can configure Postgres fairly well. I have little chance of knowing if I’m getting good performance out of most others without a serious time investment.
- nojito 4y agoThis is why the commercial offerings do not allow you publish benchmarks.
- PeterZaitsev 4y agoWhich is horrible thing. Even bad benchmarks often create create discussions
- nojito 4y agoNot true at all. Most people take benchmarks as gospel because they value their time.
- capableweb 4y agoAll benchmarks are always useless, in 90% of the cases. They could maybe give some baseline understanding, but it's important to always do your own benchmarks as your performance can be very different than what the benchmark showed, simply because the data/data structures are slightly different. Do your own benchmarks people!
- beltsazar 4y agoI wouldn't go as far as to say that benchmarks are useless, but I agree that when looking at a benchmark, it's important to be aware of how similar its data distribution and query patterns are compared to your own.
- intellix 4y agoeverytime I see the name Clickhouse I think I'm reading about a CRM tool. It's the worst name for a database ever.
- IshKebab 4y agoI agree. I actually think it's ideal for something we need at work (basically storing a crap ton of logs) but it doesn't remotely sound like it from the name. Sounds like some kind of front-end analytics tool.
- gigawatts 4y agoI have a ton of respect for the clickhouse team. We ran a massive cluster storing trillions of events across thousands of servers for years to serve a real-time reporting/ api use case and the tech is blazing fast and never once let us down.