4 ms·
Hi, Thanks for the answer. In our MYSQL database, we wanted to count number of pageviews for each website. So, the query was like SELECT COUNT(id) FROM page_v
by supz_k 6y ago
Hi,
Thanks for the answer. In our MYSQL database, we wanted to count number of pageviews for each website. So, the query was like
SELECT COUNT(id) FROM page_views WHERE website_id = x AND created_at > (start of the month)
For some websites, there are 20 million+ pageviews for each month. MYSQL goes through all of those 20 million rows (according to EXPLAIN) even with index or composite index (It took a few seconds). So, we had to pre-calculate the number of pageviews each hour and show it to the user. Then, things were worst when we had to show analytics. There we had to group by month or day. So, it took more time.
That's when we started finding a solution. And, I learned that there's something called "Analytical Databases", which are designed for those analytics purposes. How they work is completely different from MYSQL.
And, here's a benchmark on MYSQL vs Clickhouse. http://mafiree.com/blogs.php?ref=Benchmark-::-MySQL-Vs-ColumnStore-Vs-Clickhouse http://mafiree.com/blogs.php?ref=Benchmark-::-MySQL-Vs-Colum...
MYSQL does a pretty good job for many things. However, when it comes to analytics, I think it's better to use an analytical database.
As each of your users has its own database, there won't be any issues at all. In our case, all of our clients' pageviews are stored in one table, which grows at 35m records per month.
We're also new to this analytical databases thing. I'd like to know your thoughts. :)
- XCSme 6y agoThanks for the extra details! I built userTrack mostly as a cheaper alternative for smaller businesses, so they can still have access to good tools/data without paying enterprise prices, so the goal was never to support 1M+ monthly sessions, thus I never spent too much time looking into heavy-traffic performance or scaling. > For some websites, there are 20 million+ pageviews for each month. MYSQL goes through all of those 20 million rows (according to EXPLAIN) even with index or composite index (It took a few seconds). So, we had to pre-calculate the number of pageviews each hour and show it to the user. Yes, count usually "goes" through all the rows when using more complex conditions, the way to improve this is usually, as you also did, to pre-calculate the counts using databse triggers (whenever a new row is inserted, the trigger will update the total count). I am not familiar with Clickhouse, but I assume for being so fast it provides at lot fewer features/options to store and manipulate data. How hard was the transition to Clickhouse? Where you able to easily convert the DB schema and all quries?