5 ms·
Building the world’s fastest website analytics (2021)
- SmoothBrain12 4y agoAt some point, isn't it better to build your own database? And choose the trade offs that fit your project. Or pay/hire someone to do it.
- wiredfool 4y agoAnd that point is when you can’t find someone to host it for you. Though, I’d be sorely tempted to spin up a bfc of clickhouse on bare metal.
- dravita 4y agoThat one was tried, among many others covered in the article.
- JackWritesCode 4y agoClickhouse isn’t really relevant for a lot of us. SingleStore outperforms clickhouse and the latter isn’t MySQL wire compatible. Far more features come with SingleStore too. And if you speak to people running Clickhouse, they’re also maintaining a Postgres set-up. With SingleStore, you get your OLTP and OLAP in one database. So our users & sites table sit in memory (backed up by disk), meaning ultra fast read/write speeds (comparable to clustered/high availability Redis). And then we put our pageviews/events in columnstore (disk) which offers rapid performance for analytical queries.
- Yoric 4y agoThat sounds like a really bad solution, actually. Building a reliable DBMS with any kind of decent performance can easily take years. And by then, it's still not sure that you're doing any better than any of the existing solutions.
- JackWritesCode 4y agoDefinitely not. No regrets after making the move. March next year will be the two year mark.
- amirulmenjeni 4y agoI'm curious how you (and your team I presume) came to decide you need to build your own database?
- JackWritesCode 4y agoWe didn’t decide that, we used SingleStore
- aarondf 4y agoI honestly can't think of any time where "build your own database" is a good solution, unless you're a database company or maybe Google.
- nnopepe 4y agoThese people don't have the inhouse skills to maintain a simple SQL server, let alone optimize their schema (MySQL being too slow to sum/group 11m records in a timely manner? Please.) Attempting to make your own db while being ignorant of competing products would be an absolute disaster
- JackWritesCode 4y agoI completely agree! For the people following back home, who don’t want to move away from MySQL, can you let us know how to achieve a <10ms GROUP BY aggregation with a high cardinality column (11 million distinct values for “pathname”). The query was: SELECT SUM(pageviews) as total, pathname FROM pageviews GROUP BY pathname ORDER BY total DESC LIMIT 10. If we don’t hear an answer from you, I’ll be really upset. Otherwise, we may have to add your ideas to the article!
- kordlessagain 4y agoSounds pretty cool. It's interesting to see how the landscape for this has changed over the last 15 years, since I worked at Splunk. I just started working at FeatureBase a few weeks ago, here in Austin. It's Open Source and we have a 5 minute guide to get it running here: https://docs.featurebase.com/ https://docs.featurebase.com/. We also have a cloud/serverless option. FeatureBase is a B-tree database which uses Roaring Bitmaps. This makes it suitable for doing analytical queries on massive data sets immediately after ingestion. You do have to model the data properly, however.
- deleted 4y ago[deleted]
- Ocha 4y agoI have mixed feelings reading this article. There was no requirement around code refactoring, however multiple solutions were axed because of “how much of our application I was going to need to refactor”. Maybe have that as a hard requirement then and don’t invest time learning tech that doesn’t meet this requirement? Also, was that a joke about the divorce? Either way bad taste and red flag on the personality side for me.
- JackWritesCode 4y agoMy wife thought it was funny ;)
- zX41ZdbW 4y agoWhile reading the article, I found that they are rediscovering the tricks, already well-known for ClickHouse: > The biggest mistake I made was that I kept the UPDATEs in our code (we update the previous pageview with the duration, remove the bounce, etc.). In a few weeks, I'll be moving to 100% append-only by utilizing negative numbers. > For example, if you want to set bounce_rate to 0%, you would write a 1 for bounce_rate on the first pageview and then insert a duplicate with -1 for bounce_rate. And for the duplicate row, you'd have nothing set for pageviews, visits and uniques, so it would all group nicely. This is nearly a copy-paste from ClickHouse documentation. > We shard on UUID, and then we set SiteId as the sort key. We do this because we want to utilize something called "local joins" in SingleStore. Long story short, events can be joined with event_properties (allowing you to have thousands of dynamic properties per event you track), and it's fast. This statement is slightly disappointing as well. No magic, no advantage to ClickHouse. The outcome can be read as follows: "use SingleStore exactly as you'd use ClickHouse and maybe it will work alright". SingleStore advantages: - good compatibility with MySQL dialect; - offers good UPDATE/DELETE for fresh data (invalidated by this article, as they dropped the usage of UPDATE for better performance); - good support for JOINs (invalidated by this article, as they ended up using local JOINs); SingleStore disadvantages: - not open-source, worse brand recognition, the old brand (MemSQL) did not work well too; - performance claims don't validate; - older company and many initial developers no longer work there; Fathom sounds like a toy use case as the data amount is too low. ClickHouse is almost universally selected for similar and larger use cases, with hundreds of billions of events each day. PS. Nevertheless, the post is hilarious: > I was suffering from low energy in the two weeks leading up to this migration, and I was feeling awful throughout migration week, especially on migration day. On Sunday 14th March 2021, two days after we had finished the migration, my wife showed me an already-half-used bag of coffee beans I had been drinking through most of this migration, and the bag said "DECAF." Divorce proceedings are underway.
- AdamProut 4y agoClickhouse is great a single table analytical queries (Group-by + aggregates + filters over a single table). It can match any top tier columnstore database at this, maybe even better at smaller scales, and if that is all you need it will do the job well Beyond this, it has a naive query optimizer and limited ability to run distributed joins. This is why you won't find well known analytical benchmark results like TPC-H and TPC-DS for clickhouse vs other SQL data warehouses. SinglestoreDB has been doing real time analytics much longer then clickhouse and has a much more mature feature set at this point[1][2]. Our go to market is more big enterprise driven so we are definitely much less well known among developers. Revenue share-wise I suspect we are the leader in real time analytics (see disclaimer below - I'm biased but our revenue is reaching thresholds for IPO readiness). Also, if you follow Jacks later posts you'll see he replaces DynamoDB and Redis with SinglestoreDB as well. Now were getting into places Clickhouse doesn't tread at all... Disclaimer: I'm one of the cofounders of MemSQL/SingleStoreDB (still working away on making it better all these years later...). [1] https://www.singlestore.com/blog/the-technical-capabilities-your-database-needs-for-real-time-analytics/ https://www.singlestore.com/blog/the-technical-capabilities-... [2] https://dl.acm.org/doi/abs/10.1145/3514221.3526055 https://dl.acm.org/doi/abs/10.1145/3514221.3526055
- datalopers 4y agoThey would have saved a whole lot of time just using Clickhouse which is a US company and has completely severed ties with Russia [1]. Altinity has offered managed clickhouse since 2020 [2], and now Clickhouse Cloud does too [3] [1] https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine [2] https://altinity.com/cloud-database/ https://altinity.com/cloud-database/ [3] https://clickhouse.com/cloud https://clickhouse.com/cloud
- dravita 4y agoIf you're really confident that your SaaS application will only have that single workload (single table group-by) into perpetuity, it's a workable choice. But, that's a big bet against the reality of the diverse workloads required today in scale-up SaaS applications. Sure, you can stitch together specialized stores to make your own accidental distributed database for your app, but the simpler, more straight-forward and lower-latency alternative available. Yes, I'm a SingleStore employee, but it is worth considering the argument on its merit. It's also worth trying for free in production. The free tier for production is not time-bound, just capacity-bound.
- datalopers 4y agoYou should probably focus on evangelizing the features of your own database instead of broadcasting your ignorance on those of a competitor.
- hodgesrm 4y ago> Sure, you can stitch together specialized stores to make your own accidental distributed database for your app, but the simpler, more straight-forward and lower-latency alternative available. Organizations like Facebook, Amazon, Netflix, Cloudflare, and many, many others have been combining specialized data stores for many years. Stonebraker, Madden, and colleagues summarized the argument for this approach in 2007. [0] At this point I can't think of any major SaaS application I've seen recently that did not have multiple data store types. That includes event streams, which are data stores as well. [0] http://nms.csail.mit.edu/~stavros/pubs/hstore.pdf http://nms.csail.mit.edu/~stavros/pubs/hstore.pdf Disclaimer: I work for Altinity.
- nojvek 4y agoSingleStore is pretty amazing since it supports UPDATE DELETE and joins with other tables. Clickhouse updates are painful. I looked on postgres with citus extension and it only supports appends. Their paper on how they organize the columnstore and row store buffer is pretty neat. https://images.go.singlestore.com/Web/MemSQL/%7Bd1e77ba1-0e3e-4194-8438-2e289cb2b87c%7D_research-paper-cloud-native-transactions-and-analytics-in-singlestore.pdf https://images.go.singlestore.com/Web/MemSQL/%7Bd1e77ba1-0e3... It’s really cool to see usefathom take off doing millions in ARR with two engineers. SingleStore has to been a high leverage choice so they can focus on product. Power to them. Meanwhile we’ve been grappling with a mixture of Mongo, Snowflake and Postgres for more than an year with a decent sized team. The curse of VC funded startups is that with more money and more engineers, a likely outcome is more complexity. The infra becomes a complex beast that takes many quarters/years to build because there isn’t that market pressure to be time and money efficient.
- zX41ZdbW 4y agoClickHouse is almost ubiquitous for web analytics services. For example, here is news from today about Splitbee: https://news.ycombinator.com/item?id=33334104 https://news.ycombinator.com/item?id=33334104 No surprise it is using ClickHouse: https://splitbee.io/blog/new-pricing https://splitbee.io/blog/new-pricing