8 ms·
How Netflix uses Druid for realtime insights
- shaklee3 7y agoCan anyone from Netflix comment on why not clickhouse? We tried druid, and the performance was pretty bad compared to a clickhouse.
- modarts 7y agoCan confirm clickhouse is generally faster across most typical workloads
- jpgvm 7y agoClickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of that stuff yourself though these days it can read from Kafka directly which is useful. Some things that are important for Clickhouse vs Druid at big scale is the rather large difference in indexing approaches. Clickhouse uses bloom filters and other probabilistic data structures to index large chunks of data, for the most part though actually checking for rows requires a full scan of that chunk to strip false positives. This is different to Druid which uses full inverted indices for dimension filtering. The tradeoff is basically Clickhouse is cheaper, especially when scaling out but Druid is faster especially when the cluster is under heavy concurrent query load, like serving analytics dashboards or data exploration interfaces to users. Clickhouse excels when you want to scan most but not all the data most of the time. Namely reporting or bulk analytics queries that will hit most rows in a block. I consider both to be excellent databases.
- danielbln 7y agoIn our experience, on a significantly smaller scale, Clickhouse is vastly easier to operate compared to Druid, with all of its various components that all have various knobs and dials to configure and have to be orchestrated.
- jpgvm 7y agoYes this is definitely also true. Druid complexity is coming down a bit compared to where it started. These days you need brokers, middlemanagers and historicals - for queries, ingestion and storage respectively. In the past to do batch ingestion it also required Hadoop but there is now a native parallel batch ingestion system that runs on the middlemanagers as worker tasks that can read from S3/GCS/existing Druid segments. Druid is by far the more complex but you get a lot for it and with k8s it's not as hard to run/manage as it was in the past.
- gianm 7y agoDruid committer here. Fwiw, Druid was designed to run on huge clusters and that really shows up in the multi-process architecture. The idea is that if you separate the components needed for ingestion, historical processing, query routing, and coordination, then there are two benefits: they don't interfere with each other (spikes in ingestion load won't interfere with ability to query historical data), and also you can scale each one individually for your workload. You could even auto-scale some of them. For example, the original Druid cluster was operated with load-based auto-scaling for the ingestion processes. That being said we are currently working on reducing the number of processes to 4 (from the current 6) for a "standard" setup. The main reason is that at smaller scale there isn't as much of a purpose to having a larger number of processes. We're also working on removing some of the knobs. Actually, depending on what version you originally looked at, many of them might already be gone.
- danielbln 7y agoIt's been a few years since we evaluated Druid. It's great to hear that you're simplifying things, especially for smaller setups!
- devmunchies 7y ago> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?
- Joof 7y agoHave you worked at a large company before? You don't!
- catnzhat 7y agoYou acknowledge that fact and if you're making a severe change you either run a longer test or you break up the large change into manageable chunks (Google Maps is a great example of this). For something that is effectively binary and can't be changed incrementally the only way is to extend the evaluation window or make a call with limited information.
- 2rsf 7y agoregressions due to "users are used to" usually goes away after a while, you'll see a temporary dip in the results. If you are convinced it's better for the long tun then what's the point in measuring ?
- Twirrim 7y ago> How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run? That's assuming that the update contains a UI/UX change. A lot of updates that will roll out won't include that, they'll be fixing or optimising things.
- woolly 7y agoIs it possible to view this article without creating a medium account?
- tschmidleithner 7y agoIt looks like Medium is opening their paywall modal based on viewed articles (and cookies). Therefore, you could try private browsing.
- shivaas 7y agocan confirm. opening link in incognito worked for me.
- manigandham 7y agoDelete your cookies for the site.
- magicalhippo 7y ago> How can we be confident that updates are not harming our users? Maybe this is why they finally figured out the auto-playing trailer stuff was a horrible change (I cancelled my account over that). One can only hope.
- zerop 7y agoSo this is behind paywall. I am sure Netflix is not marking this story to be behind paywall and Medium can not put this story behind paywall without consent from Netflix.
- alexott 7y agoYep, i was wondering how company can put their posts that should their “super skills” behind paywall..
- manigandham 7y agoIt's not. It's medium.com asking for signin (which is a free account) but you can delete your cookies and reload to view.
- zerop 7y agoI am already logged in and it still is a featured story. It's counted against 3 features story I can read in a month. So it is behind paywall. C
- rad_gruchalski 7y agoThen log out. It’s not a paywall. Log out and you can read it. Medium is playing you.
- tjoff 7y agoProper response: delete account and don't visit medium again. Even better: sites such as hn should never allow links to sites employing dark patterns.
- manigandham 7y agoPeople are interested in the content. No need for HN to start filtering because the blog service. You can avoid the site if you want.
- dikei 7y agoI tried setting up a Druid cluster once, they have so many components that it's hard to manage. Also Druid didn't not support SQL at the time and we don't want to learn its weird query language due to the lack of documentation.
- polskibus 7y agoI wonder if materialize.io could handle such workloads at this stage.
- adamfeldman 7y agoMaterialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture https://materialize.io/docs/overview/architecture Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.
- derefr 7y agoMind you, given the “Timely Dataflow” abstraction Materialize operates on top of, if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows (and, further back, retrieve the source rows) required to “render” the particular result-rows you ask for. (Sort of like how Excel, in memory-constrained conditions, only computes the intermediate cell-values required to render the cells currently in view, and thus, if an intermediate cell requires an `XmlHttpRequest` call to resolve, that call won’t fire if the cell’s value doesn’t need resolving yet.) Because of that, you don’t really need to scale Materialize in a sharding sense. You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query launched by, say, Citus—where Citus was thinking it was talking to a bunch of Citus shard nodes each holding a table-shard named X, but was actually talking to Materialize nodes each holding a matview named X. As long as the query sent from the map-reduce job to each node is constrained in its WHERE clause to only the part of the data it expects to get from that node—rather than relying on the node to know what data it has—then the Materialize nodes would each just do the work required to supply that data (including only pulling in the parts of the configured sources required to compute that result.) That’s just my intuition from how Materialize presents itself as PG-wire-protocol compatible, though; I haven’t tried this myself, and there might be some footguns in the path of anyone really trying to implement it. And, of course, this is all irrelevant the moment you write a query that needs a pure reduce (e.g. the computation of a current finite-state-machine state over an event-stream source) rather than a map-reduce. Druid/Clickhouse/etc. can probably “scale” those, in at least the Hadoop “move the job around between each serial stage, so each stage has data-locality for the data of that stage” sense; while Materialize would give you no benefit at all in such a job over just querying a plain PG view defined on top of a Foreign Data Wrapper source.
- PedroBatista 7y ago"To keep reading this story, create a free account." Seriously? I know it's the Medium hustle but someone at Netflix should know better.
- yingw787 7y agoI think one cool feature Medium can introduce is enterprise permissioning. If you have an enterprise plan and pay Medium oodles of money, you should maintain your own permissioning control and who sees your articles while still having access to the Medium social network. Best of both worlds.
- dehrmann 7y ago> oodles of money Medium isn't worth oodles, though. Looking at Squarespace and Wordpress pricing, for a company's tech blog, it's worth $25-$100 per month. Unless there's value from the Medium brand...
- dt3ft 7y agoI guess it costs slightly less to hire someone who can manage a basic word editor, than someone who can manage a self-hosted website.
- errantmind 7y agoFirefox add-on to avoid the hassle: https://github.com/iamadamdev/bypass-paywalls-firefox https://github.com/iamadamdev/bypass-paywalls-firefox
- res0nat0r 7y agoI can read the whole article and am not signed in. It also shows up in incognito mode. That popup is usually asking you to create an account, but you should be able to just close it and continue. I see the whole article for sure behind no paywall and am not logged in.
- hankjacobs 7y agoThat should be fixed. Thanks for calling it out.
- alexandercrohde 7y agoI wonder how this choice compares to using a hosted solution like Splunk (both in capabilities and cost).
- prtaylor 7y agoThats an interesting thought.
- siddcoder 7y agoWondering if any performance comparison was done with Apache Pinot?