13 ms·
Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
- otabdeveloper4 4y ago300Gb per node, am I reading this right? Seems extremely low, I expect at least three orders of magnutide more on a real production system. Does Cassandra really suffer from performance problems at scale?
- forgot_old_user 4y agohmm you expect to see 300,000GB or 300TB per node? What does that mean?
- otabdeveloper4 4y agoIt means that if you need a whole JVM instance to service a tiny chunk of data that's less than even a terabyte then you're doing something extremely wrong. Leaves a bad impression of Cassandra, personally.
- deleted 4y ago[deleted]
- thekozmo 4y agoWe (I'm a co-founder) run the following: - 1B (10^9) rows/sec with 86 servers https://www.scylladb.com/2019/12/12/how-scylla-scaled-to-one-billion-rows-a-second/ https://www.scylladb.com/2019/12/12/how-scylla-scaled-to-one... - 1PB with 20 servers https://www.scylladb.com/presentations/operating-at-monstrous-new-scales-benchmarking-petabyte-workloads-on-scylladb/ https://www.scylladb.com/presentations/operating-at-monstrou... - Comcast moved from 970 Cassandra nodes (and 60 nodes of cache) to 78 Scylla nodes - https://www.scylladb.com/2020/01/15/comcast-sprinting-from-cassandra-to-scylla/ https://www.scylladb.com/2020/01/15/comcast-sprinting-from-c... - Palo Alto Networks run 8600 clusters(!) of ScyllaDB https://www.scylladb.com/2022/06/14/how-palo-alto-networks-replaced-kafka-with-scylladb-for-stream-processing/ https://www.scylladb.com/2022/06/14/how-palo-alto-networks-r... ScyllaDB excels in throughput and latency, we have also a better compaction algorithm that saves 37% of storage compared to C*. Usually one can replace lots of small nodes with gigantic nodes that have more resources and it allows much better management. To run 100PB Scylla will need more than 300 nodes, even thousands but definitely not what Apple throw at the problem.
- thekozmo 4y agoThere is no magic behind Scylla, mainly lots of hard work, hundreds of years of engineering, based on the former C* design which is based on Dynamo/Bigtable. The JVM is part of the problem, not all of it. The main issue is that it hides the hardware and makes tracing harder - instruction level and block level. At Scylla we strive for efficiency, every operation is tagged with a priority class for the CPU and I/O schedulers. Folks are welcome to read the blogs about those topic. Lots of details and hard work
- nazka 4y agoWhat happened to FondationDB? It is used elsewhere?
- zeristor 4y agoHow comes iTunes movies takes minutes to display a list on Apple TV, feels like minutes all the time?
- jleahy 4y agoNormally you post a question and someone posts an answer. Here someone has posted the answer and you have posted the question. Are we playing Jeopardy?
- tmpz22 4y agoOr to just say it - because the system has absurd complexity as proven by the hardware needed to run it.
- ezfe 4y agoWhile it doesn't take that long on my device, it probably is related to the fact that the iTunes store is over 20 years old and has the tech debt to prove it.
- reaperducer 4y agoHow comes iTunes movies takes minutes to display a list on Apple TV, feels like minutes all the time? I just fired up the iTunes Movies app on my AppleTV for the first time so no cache (I only watch my DVD/Blu-Ray rips), and the app started and loaded a full list of movies in a little under 3.5 seconds. If it takes you minutes, it sounds like PEBKAC.
- vonseel 4y agoIdk. Apple search APIs seem slower than competitors often; my non-tech family members have made similar comments to me about Apple Music and Apple TV+ being slower than Netflix (or Prime) / Spotify.
- reaperducer 4y agoBut the OP specified iTunes Movies, which is not AppleTV+ or Apple Music.
- rkwasny 4y agoCan be replaced with 300 servers with ScyllaDB :-)
- leetrout 4y agoBecause of no JVM? Or because of its different architecture (different caching and such)? I would expect it to still require more for high availability but from what I have heard around ScyllaDB it does seem there is a benefit to it over cassandra.
- neeh0 4y agoRelevant post: https://instagram-engineering.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 https://instagram-engineering.com/open-sourcing-a-10x-reduct...
- pclmulqdq 4y agoYou might need about 1500. I think 64 TB of flash is the standard for a 2U database server these days.
- deleted 4y ago[deleted]
- riku_iki 4y agowill ScyllaDB shrink data 1k times using some magic?
- AtlasBarfed 4y agoNo. Scylla has also stalled in support of features that Cassandra offers, which is strange because they came out of the gates like gangbusters. It is not the drop in replacement they advertise. But I'm not going to tell you "don't use scylladb". They have lots of educational whitepapers for maxing I/O and hardware that apply both to cassandra and scylladb. The biggest thing with cassandra and scylla is that you need a technical management to understand the headaches, and appreciate the silent benefits. Cassandra places often have frustrated management about the cassandra management problems, but when I ask "has your cluster ever gone down?" and they say no, even across major version upgrades and the like, expansions, etc. Also keep in mind that cassandra also has a tendency to make your big data bigger with timestamps-per-cell, replicas, etc.
- jackblemming 4y agoUnrelated but does anyone else get tired of the fetishization of BIG NUMBER? I don’t care if Facebook has billions of users if it’s a hot pile of garbage. I don’t care if some game has millions of players if it’s bad. When did BIG NUMBER overtake quality and can we go back?
- chomp 4y agoBIG NUMBER implies a dedication to the care and feeding. It’s a nudge and a wink for “come work for us” Number of users advertisement is more for investors, and maybe a broadcast for FOMO. There is, however, a tendency in our field to look for systems design patterns that handle big N numbers, and apply those to little N platforms, but I believe deep down that this is business and management dysfunction, as systems refactoring is tolerated even less than code refactoring.
- latchkey 4y ago
- redanddead 4y agoSounds fun, do tell
- eurleif 4y agoQuantity of users is social proof (more accurately, social evidence) of quality. The claim "we have high quality" is cheap: anyone can make it, and it's subjective enough that it can't be disproven in any absolute sense. But if you lie about having billions of users, you can be called out on that pretty easily; and if you say it honestly, it implies that billions of people like the quality of your product or service enough to use it. Billions of people can be wrong, but saying you have billions of users is still much better evidence of quality than saying you have quality.
- api 4y ago> Quantity of users is social proof (more accurately, social evidence) of quality. It can also be evidence of founder effects (JavaScript), lock in (Windows), network effects (most social media), etc.
- faizshah 4y agoInteresting, I thought they were trying to switch over to FoundationDB but looks like their Cassandra usage keeps growing.
- onesociety2022 4y agoApple acquired FoundationDB but from what I have heard it’s not really used much. The FoundationDB founders have left Apple and are working on other things. Cassandra is the main datastore for iCloud data.
- atombender 4y agoAre you sure? The latest SIGMOD paper on FoundationDB says it's in active use. My impression is that iCloud is built on both Cassandra and FDB, and that FDB (via Record Layer in particular) is used extensively.
- jwr 4y agoWell, since we're exchanging unsubstantiated rumors: from what I have heard it's exactly the opposite. FoundationDB is being used for transactional data and works very well. Its active development by Apple would seem to confirm that.
- exhibitapp 4y agoThese are different technologies with different use cases. FDB typically is used when there are strict ACID requirements. Apple uses both
- molszanski 4y agoI am really curious. It would make sense to use Cassandra as low logic, high volume simple storage. And then use something more relational / graph and ACID on top of it
- tpmx 4y ago300k Cassandra nodes seems a bit over the top even for a company with as many active devices as Apple. https://www.theverge.com/2022/1/28/22906071/apple-1-8-billion-active-devices-stats https://www.theverge.com/2022/1/28/22906071/apple-1-8-billio... 1.8B active devices / 300k nodes = (just) 6k devices per Cassandra node
- daniel-grigg 4y agoOr it tells us something of how much data is being scooped up per device. Certainly when I look through the raw health data collected it’s quite alarming and I’m sure that’s just a drop in the ocean.
- mwint 4y agoWhy do you think the raw health data is getting sucked off your device? That would be totally off brand for them. Apple does have a separate opt-in “Research” program to facilitate this kind of thing.
- daniel-grigg 4y agoI’m not suggesting apple is doing nefarious, it was simply an example of a large dataset. I assume they’re encrypting/scrubbing all your PII as that’s their whole motto. But unless they’re doing some voodoo magic then yes the data is leaving is your device in some form. Hence why I can view every heartbeat (aggregated by the minute) since I put on the original Apple Watch in 2015 despite changing all my devices and only restoring from iCloud. Indeed I expect it’s just part of your iCloud data storage. Actually I just launched the health app now and I can see the app explicitly asks if you want to allow sharing your data with apple, so if you say ‘yes’ then you’re not only storing but allowing apple to query your data (minus PII).
- smoldesu 4y agoIt's also off-brand for Apple to join PRISM and comply with thousands of annual requests for supposedly-inaccessible iCloud data. Neither of you will ever be proven right until we look inside those servers though, so making any conclusive statements is a mistake. Apple designed Schrodinger's datacenter.
- bluedino 4y agoOver 2PB per cluster, thousands of clusters, but only 100's of PB of data. What do they use this for? iCloud storage related stuff?
- akho 4y ago100’s of PB seems low for iCloud. iMessage, probably.
- pmcf 4y agoOn the slide shown, it says "1000s of applications."
- xvector 4y agoThey use it for storing iCloud Photos without E2EE while heavily marketing privacy
- cassonmars 4y agoThey were moving towards E2EE when everyone freaked out about the on-device perceptual hashing trade off.
- smoldesu 4y agoThey should have wheeled out a better marketing spiel than "trust us ;)" then.
- MBCook 4y agoYup. The knee-jerk privacy reaction cost us privacy.
- Gigachad 4y agoI don’t think it’s fair to say we need to accept either options. Yes the crime they are trying to stop is horrific and something must be done, but that doesn’t justify unlimited technological spyware. And the scope for abuse is so large. People in the UK are getting arrested for retweeting mean memes, it’s pretty easy to imagine Google and Apple added offensive images to their scanning and you get arrested for saving something that goes against the current political agenda. As well as the case where google locked the account of a parent who had taken photos to send to a medical expert.
- api 4y agoI keep hearing people talk about how hard Cassandra is to run properly and how many people get it wrong etc. Is there anything to this or is it just FUD and people who genuinely don’t know what they are doing?
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- jhgg 4y agoOur biggest issue with running Cassandra was related to pathological read / write patterns by some tenants on our system causing outsized availability impact due to triggering garbage collection pressure that would cause whole node GC STW pauses and severe tail latency / query degradation. We have solved these issues in a few ways, mainly: - working with the relevant product teams to implement appropriate rate limiting or improving data modeling. - introducing our own query layer, written in Rust that sits in front of Cassandra that uses a form of micro-caching called read coalescing, and also other forms of query throttling/load shedding to reduce work the database must do for hot keys/pathological patterns of access. We expose a GRPC interface from this - and this lets us centralize control of the client driver and tune it appropriately, while also getting to leverage the ever growing open source grpc traffic routing solutions (envoy, etc...) and ultimately, - switching to ScyllaDB, a C++ rewrite of Cassandra which is of course void of any garbage collection issues, and features faster overall performance and lower latencies. Scylla, however, is not without its own set of issues - and somewhat strict hardware requirements[0] thanks to the seastar engine it is built on top of. Their team however has been delightful to work with, and our platform is markedly more stable in current year than it was in years past thanks to the above factors. Operationally, however, Scylla and Cassandra are quite easy to run, the trickiest part is repairs. Common operations such as cluster expansion, or replacement of node are so common an operation that they are at this point mundane. Be wary however about read/write amplification issues inherent to LSMT databases, choosing the correct compaction strategy and tuning it appropriately can be quite key. Additionally tombstones can be quite bad for performance. In current day we offer a new more generic solution that sits on top of scylla (it would work with Cassandra too) that provides a simple interface to query KKV based data, without having to worry too much about problems like large partitions, hot keys, or tombstones! With a design like this, the underlying cluster thus far has been issue free and very easy to operate. [0]: https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency https://discord.com/blog/how-discord-supercharges-network-di...
- notacoward 4y agoThe operational complexity of managing thousands of clusters must be mind boggling. I've been on two projects managing dozens of storage clusters, the second with more data than this in some individual clusters and adding up almost this many total nodes. There were technical problems with scaling up to 10K nodes per cluster, but the operational issues mostly scaled according to number of clusters. For example, how many alerts per hour/day can you stand? Too many and you're overwhelmed; too few and you miss stuff. Walking that fine line became successively more difficult as clusters were added. Same thing with graphs and dashboards. Also, when your storage and IOPS are siloed this much you have no elasticity, so you're going to be chasing capacity or load problems much more often. On the plus side, this probably means each tenant has their own cluster, so you don't have so many worries about them affecting each other. The big question I'd have is: how many people (including on client teams) does it take to manage this much sprawl?
- sdevoid 4y agoI'm going to guess somewhere between 50 and 400 people. Smaller than 50: this is probably an org where "we maintain the source code and documentation. Running, debugging, and scaling your cluster is YOUR problem". Larger than 400: the org probably fulfills other functions besides "running thousands of production Cassandra clusters." This is a lot, and is one of the big things that surprised me when I first joined a large organization. But here's an example breakdown in no particular order: - 10 to 30 people for managing, contributing, and maintaining the open source aspect. Just a handful of engineers could contribute features and fixes to the project, but once the open source project and community gets larger it becomes a full time job. System component maintainers, foundation boards, committees, conferences, etc. add up. - 10 to 70 people for "operations". As you mentioned, the load here tends to scale with the number of clusters (customers). At the large end this is several teams, with say a team dedicated to (a) fleet-management of the individual machines in a cluster, (b) cluster lifecycle management, and (c) macro level operations above the cluster level. Alerts can't all go to this team, so some of the work is writing alerts that go to customers in a self-serve model. - 10 - 40 people for "scale projects." At this scale you have 1 - 10 customers that are on the verge of toppling the system over. They've grown and hit various system bottlenecks that need to be addressed. And you'd be lucky if they're all hitting the same bottleneck. With this many customers, it's likely that they've all adopted orthogonal anti-patterns that you need to find a fix for: too many rows, too many columns, bad query optimization, too many schema changes, too many cluster creates and destroys, etc. So you probably have multiple projects ongoing for these. - 10 - 30 people for "testing infrastructure". Everyone writes unit tests, but once you get to integration and scale testing, you need a team that writes and maintains fixtures to spin up a small cluster for some tests and a large cluster for scale tests (which your "scale projects" teams need, btw). And your customers probably need ways of getting access to small test Cassandra clusters (or mocks of the same) for THEIR integration and scale tests, since Cassandra is just a small part of their system. - 10 - 30 people for automating resource scaling and handling cost attribution. These may not be one function, but I'm lumping them together. "Operations" might handle some of the resource scale problems, but at some point it's probably worth a team to continually look for ways to manage the multi-dimensional scaling problem that large cluster software systems inevitably create. (Is it better to have few large nodes, or many small nodes?) You need some way of attributing cost back to customer organizations, otherwise you're paying $50M because one engineer on the weather team forgot to tear down a test cluster in one automated test 6 months ago and... You need to make sure that growth projections for customers are defined and tracked so you have enough machines on hand. - I'll add that it'll be worth adding whole teams for some of the more complex internal bits of this system, even if the actual rate of change in that sub-system is not very high. At this scale organizations need to optimize for stability, not efficiency. You don't want to be in the situation where the only person who understands the FizzBuzz system leaves and now dozens of people/projects are blocked because nobody understands how to safely integrate changes into FizzBuzz. - Things not covered: security, auditing, internal documentation, machine provisioning, datacenter operations, operating system maintenance, firmware maintenance, new hardware qualification, etc. Maybe there's an entire organization dedicated to each of these, in which case you get it for free. If not, some of your time needs to be spent on these. (Even "free" might have a cost as you need to integrate with those services and update when those services change.)
- rektide 4y agoI'd love info on how much Apple contributed to Cassandra!
- _benedict 4y agoI’m sure Scott’s talk went into detail about this, but I can safely say that his team contributes a great deal to Cassandra
- brianwawok 4y agoWell, when I worked on Cassandra at datastax, they hired away a lot of my coworkers to work for Apple.
- redanddead 4y agoI'm under the belief that Redis is much faster than Cassandra, am I crazy to think that Apple or any company really should have a transition plan? Why isn't redis used more?
- mplewis 4y agoCassandra solves different problems than Redis is typically used for.
- simonw 4y agoRedis is for when your data fits in memory.
- nahname 4y agoRedis is a cache
- bufferoverflow 4y agoRedis is not a cache. It has all kinds of data structures. https://redis.com/redis-enterprise/data-structures/ https://redis.com/redis-enterprise/data-structures/
- gscho 4y agoRedis is not simply a cache. It can be used as one but it supports many other data structures and use cases beyond that.
- jitl 4y agoBy default redis will drop your data when under memory pressure. It’s a different thing.
- iampims 4y agoI’d be curious to know what the orchestration platform looks like for running 300,000 nodes and 1,000s of clusters.
- geerlingguy 4y agoI know some of their backend services use a lot of Ansible (for at least the automation / config portion), but I don't know which ones. Apple's devs are pretty mum on the specific stacks they're using, usually.
- Luker88 4y agoCouple of things as always: Cassandra works really bad with fat nodes (lots of data on one node), and much much better with a lot of small nodes, and 100PB with 300K nodes confirms this. Scylla scales better vertically, but don't know how much. Some comments are already comparing this to pgsql/mysql/whatever. Please don't. You can't make the same queries even though the language seems to support it. Cassandra is good at ingesting data, bad at deleting, really really bad at anything remotely relational. Errors are almost pointless. I'm going to point at an older comment of mine on cassandra: https://news.ycombinator.com/item?id=20430925#20432564 https://news.ycombinator.com/item?id=20430925#20432564 The takeaway should be: Yes, cassandra/scylla can be really fast and scale a lot. But it is also very probably unusable for your use case. Don't trust what the CQL language says you can do. Don't get me started on how bad the CQL language is, either.
- raverbashing 4y ago> older comment of mine on cassandra: https://news.ycombinator.com/item?id=20430925#20432564 https://news.ycombinator.com/item?id=20430925#20432564 Wow, that's surprisingly bad For real, I'm not against java, but I find time and again that those services that depend on the JVM have lots of "weird stuff". Not because Java is bad but Java fans (which you usually are when you start to develop something this big) think a lot of this crap is acceptable.
- pmcf 4y agoIn my (extensive) experience in infrastructure. When people say the JVM is the problem, the JVM is never the problem. It's usually just a symptom of something else and lazy ops people just want to blame something and throw up their hands. I've never had to "tune a JVM" to make things work. I'll give you an example in Spark. We had a huge job that was failing and after checking the logs, it was when results were being spooled to disk. More log diving showed a lot of GC on every node. At that point we could have gone down the route of tuning something in the JVM, but more digging found the real culprit. IOstats when the jobs ran showed while reading data, writes were completely blocked and write latency was in the 100s of ms. The spark executor trying to dump data was blocked and the first symptom that things were falling apart was... GC. We changed the scheduler on the nodes and magically everything worked great. The VM in JVM is virtual machine. Same rules for resources apply and if you run out of resources, don't expect the mythical ops faeries to save you.
- iamgopal 4y agoCost apart, can firestore ( firebase ) can handle this much data ?
- rrdharan 4y agoYes.
- aprdm 4y agoWhat use cases are good for Cassandra vs Postgres or Mongo or Elastic ? I've used the latter 3 a lot, never used Cassandra.
- jjirsa 4y agoCassandra is very good when all of these are true: - You're willing to give up some common SQL-isms for leaderless, multi-dc, high-availability - You're going to grow to need more than one machine taking writes - if you can fit on one normal commodity machine, just use something designed for one machine. Cassandra sacrifices a lot to scale to ~thousand-machines-per-cluster, so if you can use one machine, don't bother with Cassandra. - You can model your data in a way that does SELECTs without JOINs, and always uses AT LEAST the partition key in the WHERE clause. Denormalizing and duplicating your data is PROBABLY ok. - You're willing to run Java (so you understand that you may need to tune the JVM eventually), and you're willing to learn about data modeling before you just start writing code If all of that is true, Cassandra starts being interesting.
- IOT_Apprentice 4y agoI thought Apple bought a company that did Cassandra like functionality and started using that?