5 ms·
At Twitter we're working on using Cassandra to store tweets
- technoweenie 17y agoEvan Weaver blogged about Cassandra (http://blog.evanweaver.com/articles/2009/07/06/up-and-running-with-cassandra/ http://blog.evanweaver.com/articles/2009/07/06/up-and-runnin...) awhile ago and used Twitter as a sample schema to show how it worked.
- alexro 17y agoFrom that post: "Don't use EC2 or friends except for testing; the virtualized I/O is too slow" kind of restricts Cassandra usage for an average Web 2.0 startup Added: just to clarify that: when starting small the last thing you want to do is either buy expensive infrastructure or hack something cheap instead of doing it properly. If Cassandra won't perform well in the virtualized environment, it will be necessary to first use some other means of storing data and "go cassandra" later if needed.
- antirez 17y agoBut everything resembling a DB is slow on EC2 since I/O is slow and memory accesses are slow, so I think the problem is EC2 and not Cassandra, MySQL, ...
- alexro 17y agoHence the hopes it won't be much more slower
- jbellis 17y agoto clarify, Cassandra is still faster on a per-node basis than relational databases on EC2, but if you compare EC2 numbers with real hardware and wonder "why is it so slow?" that is why. :) that said, there was a thread yesterday about Joe Stump's benchmarks showing EC2 I/O not being as bad as most people think so possibly they have made some improvements since. As j3fft mentions, Joe Stump is also in TFA talking about using Cassandra.
- j3fft 17y agoif you look on the OP's thread, Joe Stump talks about using Cassandra for SimpleGeo, which I believe is completely hosted on EC2.
- antirez 17y agoI read: "We evaluated a lot of things: a custom mysql impl, voldemort, hbase, mongodb, memcachdb, hypertable, and others" I wonder if among the others there is Redis too, if so (and somebody at twitter is listening) I could love to know what do you think are the biggest weakness in Redis and if you think in the short term it's more important a fault-tolerant cluster implementation (the redis-cluster project) or virtual memory to support datasets bigger than RAM into a single instance. Thanks in advance for any reply.
- lsb 17y agoI'm not at Twitter, but what's the tuning parameter to specify that every x seconds data goes to disk?
- antirez 17y agohello, it's documented in the default redis.conf comments, the parameter is called "save": http://github.com/antirez/redis/raw/master/redis.conf http://github.com/antirez/redis/raw/master/redis.conf
- siculars 17y agoi would probably say the large memory footprint of redis , especially at 64bit vs 32bit.
- jcapote 17y agoRedis requires you have enough RAM to hold your entire dataset in memory...which seems impossible for something as large as twitter
- briansmith 17y agoDoes Redis really require the entire dataset to be in RAM, or does it just require enough virtual memory to hold the dataset? In other words, couldn't you just let it swap?
- notaddicted 17y ago
- chanux 17y agoDigg was considering Cassandra too http://blog.digg.com/?p=966 http://blog.digg.com/?p=966
- uggedal 17y agoDigg is in process to convert all data to Cassandra (mentioned in their NoSQL East presentation).
- anApple 17y agoI love how twitter is constantly trying to reinvent the wheel for a simple message passing service.
- mullr 17y agoSurely a very large message passing service has different requirements than your typical system.
- diego 17y agoA highly-scalable, multicast, real-time, distributed message passing system with full-text search, archiving, deletions, dynamic subscription changes, access restrictions, open APIs with quotas, abuse detection mechanisms, spam fighting algorithms, different authentication protocols, etc. Still think it's so simple?
- majke 17y agohighly-scalable - now:maybe, year ago:no multicast - are there still stupid limits on max number of subscriptions? real-time - sort of. have you measured the latency? depends what you mean by realtime. distributed message passing system - distributed:yes, message passing:it depends what you mean. is polling considered as messaging? archiving - what? deletions - doesn't remove the items from search. I consider it as not-working. dynamic subscription changes - yes. access restrictions - sort of. open API - yes ...
- evgen 17y agoOkay, I'll take the bait... > highly-scalable Everything I take from our set of OTS components will have live, verifiable examples of running at scale. > multicast, real-time, distributed message passing system For this the choice really comes down to RabbitMQ or ejabberd. An XMPP solution is appealing for the obvious benefits of having a "presence" concept, but an AMQP solution keeps us closer to the current reality of Twitter. So we start with a large distributed rabbit setup. A few clusters scattered across the world, connected via shovel pipes and using some nested queue/exchange plumbing to wire it all together. > full-text search Some queues dump to a FTS setup that keeps recent messages in RAM and migrates them to disk as they get older. SOLR is probably a better solution here, but I know the Sphinx delta/full-index model better and would reach for that first. > archiving Dump a firehose queue to disk and send copies/updates out to various services that want it. Pushing it to HDFS would probably be the first choice I would look at, just because it is easy to go from there to various Hadoop analytics. > deletions Nothing is ever deleted, it just becomes invisible. This is just a flag to add to the FTS indexes and archives. > dynamic subscription changes, access restriction Handled completely by the message queues. > open APIs with quotas, abuse detection mechanisms, spam fighting algorithms, different authentication protocols, etc. None of these is particularly difficult once you have the general framework setup, and the structure of the system actually makes it pretty easy to wire in things like realtime abuse and spam prevention once you get rolling. There is only one thing hard about building a better Twitter, getting the userbase to make it worthwhile. If Twitter was in a different market where the network effect was not so strongly self-reinforcing then it would have been cloned and re-implemented better back when the fail whale was our constant companion, but this is not the case so making a "better" twitter is of little value.
- rbranson 17y agoGlad to hear they decided against writing their own in Scala.