3 ms·

Just went through the slides... watching now. Very interesting. One thing that is a bit surprising is the diversity of key-value/caching technologies. I felt li

by po 15y ago

Just went through the slides... watching now. Very interesting. One thing that is a bit surprising is the diversity of key-value/caching technologies. I felt like there was a lot of overlap in capability between Varnish, redis, memcached, mongodb, etc...

simonw 15y ago
I'm a big fan of polyglot persistence - we're using MySQL, memcached, redis, Solr, Varnish and MongoDB now. MySQL is where all of our key data lives up. I trust it, it's backed up, and the entire site can be recreated just from the MySQL dump. Redis and Solr are both used for denormalisation. Solr provides search and our core calendar view, and is updated every 60-90 seconds by a cron job. It's replicated, which means that our calendar view (the most expensive page on the site) scales horizontally with the number of replicas. Redis powers a few features, most notably pages that show which of your Twitter contacts are attending an event (a simple Redis set intersection, which Redis will happily perform 100,000 times a second). It's also used for our message queue, which means I don't have to run RabbitMQ as well. memcached is used for caching. I could use Redis for this, but the nice thing about memcached is that it has a hard memory limit and will throw away keys without any fuss when it hits that limit. It's also a good idea to keep resources set aside for caching separate from resources being used for other purposes, in my opinion. Varnish is currently just used as a layer in front of our JavaScript badges ( http://lanyrd.com/services/badges/ http://lanyrd.com/services/badges/ ), purely to protect us against a super high traffic site deploying our badges (all badge requests are cached for 10 minutes, and Varnish handles dogpiling for us). I'd like to use Varnish in front of the main site as well just for logged out users, but I haven't had time to deploy that yet. Our badges are also designed to not block the loading of your site if we're down for some reason - varnish helps a bit there as well. See our badge performance notes here: http://lanyrd.com/services/badges/docs/#performance http://lanyrd.com/services/badges/docs/#performance We recently started storing application logs in MongoDB, mainly as an experiment. MongoDB is very fast at writes, and lets us easily run structured queries across our logs. I don't care too much about persistence here, since the data isn't as valuable as e.g. our core database of conferences.
- Swannie 15y ago
  So it this a question of: Learning new tools and understanding their idiosyncrasies vs. building new functionality on top of existing tools? Where learning new tools is more interesting, so it wins? :P

by po 15y ago

simonw 15y ago

I'm a big fan of polyglot persistence - we're using MySQL, memcached, redis, Solr, Varnish and MongoDB now. MySQL is where all of our key data lives up. I trust it, it's backed up, and the entire site can be recreated just from the MySQL dump. Redis and Solr are both used for denormalisation. Solr provides search and our core calendar view, and is updated every 60-90 seconds by a cron job. It's replicated, which means that our calendar view (the most expensive page on the site) scales horizontally with the number of replicas. Redis powers a few features, most notably pages that show which of your Twitter contacts are attending an event (a simple Redis set intersection, which Redis will happily perform 100,000 times a second). It's also used for our message queue, which means I don't have to run RabbitMQ as well. memcached is used for caching. I could use Redis for this, but the nice thing about memcached is that it has a hard memory limit and will throw away keys without any fuss when it hits that limit. It's also a good idea to keep resources set aside for caching separate from resources being used for other purposes, in my opinion. Varnish is currently just used as a layer in front of our JavaScript badges ( http://lanyrd.com/services/badges/ http://lanyrd.com/services/badges/ ), purely to protect us against a super high traffic site deploying our badges (all badge requests are cached for 10 minutes, and Varnish handles dogpiling for us). I'd like to use Varnish in front of the main site as well just for logged out users, but I haven't had time to deploy that yet. Our badges are also designed to not block the loading of your site if we're down for some reason - varnish helps a bit there as well. See our badge performance notes here: http://lanyrd.com/services/badges/docs/#performance http://lanyrd.com/services/badges/docs/#performance We recently started storing application logs in MongoDB, mainly as an experiment. MongoDB is very fast at writes, and lets us easily run structured queries across our logs. I don't care too much about persistence here, since the data isn't as valuable as e.g. our core database of conferences.

Swannie 15y ago
So it this a question of: Learning new tools and understanding their idiosyncrasies vs. building new functionality on top of existing tools? Where learning new tools is more interesting, so it wins? :P