8 ms·
Keeping Instagram up with over a million new users in twelve hours
- nodesocket 15y agoGreat stuff, love node2dm, and didn't know about statsd + graphite.
- mikeyk 15y agoThanks! This was my first node.js project, would love any feedback on it.
- nodesocket 15y agoAlso, any reason you guys are not using MixPanel for storing events as well, besides of the costs?
- Sujan 15y agoThought about adding a tool like newrelic.com to your toolset?
- mikeyk 15y agoNewRelic's product screenshots look really nice, but 1) the hosted nature and 2) the price were turnoffs. We might revisit it at some point, though--there are a few features in there that would be awesome to have.
- Sujan 15y agoHad the same first impression, but compared to rolling your own solution it's still a very good ROI. And now I'm getting more and more servers and projects on it, because it's just so handy. (Btw not connected in any way, jsut a very happy user that convinced enough companies to try them, that I think you would be a good match, too)
- benologist 15y agoI think New Relic saves us money - the amount of optimizations we've done because of them is just crazy and by now would have saved us a couple servers and hardware upgrades at least. Their sales dudes like startups, we were only lightly funded when we started using them - garth@newrelic could probably help you.
- sciurus 15y agoA slight tangent, since I saw that instagram are using both Graphite and Munin- Collectd just added a plugin to send metrics to Graphite. You might want to try it for tracking your machine stats over time. http://collectd.org/wiki/index.php/Plugin:Write_Graphite http://collectd.org/wiki/index.php/Plugin:Write_Graphite http://collectd.org/ http://collectd.org/
- ab 15y agoAlong the same lines, are you doing anything special with munin to make it fast? We've had performance issues with the RRDs and graph generation that led us to pipe metrics to graphite with collectd.
- mikeyk 15y agoWe've had to split munin across three masters (by machine role) because the graphing job was just locking on IO. Munin 2.0 moved over to all-dynamic CGI graphing, but I haven't gotten the chance to play with it yet.
- jcastro 15y agoWhat OS are you deploying on EC2?
- mikeyk 15y agoWe use Ubuntu, running Natty Narwhal. Every Ubuntu version before that had some (unique) bugs that would bite us under load. Natty has been by far the most stable.
- zupreme 15y agoThanks for OpenSourcing Node2dm. I think I'll take that for a spin this weekend.
- EAMiller 15y agoWhat sort of hosting do you use for your main Pg (and Redis) instances?
- simonw 15y agoFrom the mentions of EBS snapshots in the article, I'm going to guess they're all on Amazon EC2.
- itsprofitbaron 15y agoThey use Amazon EC2 there's some more info here: http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value-pairs http://instagram-engineering.tumblr.com/post/12202313862/sto...
- terhechte 15y agoCongratulations. Really impressive how solid you handled the Android onslaught.
- kunalmodi 15y agoare you guys sharding redis? or does it all fit in a single machine?
- mikeyk 15y agoWe have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just single-node installs.
- cdelsolar 15y agoHow do you handle write replication? I haven't found any good document on how to do this (meaning, if the Redis master goes down, a slave should be promoted to master immediately).
- mikeyk 15y agoYou can hypothetically use something like hearbeatd to do it; we run every Redis master with an attached slave and manually failover for now. For a small team like ours, we prefer solutions that are easy to reason about and get back into a healthy state (it would take one server deploy to point all appservers at new Redis master), rather than fully automated failover and the "fun" split-brain issues that ensue. Of course that may change as we build out our Ops team, etc.
- bond 15y agoDoes anyone has some info on the architecture required to maintain a service like this? Servers, db, etc?
- peterwwillis 15y agoWhy use Graphite instead of Ganglia? Ganglia uses RRDs. It's been around forever, it's fairly low on resource use, it's fast, and you can generate custom graphs like with Graphite. I actually ended up doing some graphs with google charts and ganglia last time I messed with it. (Also, nobody has really simple tools to tell you which of your 3,000 cluster nodes has red flags in real time and spit them into a fire-fighting irc channel so we had to write those ourselves in python) "Takeaway: if read capacity is likely to be a concern, bringing up read-slaves ahead of time and getting them in rotation is ideal" Sorry but this is not 'ideal', this is Capacity Planning 101. If you're launching a new product which you expect to be very popular, take your peak traffic and double or quadruple it and build out infrastructure to handle it ahead of time. I thought this was the whole point of the "cloud"? Add a metric shit-ton of resources for a planned peak and dial it down after.
- njharman 15y agoCloud is also (more so hopefully) about, dynamically ramping up based on actual usage. Vs making guesses about future capacity needs. Cloud is to capacity planning as agile is to waterfall. Re RRD, have you read about graphite? "Graphite originally did use RRD for storage until fundamental limitations arose that required a new storage engine."
- peterwwillis 15y agoDynamically ramping is nice until your site explodes and you need 20 minutes to get more capacity. Versus just pre-allocating it and not going down. Call me crazy, some people don't like to be down for 20 minutes. Hmm, didn't know that. Too bad they didn't just extend RRD. Did they say what the limitations were? I see a note about high volume causing lots of writes and implementing caching to deal with it, but that can be dealt with via tuned filesystem parameters... Ah, I found the page: http://graphite.wikidot.com/whisper http://graphite.wikidot.com/whisper RRD can be tuned to ignore 'irregular' data points, or include them all. The timestamp issue can be a problem but there are methods to deal with order of updates (like take them via tcp, or rrd merge tools). If you have a lot of RAM to spare, an excellent hack is putting your RRDs with the highest amount of writes in a tmpfs volume and rsync'ing them regularly (it's insanely fast, trust me). More on tuning RRD: http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html http://sourceforge.net/apps/trac/ganglia/wiki/rrdcached_integration http://sourceforge.net/apps/trac/ganglia/wiki/rrdcached_inte... http://community.zenoss.org/docs/DOC-4696 http://community.zenoss.org/docs/DOC-4696
- lenn0x 15y agoWhat kind of instances are you guys running for Redis/memcached? I am a bit surprised on the numbers here, but to be fair I don't do much in the virtualization world. With low cpu overhead, it sounds like you might be saturating the number of interrupts on the network card if it's not a bandwidth issue. Memcache can usually push 100-300k/s on an 8-core Westmere (could go higher if you removed the big lock). Redis on the other hand with pinned processes to each physical core can do about 500,000/s. We (Twitter) saw saturation around 100,000~ on CPU0, what tipped us off was ksoftirq spinning at 100%. If you have a modern server and network card, just pin each IRQ for every TX/RX queue to an individual physical core.
- mikeyk 15y agoThose are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
- lenn0x 15y agoTaskset is fine for the process pinning. Don't forget about hyperthreading, you want to try to keep each thread on each hardware thread. IRQ pinning, see an example script I have: http://chrisgoffinet.com/pin_network.sh http://chrisgoffinet.com/pin_network.sh This will set queues 0-7 to specific smp affinity slots.
- rkurian 15y agoWhy do you guys use both memcache and Redis ? Redis also has LRU cache functionality.
- lenn0x 15y agoBecause in-house we have a custom version of memcache. We rewrote memcache's slab allocator, and for some use cases, is better at memory efficiency than Redis.
- statictype 15y agoIsn't there a risk with EBS snapshots that the snapshot of a live instance could have been taken while your db engine was in the middle of a transaction and leave the data in the newly spun instance in an inconsistent state? Is it that EBS snapshots are engineered to prevent this? Or just that it's not likely to happen in practice?
- mikeyk 15y agoYes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.
- nupark2 15y agoAre EBS snapshots not block-level atomic? In theory you should get a PITR image without stopping anything, assuming that: 1) The file system correctly orders or journals operations (I'm not familiar with XFS, but this is the case with FFS2/FreeBSD, ZFS, ext3/4 journaling, etc). 2) The database system correctly orders or journals operations, and properly fsync(s) to disk (which postgreSQL does) Of course, there's no harm to an abundance of caution with something like this.
- mikeyk 15y agoThey are, but we software-RAID our EBS drives to get better write throughput, and we put the Write-Ahead Logs (WALs) on a different RAID from the main database, so when you have both of those going on, you need something else to atomically snapshot our PG databases.
- 8ig8 15y ago> We use the counters to track everything from number of signups per second. Per second... It must be quite a moment when you reach this point.
- canop_fr 15y agoTo put it in perspective : there are less than 5 human births per seconds. So if they want to keep their counter greater than that for a long time, they'll probably have to extend their market beyond humans.
- drivebyacct2 15y agoWhat percentage of processing power is spent on making me look like a hipster?
- mkjones 15y agoPretty sure that all happens on the client (your phone).
- jurre 15y agoVery interesting read, but doesn't New Relic do all these things for you? Maybe it's not possible to use with their setup?
- camwest 15y agoI'm interested in comparing statsd to a commercial product like New Relic as well. -C
- Sujan 15y agoStatsd and NewRelic are very different. NewRelic gives you mainly a predefined set of metrics, where you just have to install the agent to get them. Then. There's an additional module where you can send your own set of metrics and display them. Statsd on the contrary is 'only' a tool to collect and then display metrics. You have to define everything you eant to measure yourself (or use plugins to your app). So these two are definitely related, but better used for different (although overlapping) jobs.
- gflarity 15y agoWe use statsd, graphite, redis and node as well. You might be interested some of my projects relating to these: https://github.com/gflarity/nervous https://github.com/gflarity/nervous https://github.com/gflarity/response https://github.com/gflarity/response https://github.com/gflarity/qdis https://github.com/gflarity/qdis
- ganilb 15y agoI am curious to find out why there was a need to develop your own C2DM server - what was lacking in Google's C2DM server? I am a C2DM newbie so pardon my ignorance.
- rkurian 15y agoIt looks like you guys use Redis for a lot of different functionality. It would be great to see an article on how you guys use Redis.
- nboutelier 15y agoIm curious to know what kind of EC2 instance they are running the master Postgresql on and if they've had any write bottle necks. Im using Postgres for an app, and am worried about running into write issues.
- olegi 15y agoHello! Question about quality insta-photos on Android. I have JPG from SGS2 - http://kia4sale.narod.ru/insta/01.jpg http://kia4sale.narod.ru/insta/01.jpg This is http://kia4sale.narod.ru/insta/02.jpg http://kia4sale.narod.ru/insta/02.jpg instaphoto (Earlybird) from Android version This is http://distilleryimage9.instagram.com/662ade7483ce11e19e4a12313813ffc0_7.jpg http://distilleryimage9.instagram.com/662ade7483ce11e19e4a12... - instaphoto from SGS2 JPG but on iPhone 4. Question: why instaphoto on Android version in blurry? Thanks.
- andrewdunstan 15y agoPGFouine is nice, but it needs a major do-over. It would be good written with a plpgsql backend running against database loaded csv log files, so that it could handle huge logs, unlike now.