mikeyk

10 ms·

61.
by mikeyk 15y ago
They are, but we software-RAID our EBS drives to get better write throughput, and we put the Write-Ahead Logs (WALs) on a different RAID from the main database, so when you have both of those going on, you need something else to atomicall
62.
by mikeyk 15y ago
Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
63.
by mikeyk 15y ago
Yes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.
64.
by mikeyk 15y ago
To clarify: we did do a fair amount of capacity planning and elastic ramp-up/pre-allocating of our infrastructure, but no prediction is perfect, so the blog was about diagnosing and addressing issues as they crop up under unprecedented load
65.
by mikeyk 15y ago
We have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just
66.
by mikeyk 15y ago
We use Ubuntu, running Natty Narwhal. Every Ubuntu version before that had some (unique) bugs that would bite us under load. Natty has been by far the most stable.
67.
by mikeyk 15y ago
NewRelic's product screenshots look really nice, but 1) the hosted nature and 2) the price were turnoffs. We might revisit it at some point, though--there are a few features in there that would be awesome to have.
68.
by mikeyk 15y ago
Thanks! This was my first node.js project, would love any feedback on it.
69.
Keeping Instagram up with over a million new users in twelve hours(instagram-engineering.tumblr.com)
279 points by mikeyk 15y ago|53 comments
70.
Pandolist Part One: iOS Magicians(pandodaily.com)
1 points by mikeyk 15y ago|0 comments
71.
by mikeyk 15y ago
Hey, author here. To clarify, our uploads go to our servers first (where we resize for thumbnails, etc) then go to S3.
72.
Instagram Engineering Challenge: The Unshredder(instagram-engineering.tumblr.com)
156 points by mikeyk 15y ago|66 comments
73.
by mikeyk 15y ago
Yep, ~30 inserts per second go in, so it's not static.
74.
by mikeyk 15y ago
Honestly, it was mostly the concision that struck me at the time. Looking at the zipmap source now, I think it's more or less where my code tends to end up in terms of comments--some motivation / higher-level comments at the top, and most f
75.
by mikeyk 15y ago
Back when we were getting started with Redis, the readability / concise nature of the project was one of the things that most excited me about it (here's what it looked like around then: https://github.com/antirez/redis/tree/0b420168b485d0
76.
Storing hundreds of millions of simple key-value pairs in Redis(instagram-engineering.tumblr.com)
153 points by mikeyk 15y ago|54 comments
77.
by mikeyk 15y ago
We run several dozen instances, so managing that list manually became cumbersome. Being able to create an instance, and moments later have that instance easily SSH-able by anyone on the team without requiring an additional commit & pull
78.
Simplifying EC2 SSH connections(instagram-engineering.tumblr.com)
23 points by mikeyk 15y ago|5 comments
79.
by mikeyk 15y ago
We love Redis at Instagram (it powers a lot of our systems...more write-ups on these soon), and considered it for our ID generation, but it would have introduced a single point of failure, unless we split the load between several Redis inst
80.
by mikeyk 15y ago
Hey Ken, (author here) There's a 1:1 mapping between the user's hash modulo the number of shards, and the table they're writing to. So, if we had 1000 logical shards, we have 1000 schema/tablespaces, with the same tables in each. And the da
81.
by mikeyk 15y ago
I used to work at Meebo, which hosts a large CouchDB system , but when it came time to choose a solution for Instagram we went with PGSQL. There was a lot I really liked about Couch, especially for the analytics system we built at Meebo (wh
82.
by mikeyk 15y ago
Thanks for the info, I updated the post to mention that the write bottleneck isn't an issue at Flickr.
83.
by mikeyk 15y ago
Valid questions! We're on EC2, which has its set of limitations but means we can run a 10 million + user system with two-and-a-half engineers (and no ops team / overhead). So while we hear about more and more folks using SSDs in their DBs,
84.
by mikeyk 15y ago
Yep--definitely something to watch out for. At worst, though, you'd have a duplicate key when trying to insert, and can re-try without the risk of having a duplicate ID floating around your system.
85.
by mikeyk 15y ago
We considered the composite key, but we often have to store keys in other systems like Redis, where having a single 64-bit integer makes it more portable and stored compactly (Redis, for example, has an optimization when storing integer val
86.
by mikeyk 15y ago
We set-up the schemas ahead of time, and then each lookup is (with, say, 1000 schemas): user_id % 1000 -> schema ID schema ID -> database ID then select FROM schemaID.tablename etc on that particular database.
87.
by mikeyk 15y ago
Thanks for the comment! We don't need the IDs to be exactly sortable, only roughly sortable within a second or so. As long as the clock doesn't move backwards on any given machine (we use ntpd in its gradual-adjustment mode), the IDs are un
88.
by mikeyk 15y ago
Right now, it's a lookup dict in our Django app--which involves brief downtime just to update the shard map when moving the data (more on this in another post). We hash on user ID (in most cases) and then look the shard # in the dict, then
89.
by mikeyk 15y ago
(author here) I think this should also be possible with MySQL's stored functions, but one huge benefit to PostgreSQL is the schema/tablespace feature, since it means all our logical shards all live inside one database (you could do somethin
90.
by mikeyk 15y ago
Pgfouine is awesome, but one thing that's appealing about this one is the traceback showing where the query occurs--not always obvious given the Django ORM.

More ›