10 ms·

Reddit: Lessons Learned From Scaling To 1 Billion Pageviews A Month

229 points by jpmc 13y ago

srj55 13y ago
hmm...no love for django here.
- deleted 13y ago
  [deleted]
- falcolas 13y ago
  I'm currently using Django, and I can understand where they are coming from. The sheer amount of code that a simple request has to go through to receive an answer is staggering sometimes, such as 18 line tracebacks just to identify an authentication problem... My own projects are not yet large enough to have this cause an issue, but I can see where something the size of Reddit would indeed have issues that even the most aggressive caching can't resolve.
- film42 13y ago
  It's not a matter of rating frameworks, it's a matter of purpose. Django and rails are designed to go from nothing to product as quickly as possible. Plus, thanks to better hardware and scaling techniques, it's easy and affordable to stick to frameworks longterm (instagram and github, for example). Sure, they could they rewrite their site in C++, Go or Erlang, but they would lose their ability to rapid prototype new features.
human_error 13y ago
> Used the Pylons (Django was too slow), a Python based framework, from the start This isn't quite right. It was web.py at the beginning. They have started using Pylons after Conde Nast acquisition.
- showerst 13y ago
  Originally it was actually written in LISP. =) http://blog.reddit.com/2005/12/on-lisp.html http://blog.reddit.com/2005/12/on-lisp.html
- jedberg 13y ago
  Yes, I glossed over that part. We didn't use web.py very long.
  - dmead 13y ago
    one time, rob malda commented back to me on something. if you could do that same, that'd be greeaaaat
    - jedberg 13y ago
      I think this kind of thing doesn't really fly on HN, but sure. :)
- sologoub 13y ago
  It's a very interesting assertion to make, but really ambiguous. Slow how? Is it slow to render the page, slow DB access, slow to build? It would also be interesting to know the versions and any backstory. My guess is that none of this info exists because it, like most things done in a rush/under pressure, was probably attempted, didn't work right away and then tossed.
- ketralnis 13y ago
  And even calling it Pylons is a bit of a stretch. It's an ancient version of Pylons, of which most of the innards have been replaced over time.
jzelinskie 13y ago
This looks like a summary of the talk on InfoQ on the subject: http://www.infoq.com/presentations/scaling-reddit http://www.infoq.com/presentations/scaling-reddit
- seiji 13y ago
  highscalability is a strange reposty/blogspam aggregation thing that takes information from other places and just puts it up on their own site. I think they started having some original content, but it's still mostly second hand reports of source material found elsewhere. (Think of it more as somebody's personal notes about how things work and not an exclusive source of breaking news or architecture revelations.)
  - chaz 13y ago
    I find a lot of value in the summarization. Frankly, I'd rather read the notes on a 38 minute video and maybe watch the original source, rather than have to watch 38 minutes without knowing what the value will be. Aggregation and filtering is value, too. Like HN, it's a channel with the expectation of a certain type of content. I can't possibly discover good tech talks (or any other content) entirely on my own.
  - nasalgoat 13y ago
    The signal/noise ratio on highscalability is significantly good that it is well worth following - the quality and usefulness of the content is often much better than even HN, of which I have about a 1/100 ratio of click to ignore.
  - ketralnis 13y ago
    Yes, nobody was interviewed or anything to put this together. They just cobbled together some (mostly very old!) articles
    - jedberg 13y ago
      I wish they would put this disclaimer at the top of their articles. :(
misiti3780 13y ago
Is it common for people to use PostGres for a key-value store in production (rather than redis)?. This is the first time I have heard of it, and I am just starting to use PostGres now, so I was a bit surprised
- mason55 13y ago
  Depends what other features you need. redis will be better if you need to do things like time-based collections, sorting by z-score, etc. Postgres gives you mature clustering (at least for master-slave), a mature tooling ecosystem (PgAdmin has been around for awhile, I didn't say it was good), better access controls, etc. Like anything else in engineering, it's all a series of trade-offs to figure out what fits your needs the best.
- falcolas 13y ago
  Redis doesn't have a great solution for the D(urability) in ACID yet. PostgreSQL can ensure that changes are durable while being performant; redis can't do that as well. Redis does offer AOS for durability, but it's not nearly as mature as PostgreSQL (and comes with all sorts of caveats).
- rosser 13y ago
  It's still relatively new functionality, so I wouldn't expect to see it in wide use, but we're currently trying it out in a limited, point-solution kind of role. (We're a Postgres-mostly shop already.) So far, everything's working as well as I'd have expected from something released by the PostgreSQL community.
  - yummyfajitas 13y ago
    The functionality was always there: CREATE TABLE kvstore ( key VARCHAR(128) NOT NULL UNIQUE, value VARCHAR(128) NOT NULL ); Or are you referring to hstore?
    - ketralnis 13y ago
      And this is how reddit does it, reddit doesn't use hstore.
WestCoastJustin 13y ago
This appears to a summery of an InfoQ presentation, which was discussed about two weeks ago @ https://news.ycombinator.com/item?id=6222726 https://news.ycombinator.com/item?id=6222726
falcolas 13y ago
I can certainly appreciate what Reddit has accomplished, but the thought of losing the abilities of a full RDBMS for a key-value store makes my hair stand on end. I've yet to find schema changes limiting in my ability to code against a DB (and I use MySQL, which is one of the most limiting in this regard). Plus, I appreciate the ability to offload things like data consistancy and relationships to the database. I understand, however, where others might not feel the same way.
- diego 13y ago
  The long tail of startups will rarely need something other than a relational database because they won't get to a scale anywhere near Reddit's. It's not that others don't "feel" the same way; there's a reason all those technologies exist. If you want to know, go work for Google, Twitter, Facebook, LinkedIn, etc.
  - j_baker 13y ago
    I would argue the exact opposite. The average startup is likely using MySQL as a glorified key-value store already anyway, and they're likely using it in lieu of a more appropriate datastore because people tell them they don't need a NoSQL database until they get to Google-size. The lesson is: match your database to your use-case, not the other way around. Need advanced querying/reporting options? Get a warm, fuzzy feeling from a SQL prompt? Use MySQL. Want a plain jane key-value store? Use Voldemort/Kyoto Cabinet. Want flexible schemas? Use MongoDB. Want a Key-value store with secondary indexes and lots of scaling capabilities? Use Cassandra/HBase. Want a powerful datastore that's supported by a BigCo? Use DynamoDB or Cloud Datastore.
    - diego 13y ago
      That's not the exact opposite. Because most people are familiar with SQL databases, that's usually what they use. That's the case with the grandparent post. "Use what you know, and works for your case" is better for a startup than learning a trendy technology because you believe it might be better for your use case. Three years ago at IndexTank we were looking for a SimpleDB replacement because it just didn't work as advertised. We explored a bunch of options, and we paid a significant cost to find out that deploying Cassandra would not be worth it for us. If you have never used Cassandra and choose it because you "want a Key-value store with secondary indexes and lots of scaling capabilities" then you're in for a world of hurt.
chrismealy 13y ago
Queues were a saviour. When passing work between components put it into a queue. You get a nice little buffer. What does reddit use for queuing?
- jeffasinger 13y ago
  I believe they use RabbitAMQP
- ptolts 13y ago
  Came here to find that out as well!
- ketralnis 13y ago
  rabbitmq
peterwwillis 13y ago
You notice how in these recaps, all you read about is "I learned that X does Y"? They don't seem to have much in the way of lessons to take heed of for all situations. It's more like, "If you use this specific key/value store, tweak the thingimabob to sassyfraz to make sure your dingo does wibblydong." So if my platform doesn't use that store, your lesson is pointless. If it's a problem with an application, it's great that you're pointing it out, but if it was just oversight by lazy engineers, leave it out. Then there's the wise lessons on general topics, like the idea that you should "wait until your site grows so you can learn where your scaling problems are going to be". I'm pretty sure we know what your scaling problems are going to be. Every single resource in your platform and the way they are used will eventually pose a scaling problem. Wait until they become a problem, or plan for them to become a problem? I'm not that crazy. It really doesn't take a lot of time to plan ahead. Just think about what you have, take an hour or two and come up with some potential problems. Then sort the problems based on most-imminent most-horrible factors and make a roadmap to fix them. I know nobody likes to take time to reflect before they start banging away, but consider architectural engineering. Without careful planning, the whole building may fall apart. (Granted, nobody's going to die when your site falls apart, but it's a good mindset to be in)
- diego 13y ago
  No, you have no idea what your scale problems are going to be (if you ever have them). That is because if you get lucky and your application scales, it (and the world) will change significantly from what it is today. Let me tell you a story: in 1998 at Inktomi (look it up) we had a distributed search engine running on Sun hardware. We could not have anticipated that we'd need to migrate to Linux/PC because Sun's prices would make it impractical for us to continue scaling using their hardware. It took us three years to make the switch, and that's one of the reasons we lost to Google. Had we started two years later (when Gigabit ethernet became available for no-brand PC hardware), then we would have built the entire thing on Linux to begin with. "It really doesn't take a lot of time to plan ahead." Have you ever experienced the growth of a startup to see your infrastructure cost soar to five, six, seven figures per month? Two hours will get you as far as "one day we'll probably need to replace MySQL by something else." What you don't know is what that something else will be. Too many writes per second? Reads? Need for geographical distribution? A schema that changes all the time because you need features and fields you never imagined? Will you need to break up your store into a service-oriented architecture with different types of components? Will you run your own datacenter, or will you be on AWS? What will be the maturity level of different pieces of software in two years? I hope you get the point.
chum 13y ago
Recode Python functions in C From a security standpoint, this sounds like a bad idea
- jedberg 13y ago
  By the time it hits the C code it should be sanitized, but yes, it does add some security overhead.
- ketralnis 13y ago
  That's a really general statement, people write C code all of the time. You just have to be more careful. That is, you have to actually be a C programmer instead of a Python programmer cobbling together some C. That said, reddit uses a mix of straight-C and Cython-ised Python, which is a bit like the best of both worlds.
exhaze 13y ago
Jeremy also gave a great Airbnb tech talk on this topic: http://nerds.airbnb.com/reddit-netflix-and-beyond-building-scalable-and-reliable-architectures-in-the-cloud/ http://nerds.airbnb.com/reddit-netflix-and-beyond-building-s...
jjwiseman 13y ago
"Do not keep secret keys on the instance." I'm curious how people deal with this--what approaches do you use?
- jedberg 13y ago
  Amazon now provides a service to give you on instance keys: http://aws.amazon.com/iam/faqs/#What_is_IAM_roles_for_EC2_instances http://aws.amazon.com/iam/faqs/#What_is_IAM_roles_for_EC2_in... Before that at Netflix we developed a service that would hand out temporary keys to the requestor when they presented a proper certificate. At reddit we put the secret keys on the instance, which was bad. :)
  - arohner 13y ago
    Very cool. Are there any publicly available options for non-AWS services?
ivanbrussik 13y ago
Just out of curiosity what does "stay as schemaless" as possible that did not read right?
continuations 13y ago
> For comments it’s very fast to tell which comments you didn’t vote on, so the negative answers come back quickly. Can you get into more details about how this is used? If reddit needs to display a page that has 100 comments, do they query Cassandra on the voting status of the user on those 100 comments? I thought Cassandra was pretty slow in reads (slower than postgres) so how does using Cassandra make it fast here?
- extesy 13y ago
  As far as I understand, since user most likely voted only on a small subset of those 100 comments (say 3) and negative lookups are very fast because of bloom filters [1], therefore all lookups combined are fast. [1] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
  - continuations 13y ago
    That makes sense. Thanks.
- ketralnis 13y ago
  > Can you get into more details about how this is used? If reddit needs to display a page that has 100 comments, do they query Cassandra on the voting status of the user on those 100 comments? Sort of, yeah. There are two versions of this, the old way and the new way. Old way: keep a cache of the last time they voted. Remove any comments from those 100 that are younger than the last vote (since they can't possibly have voted on them). Then look up the keys containing the remaining ones. Most of them hit the bloom filter, those that pass the bloom filter actually get looked up. In the worst case this is all 100 comments, which can hit up up to 1/RF of your Cassandra nodes. The worst case doesn't happen often. The new way is a little different, you have one Cassandra row (so only one machine) containing all of the votes for the user (perhaps further limited to a given link ID or date). You hit that one node for all 100 comments. If you have per-row bloom filters, see Old Way for the rest. > I thought Cassandra was pretty slow in reads (slower than postgres) so how does using Cassandra make it fast here? "Fast" and "slow" as used here is very naive, performance is usually more complicated than simple statements like this. I guess if you had 1 postgres node with 1 key and one Cassandra node with 1 key, and only 1 client, you could answer simple generalisations like this. But reddit has thousands of concurrent hits spread over hundreds of servers with varying amounts of RAM, I/O performance, network bottlenecks, and usage profiles. The single biggest win is that you can easily horizontally scale Cassandra. Just add more nodes until it's "fast". But even that's a gross simplification. For another example, if you scale Postgres by adding a bunch of replicas and choose a random one to read from for a given key, then they all have the same data in RAM, so your block cache miss rate is very high (that is, your effective block cache is the amount of RAM in one machine). Additionally, your write performance is capped to the write performance of your one master. Your replication throughput is capped to his outbound network bandwidth. So you want a subset of the data on all of them so that whichever machine you ask has a very high likelihood of having that data in block cache. So you shard it by some function. But then you want to add more postgres machines, you have to migrate the data somehow, without shutting down the site. You've now more or less written Cassandra.
727374 13y ago
"Treat nonlogged in users as second class citizens. By always giving logged out always cached content Akamai bears the brunt for reddit’s traffic. Huge performance improvement. " This is the lowest of low hanging fruit. Many people don't realize it but a ton of huge media sites use Akamai to offload most of their "read-only" traffic.
- ketralnis 13y ago
  Definitely true, and one of the earliest and longest-standing optimisations. Even pre-Akamai, we had simple caching, both whole-page and per-object/query.
callmeed 13y ago
Can someone elaborate/clarify this: > Users connect to a web tier which talks to an application tier. So, I'm assuming the web tier is nginx/haproxy and the application tier is Pylons. Are the 240 servers mentioned all running both the web tier and the app tier?
- computer 13y ago
  Presumably the web tier does slightly more than just reverse proxying. For example, it could build (render) pages based on an internal RedditAPI it queries. This RedditAPI (application layer) would then basically be a distributed database front-end with some state, like user sessions. Seperating it at that point allows the web tier to offload much of the work (mostly rendering), while keeping it stateless, thus allowing effortless scaling of that tier.
- ketralnis 13y ago
  No. Some are postgres, some are cassandra, some are web servers (haproxy), some are app servers (the reddit app running inside of pylons), etc
skeletonjelly 13y ago
jedberg - you speak of automation, did you use anything (or is there anything in use currently) that handles auto scaling for EC2? puppet/chef/ansible etc? Or was this all done by hand?
gbog 13y ago
> Stay as schemaless as possible. It makes it easy to add features. All you need to do is add new properties without having to alter tables. And at the same time they use and praise Postgres a lot, so it cannot be about NoSQL. I am wondering what they mean exactly. From my own tendency, it should mean use a few very big and narrow tables in the form of "who - do - what - when - where", eg "userA - vote up - comment1 - timestamp - foosubreddit", and also "userB - posted - link1 - timestamp - barsubreddit" Then in the same table you get kinda all events happening in the site, and you are somewhat schemaless, in the sense that adding a new functionality do not require schema change. If someone with inner insight can confirm this is not too far from what reddit team meant, I'd appreciate.
- free652 13y ago
  >Postgres is a great database. It makes a wonderful, really fast key-value store ^^ They probably have a basic schema, the rest is in KV store (guess)
- ketralnis 13y ago
  > And at the same time they use and praise Postgres a lot, so it cannot be about NoSQL. We had a basic schema that basically made postgres into a K/V store. So we had both.
na85 13y ago
Reddit is an interesting case; they seem to have almost unlimited amounts of user good will. Case in point: I get the "you broke reddit" pageload failure message an awful lot and I'm sure others do too. How many other sites have userbases that would tolerate such a high number of errors?
- hboon 13y ago
  Not many perhaps. But Twitter did too.
- ape4 13y ago
  Its a discussion site. Not an eCommerce site.

229 points by jpmc 13y ago

srj55 13y ago

hmm...no love for django here.

deleted 13y ago
[deleted]
falcolas 13y ago
I'm currently using Django, and I can understand where they are coming from. The sheer amount of code that a simple request has to go through to receive an answer is staggering sometimes, such as 18 line tracebacks just to identify an authentication problem... My own projects are not yet large enough to have this cause an issue, but I can see where something the size of Reddit would indeed have issues that even the most aggressive caching can't resolve.
film42 13y ago
It's not a matter of rating frameworks, it's a matter of purpose. Django and rails are designed to go from nothing to product as quickly as possible. Plus, thanks to better hardware and scaling techniques, it's easy and affordable to stick to frameworks longterm (instagram and github, for example). Sure, they could they rewrite their site in C++, Go or Erlang, but they would lose their ability to rapid prototype new features.

human_error 13y ago

> Used the Pylons (Django was too slow), a Python based framework, from the start This isn't quite right. It was web.py at the beginning. They have started using Pylons after Conde Nast acquisition.

showerst 13y ago
Originally it was actually written in LISP. =) http://blog.reddit.com/2005/12/on-lisp.html http://blog.reddit.com/2005/12/on-lisp.html
jedberg 13y ago
Yes, I glossed over that part. We didn't use web.py very long.
- dmead 13y ago
  one time, rob malda commented back to me on something. if you could do that same, that'd be greeaaaat
  - jedberg 13y ago
    I think this kind of thing doesn't really fly on HN, but sure. :)
sologoub 13y ago
It's a very interesting assertion to make, but really ambiguous. Slow how? Is it slow to render the page, slow DB access, slow to build? It would also be interesting to know the versions and any backstory. My guess is that none of this info exists because it, like most things done in a rush/under pressure, was probably attempted, didn't work right away and then tossed.
ketralnis 13y ago
And even calling it Pylons is a bit of a stretch. It's an ancient version of Pylons, of which most of the innards have been replaced over time.

jzelinskie 13y ago

This looks like a summary of the talk on InfoQ on the subject: http://www.infoq.com/presentations/scaling-reddit http://www.infoq.com/presentations/scaling-reddit

seiji 13y ago
highscalability is a strange reposty/blogspam aggregation thing that takes information from other places and just puts it up on their own site. I think they started having some original content, but it's still mostly second hand reports of source material found elsewhere. (Think of it more as somebody's personal notes about how things work and not an exclusive source of breaking news or architecture revelations.)
- chaz 13y ago
  I find a lot of value in the summarization. Frankly, I'd rather read the notes on a 38 minute video and maybe watch the original source, rather than have to watch 38 minutes without knowing what the value will be. Aggregation and filtering is value, too. Like HN, it's a channel with the expectation of a certain type of content. I can't possibly discover good tech talks (or any other content) entirely on my own.
- nasalgoat 13y ago
  The signal/noise ratio on highscalability is significantly good that it is well worth following - the quality and usefulness of the content is often much better than even HN, of which I have about a 1/100 ratio of click to ignore.
- ketralnis 13y ago
  Yes, nobody was interviewed or anything to put this together. They just cobbled together some (mostly very old!) articles
  - jedberg 13y ago
    I wish they would put this disclaimer at the top of their articles. :(

misiti3780 13y ago

Is it common for people to use PostGres for a key-value store in production (rather than redis)?. This is the first time I have heard of it, and I am just starting to use PostGres now, so I was a bit surprised

mason55 13y ago
Depends what other features you need. redis will be better if you need to do things like time-based collections, sorting by z-score, etc. Postgres gives you mature clustering (at least for master-slave), a mature tooling ecosystem (PgAdmin has been around for awhile, I didn't say it was good), better access controls, etc. Like anything else in engineering, it's all a series of trade-offs to figure out what fits your needs the best.
falcolas 13y ago
Redis doesn't have a great solution for the D(urability) in ACID yet. PostgreSQL can ensure that changes are durable while being performant; redis can't do that as well. Redis does offer AOS for durability, but it's not nearly as mature as PostgreSQL (and comes with all sorts of caveats).
rosser 13y ago
It's still relatively new functionality, so I wouldn't expect to see it in wide use, but we're currently trying it out in a limited, point-solution kind of role. (We're a Postgres-mostly shop already.) So far, everything's working as well as I'd have expected from something released by the PostgreSQL community.
- yummyfajitas 13y ago
  The functionality was always there: CREATE TABLE kvstore ( key VARCHAR(128) NOT NULL UNIQUE, value VARCHAR(128) NOT NULL ); Or are you referring to hstore?
  - ketralnis 13y ago
    And this is how reddit does it, reddit doesn't use hstore.

WestCoastJustin 13y ago

This appears to a summery of an InfoQ presentation, which was discussed about two weeks ago @ https://news.ycombinator.com/item?id=6222726 https://news.ycombinator.com/item?id=6222726

falcolas 13y ago

I can certainly appreciate what Reddit has accomplished, but the thought of losing the abilities of a full RDBMS for a key-value store makes my hair stand on end. I've yet to find schema changes limiting in my ability to code against a DB (and I use MySQL, which is one of the most limiting in this regard). Plus, I appreciate the ability to offload things like data consistancy and relationships to the database. I understand, however, where others might not feel the same way.

diego 13y ago
The long tail of startups will rarely need something other than a relational database because they won't get to a scale anywhere near Reddit's. It's not that others don't "feel" the same way; there's a reason all those technologies exist. If you want to know, go work for Google, Twitter, Facebook, LinkedIn, etc.
- j_baker 13y ago
  I would argue the exact opposite. The average startup is likely using MySQL as a glorified key-value store already anyway, and they're likely using it in lieu of a more appropriate datastore because people tell them they don't need a NoSQL database until they get to Google-size. The lesson is: match your database to your use-case, not the other way around. Need advanced querying/reporting options? Get a warm, fuzzy feeling from a SQL prompt? Use MySQL. Want a plain jane key-value store? Use Voldemort/Kyoto Cabinet. Want flexible schemas? Use MongoDB. Want a Key-value store with secondary indexes and lots of scaling capabilities? Use Cassandra/HBase. Want a powerful datastore that's supported by a BigCo? Use DynamoDB or Cloud Datastore.
  - diego 13y ago
    That's not the exact opposite. Because most people are familiar with SQL databases, that's usually what they use. That's the case with the grandparent post. "Use what you know, and works for your case" is better for a startup than learning a trendy technology because you believe it might be better for your use case. Three years ago at IndexTank we were looking for a SimpleDB replacement because it just didn't work as advertised. We explored a bunch of options, and we paid a significant cost to find out that deploying Cassandra would not be worth it for us. If you have never used Cassandra and choose it because you "want a Key-value store with secondary indexes and lots of scaling capabilities" then you're in for a world of hurt.

chrismealy 13y ago

Queues were a saviour. When passing work between components put it into a queue. You get a nice little buffer. What does reddit use for queuing?

jeffasinger 13y ago
I believe they use RabbitAMQP
ptolts 13y ago
Came here to find that out as well!
ketralnis 13y ago
rabbitmq

peterwwillis 13y ago

You notice how in these recaps, all you read about is "I learned that X does Y"? They don't seem to have much in the way of lessons to take heed of for all situations. It's more like, "If you use this specific key/value store, tweak the thingimabob to sassyfraz to make sure your dingo does wibblydong." So if my platform doesn't use that store, your lesson is pointless. If it's a problem with an application, it's great that you're pointing it out, but if it was just oversight by lazy engineers, leave it out. Then there's the wise lessons on general topics, like the idea that you should "wait until your site grows so you can learn where your scaling problems are going to be". I'm pretty sure we know what your scaling problems are going to be. Every single resource in your platform and the way they are used will eventually pose a scaling problem. Wait until they become a problem, or plan for them to become a problem? I'm not that crazy. It really doesn't take a lot of time to plan ahead. Just think about what you have, take an hour or two and come up with some potential problems. Then sort the problems based on most-imminent most-horrible factors and make a roadmap to fix them. I know nobody likes to take time to reflect before they start banging away, but consider architectural engineering. Without careful planning, the whole building may fall apart. (Granted, nobody's going to die when your site falls apart, but it's a good mindset to be in)

diego 13y ago
No, you have no idea what your scale problems are going to be (if you ever have them). That is because if you get lucky and your application scales, it (and the world) will change significantly from what it is today. Let me tell you a story: in 1998 at Inktomi (look it up) we had a distributed search engine running on Sun hardware. We could not have anticipated that we'd need to migrate to Linux/PC because Sun's prices would make it impractical for us to continue scaling using their hardware. It took us three years to make the switch, and that's one of the reasons we lost to Google. Had we started two years later (when Gigabit ethernet became available for no-brand PC hardware), then we would have built the entire thing on Linux to begin with. "It really doesn't take a lot of time to plan ahead." Have you ever experienced the growth of a startup to see your infrastructure cost soar to five, six, seven figures per month? Two hours will get you as far as "one day we'll probably need to replace MySQL by something else." What you don't know is what that something else will be. Too many writes per second? Reads? Need for geographical distribution? A schema that changes all the time because you need features and fields you never imagined? Will you need to break up your store into a service-oriented architecture with different types of components? Will you run your own datacenter, or will you be on AWS? What will be the maturity level of different pieces of software in two years? I hope you get the point.

chum 13y ago

Recode Python functions in C From a security standpoint, this sounds like a bad idea

jedberg 13y ago
By the time it hits the C code it should be sanitized, but yes, it does add some security overhead.
ketralnis 13y ago
That's a really general statement, people write C code all of the time. You just have to be more careful. That is, you have to actually be a C programmer instead of a Python programmer cobbling together some C. That said, reddit uses a mix of straight-C and Cython-ised Python, which is a bit like the best of both worlds.

exhaze 13y ago

Jeremy also gave a great Airbnb tech talk on this topic: http://nerds.airbnb.com/reddit-netflix-and-beyond-building-scalable-and-reliable-architectures-in-the-cloud/ http://nerds.airbnb.com/reddit-netflix-and-beyond-building-s...

jjwiseman 13y ago

"Do not keep secret keys on the instance." I'm curious how people deal with this--what approaches do you use?

jedberg 13y ago
Amazon now provides a service to give you on instance keys: http://aws.amazon.com/iam/faqs/#What_is_IAM_roles_for_EC2_instances http://aws.amazon.com/iam/faqs/#What_is_IAM_roles_for_EC2_in... Before that at Netflix we developed a service that would hand out temporary keys to the requestor when they presented a proper certificate. At reddit we put the secret keys on the instance, which was bad. :)
- arohner 13y ago
  Very cool. Are there any publicly available options for non-AWS services?

ivanbrussik 13y ago

Just out of curiosity what does "stay as schemaless" as possible that did not read right?

continuations 13y ago

> For comments it’s very fast to tell which comments you didn’t vote on, so the negative answers come back quickly. Can you get into more details about how this is used? If reddit needs to display a page that has 100 comments, do they query Cassandra on the voting status of the user on those 100 comments? I thought Cassandra was pretty slow in reads (slower than postgres) so how does using Cassandra make it fast here?

extesy 13y ago
As far as I understand, since user most likely voted only on a small subset of those 100 comments (say 3) and negative lookups are very fast because of bloom filters [1], therefore all lookups combined are fast. [1] https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter
- continuations 13y ago
  That makes sense. Thanks.
ketralnis 13y ago
> Can you get into more details about how this is used? If reddit needs to display a page that has 100 comments, do they query Cassandra on the voting status of the user on those 100 comments? Sort of, yeah. There are two versions of this, the old way and the new way. Old way: keep a cache of the last time they voted. Remove any comments from those 100 that are younger than the last vote (since they can't possibly have voted on them). Then look up the keys containing the remaining ones. Most of them hit the bloom filter, those that pass the bloom filter actually get looked up. In the worst case this is all 100 comments, which can hit up up to 1/RF of your Cassandra nodes. The worst case doesn't happen often. The new way is a little different, you have one Cassandra row (so only one machine) containing all of the votes for the user (perhaps further limited to a given link ID or date). You hit that one node for all 100 comments. If you have per-row bloom filters, see Old Way for the rest. > I thought Cassandra was pretty slow in reads (slower than postgres) so how does using Cassandra make it fast here? "Fast" and "slow" as used here is very naive, performance is usually more complicated than simple statements like this. I guess if you had 1 postgres node with 1 key and one Cassandra node with 1 key, and only 1 client, you could answer simple generalisations like this. But reddit has thousands of concurrent hits spread over hundreds of servers with varying amounts of RAM, I/O performance, network bottlenecks, and usage profiles. The single biggest win is that you can easily horizontally scale Cassandra. Just add more nodes until it's "fast". But even that's a gross simplification. For another example, if you scale Postgres by adding a bunch of replicas and choose a random one to read from for a given key, then they all have the same data in RAM, so your block cache miss rate is very high (that is, your effective block cache is the amount of RAM in one machine). Additionally, your write performance is capped to the write performance of your one master. Your replication throughput is capped to his outbound network bandwidth. So you want a subset of the data on all of them so that whichever machine you ask has a very high likelihood of having that data in block cache. So you shard it by some function. But then you want to add more postgres machines, you have to migrate the data somehow, without shutting down the site. You've now more or less written Cassandra.

727374 13y ago

"Treat nonlogged in users as second class citizens. By always giving logged out always cached content Akamai bears the brunt for reddit’s traffic. Huge performance improvement. " This is the lowest of low hanging fruit. Many people don't realize it but a ton of huge media sites use Akamai to offload most of their "read-only" traffic.

ketralnis 13y ago
Definitely true, and one of the earliest and longest-standing optimisations. Even pre-Akamai, we had simple caching, both whole-page and per-object/query.

callmeed 13y ago

Can someone elaborate/clarify this: > Users connect to a web tier which talks to an application tier. So, I'm assuming the web tier is nginx/haproxy and the application tier is Pylons. Are the 240 servers mentioned all running both the web tier and the app tier?

computer 13y ago
Presumably the web tier does slightly more than just reverse proxying. For example, it could build (render) pages based on an internal RedditAPI it queries. This RedditAPI (application layer) would then basically be a distributed database front-end with some state, like user sessions. Seperating it at that point allows the web tier to offload much of the work (mostly rendering), while keeping it stateless, thus allowing effortless scaling of that tier.
ketralnis 13y ago
No. Some are postgres, some are cassandra, some are web servers (haproxy), some are app servers (the reddit app running inside of pylons), etc

skeletonjelly 13y ago

jedberg - you speak of automation, did you use anything (or is there anything in use currently) that handles auto scaling for EC2? puppet/chef/ansible etc? Or was this all done by hand?

gbog 13y ago

> Stay as schemaless as possible. It makes it easy to add features. All you need to do is add new properties without having to alter tables. And at the same time they use and praise Postgres a lot, so it cannot be about NoSQL. I am wondering what they mean exactly. From my own tendency, it should mean use a few very big and narrow tables in the form of "who - do - what - when - where", eg "userA - vote up - comment1 - timestamp - foosubreddit", and also "userB - posted - link1 - timestamp - barsubreddit" Then in the same table you get kinda all events happening in the site, and you are somewhat schemaless, in the sense that adding a new functionality do not require schema change. If someone with inner insight can confirm this is not too far from what reddit team meant, I'd appreciate.

free652 13y ago
>Postgres is a great database. It makes a wonderful, really fast key-value store ^^ They probably have a basic schema, the rest is in KV store (guess)
ketralnis 13y ago
> And at the same time they use and praise Postgres a lot, so it cannot be about NoSQL. We had a basic schema that basically made postgres into a K/V store. So we had both.

na85 13y ago

Reddit is an interesting case; they seem to have almost unlimited amounts of user good will. Case in point: I get the "you broke reddit" pageload failure message an awful lot and I'm sure others do too. How many other sites have userbases that would tolerate such a high number of errors?

hboon 13y ago
Not many perhaps. But Twitter did too.
ape4 13y ago
Its a discussion site. Not an eCommerce site.