32 ms·
Caching at Reddit
- akjainaj 10y agoTaking into account how reddit performs, I'll take this as a guide on how not to use cache. (This is a joke please. Understand it as such. I know reddit has the problems it has because it is severely understaffed)
- dageshi 10y agoHonestly I do recall periods when reddit wasn't performing particularly well but recently it's seemed fine to me.
- akjainaj 10y agoFor me all pages take 3+ seconds to load (generation time), which means reddit is easily the slowest site I visit.
- praneshp 10y agoHave you tried turning off all adblockers on your browser? Or is your time before all those come into play?
- pfg 10y agoFWIW, a GET / (front page) while logged in takes 1-1.5s for me (TTFB). ~3s till load with ads blocked, ~5.5s with ads. Both numbers without caching, but that only seems to shave off about 200ms (most of the time is spent in JS and rendering the site).
- praneshp 10y agoOkay, thank you.
- akjainaj 10y agoFront page does indeed take that. But many links (comment pages) take around 3 seconds as I mentioned (generation time, has nothing to do at all with my browser) and pages with hundreds of comments (like most links in the default subs) can easily take over 5 seconds to generate.
- gempir 10y agoThey are number 19 on Alexa and are seriously understaffed for the traffic they are getting. I think they are holding up just fine. All they monetize is Reddit gold and a few small ads, which I'm not even sure are real ads or just ads for more reddit itself
- slavik81 10y ago> Performance matters. It took 8.4 seconds to load the Reddit front page on my phone. Hacker News took 1.1 seconds. This feels like advice from the overweight gym teacher on how to do pushups. The desktop Reddit site took 2.2 seconds over the same connection, by the way. It seems like it would be much more valuable to optimize whatever is taking up >75% of page time on mobile.
- dsp1234 10y agoAnother point of data, on mobile, navigating to reddit.com: T 0: white page T 5: to see the reddit header with logo, the rest of the page is still white T 16: to see the bar stating that I should use the app, click on the tiny mobile site link T 25: page visible behind the "app" bar stating I should use the app T 32: the bar is gone, and the site is "usable"
- disiplus 10y agoreddit has i.reddit.com (old mobile page) that is faster then the new one. i hope they dont turn that off because i use it sometimes. the new one is just slow.
- SOLAR_FIELDS 10y agoI use this too. It's also better UX in my opinion. In the new site, a lot of the buttons are either hard to see or too small to reliably hit with the finger on the first try.
- yoodenvranx 10y agoI am glad to see that I am not alone with my preference of the old mobile site. It fast, simple and it does not have any unnecessary whitespace and padding which just waste valuable space.
- Laforet 10y agoThe new mobile site is slkw fails in basic functions such as as preventing double posting - which happens all the time because the UI is often irresponsive.
- eunoia 10y agoSome back of the envelope math for their caching costs: 54 x R3.2xlarge EC2 instances On demand = $314,571.6/year w/ 1 year term = $166,860/year w/ 3 year term = $110,340/year w/ convertible 3 year term = $150,174/year Is that a lot? Seems like a lot
- degenerate 10y agoFor a text link site that lets you vote submissions up and down? Yes that's a lot. I'm sure a lot of the decisions at reddit corp were along the lines of, "well, we can't go back and change that now, so let's do whatever we need to fix it and move forward" AKA lots of RAM and caching all over the place.
- mwpmaybe 10y agoIf a company is grossing $20MM/year and stands to improve that by more than 0.75% by having a faster site, then yes, it makes sense to spend 0.75% of your revenue to do so.
- flashman 10y agoOne of my favourite breeds of internet comment is "I'm sure five competent engineers could run Reddit at its current scale."
- praneshp 10y agoIf you think that's all reddit is (text link, with up and down), you have to go use it a little bit. Subreddits, comments, self-hosted images, etc disagree.
- user5994461 10y agoHard to tell. Don't know how much volume they have. Going from a 50th site in the world in a niche topic (stack overflow) to a 27th worldwide that target all audience. It's entirely possible that it grows the traffic by 100x. Anyway, who cares about money? 100k a year is peanuts. At this scale, if you have a solution that works well, you're good. Not even worth thinking about it.
- sciurus 10y agoThe pain of static slab allocation is real! Changing usage patterns causing problems can be tricky to track down too; mcsauna looks helpful for this. Upgrading to memcached 1.4.25 and running with "slab_reassign,slab_automove,lru_crawler,lru_maintainer" was a huge improvement for our primary memcached cluster at Eventbrite.
- Florin_Andrei 10y agoAre those visualizations done in Grafana?
- dkasper 10y agoYes, Grafana with Graphite as the backend store.
- jjoe 10y agoThe slowness of page load mentioned by folks here is the reason why I think caching at the HTTP level (ex: Varnish) is much more efficient than caching at the service level (ex: Memcached), which is much further down the stack and is bound to be latency-sensitive. Because it's much less entangled in your code and deep into your infrastructure (less technical debt). A hybrid approach can work too but only if it's light and unobtrusive. By the way, and I'm going out on a limb with my shameful plug, I built a Varnish-as-a-Service kind of infrastructure called Cachoid ( https://www.cachoid.com https://www.cachoid.com ). But to my own defense, I'm putting my energy, time, and money where my mouth is.
- ssambros 10y agoTo my understanding Reddit serves a highly customized content to each logged in user. Can you help me understand how HTTP level caching will solve this problem more efficiently than their service level caching does right now?
- jjoe 10y agoYou cache non-logged in users to start with. And then you cache based on sessions (logged in users) because you don't really need to show fresh votes on each visit and right away (admitted to it in the article). Plus there's lots of room for ESI.
- Klathmon 10y agoThey already do those things. Logged out users see a "snapshot" of the page updated every so often. And I really don't think that caching pages per session would really help with their load all that much. Why not just use HTTP cache headers at that point? Plus while you don't really need to show votes ASAP, logged in users will want up to date comments.
- jjoe 10y agoThis is where ESI helps. You can portions of the page that don't change much.
- bluedino 10y agoI wonder how much better/worse the site would run if they had their own hardware like StackExchange. And I wonder how StackExchange would run if it were on AWS.
- brokenmachine 10y agoI read an article about StackExchange's servers and it seemed super optimized, although I'm not an expert. I can't imagine AWS would be as optimized for exactly StackExchange's requirements. So I think StackExchange would run slower on AWS. Reddit, on the other hand, is the worst performing out of all the top sites that I can think of.
- nasalgoat 10y agoIn my experience running large sites, dedicated hardware is not only cheaper but orders of magnitude faster because you can finely-tune the hardware to very, very specific use cases, and you can have a fully private, highly optimized network with very small cable runs to effectively eliminate network latency issues. Most of my current job involves solving problems that are caused by cloud limitations.
- mtanski 10y agoAnd if you need throughput, 40gigE / 56gigE is not that hard expensive with your own hardware. You end up with like 4x to 10x servers to handle the same load.
- manigandham 10y agoNick Craver (one of their engineers) writes a lot about their architecture. They would be much slower in the cloud. https://nickcraver.com/blog/archive/ https://nickcraver.com/blog/archive/
- gtirloni 10y agoOnce you get everything automated through those cloud APIs, it's all very convenient. The on-prem alternative is to run something like OpenStack, which is a major PITA.
- nodesocket 10y agoI wonder if switching from memcached to redis would make a bottom line difference in terms of the number of instances needed (cost) and performance?
- deedubaya 10y agoMaybe a difference in performance, but caching comes down to bits-to-be-stored. That likely wouldn't change.
- nodesocket 10y agoPerhaps they are already doing it, but couldn't they use client side compression as well?
- cookiecaper 10y agoredis does have some built-in features that automatically compact data. [0] I'm sure reddit has considered this, but worth noting. [0] https://redis.io/topics/memory-optimization https://redis.io/topics/memory-optimization
- sgmansfield 10y agoclient-side compression works out better for a couple of reasons: 1. You use less bandwidth in/out of the box. This matters in AWS because your bandwidth is (relatively) limited. 2. Your CPU usage to (de)compress is distributed over a larger fleet. The clients spend the CPU time, not the cache servers.
- cookiecaper 10y agoI thought the same thing. I stuck with memcached for quite a while because I didn't see a real reason to switch to redis, but after hitting some repeated annoyances with memcached, I gave redis a try and very happy I did. It's much better. Is memcached even maintained anymore?
- sgmansfield 10y ago
- deleted 10y ago[deleted]
- jrowley 10y agoThe cache-perma seems pretty clever to me. > For example, when new comments are added or votes are changed, we don’t simply invalidate the cache and move on—this happens too frequently and would make the caching near useless. Instead, we update the backend store (in Cassandra) as well as the cache. Fallback can always happen to the backend store if need be, but in practice this rarely happens. In fact, permacache is one of our best hit rates—over 99%. They basically have their application state duplicated in both places. Interesting architectural choice.
- hehheh 10y agoThis sounds an awful lot like an old(ish?) technique termed "write-through caching". Building data structures that can take advantage of write-through caching at reddit-comment-scale does sound like it'd be an interesting problem to optimize.
- jedberg 10y agoThat's basically what it is. The data is abstracted away so that when you do a write it just goes to both places as needed.
- ksec 10y agoSlightly Off Topic, as of 2017, what's the advantage of Memcached over Redis. I thought we are basically in the era of Redis.
- greyskull 10y agoSorta answered under here: https://www.reddit.com/r/programming/comments/5or8m1/caching_at_reddit/dclhgsz/ https://www.reddit.com/r/programming/comments/5or8m1/caching...
- justintbassett 10y agoalso it sounds too similar to "reddit"; it would lead to hilarious Abbott and Costello-esque hijinks
- antirez 10y agoIt's hard to talk in general terms but Memcached is threaded so you can saturate your CPUs without requiring multiple instances, and Redis has more advanced features both from the POV of programmer API facing and operations. But if we want to zoom on Reddit itself, the fact of using data structures and changing caching paradigm to really use Redis the proper way, that is, storing metadata in Redis and not just only in their main DB, I suspect would provide a very big boost to Reddit. For some reason Reddit always has been an "anti Redis" shop. I'm sure they have their good reasons, and btw I love Reddit too much to complain, whatever they do to run it, I don't care as long as they provide such a wonderful service to the community :-) But... their use case is IMHO one that you can accelerate tremendously by using Redis. I ran the most popular Italian Reddit-alike site for years, and I wrote a simple Reddit clone that uses Redis so I had the opportunity of exploring the problem a bit.
- gooeyblob 10y agoReddit is most definitely not an "anti Redis" shop! If we were to rewrite it today, we might start with Redis. We've just put so much work into making memcached reliable and easy to understand for developers that the possible benefits of switching to Redis for many of these use cases don't outweigh the operational knowledge we have from years of running memcached at this scale. That said - we use Redis in at least 2 (soon to be 3) capacities across the site for different services (outside of the monolith) and it works really well.
- bbeausej 10y agoThanks for sharing the details. It's impressive to see the memory allocation and pool size for a site handling this much traffic. I would love to get some more information on reddit's platform overall traffic volume as I feel this would complement the discussion nicely.
- chime 10y ago> When you vote, your vote isn’t instantly processed—instead, it’s placed into a queue. I remember looking into this a while ago and was bewildered to find that when I upvoted or downvoted, there was no XHR call to the backend! There was no hidden iframe/image, no silent form post. Absolutely no network activity. Yet when I refreshed, my vote was shown correctly. I thought I was going crazy. This was long ago so I'm a bit fuzzy on the details but after a bit of digging, I found the most elegant data collection technique I've ever seen. Instead of sending network data when I voted, a local cookie was set with the link id and vote value. Then when I went to another page, my browser naturally sent the cookie to the server, where I believe it was processed, and then a fresh cookie was sent back to my browser. I could vote on 10 links, the local cookie would get large and then on the next page refresh, the backend would receive my batch of votes, process them, and send me a fresh cookie again. I don't think they do that now and I've never seen anyone do something like this. Even HN just makes an XHR call on voting. After twenty years on the web, it's not often that I am surprised so this was quite a thrill.
- dirtae 10y agoWhat if the next page that you go to isn't on Reddit?
- jhanschoo 10y agoThen the next time you visit reddit, I guess.
- test1235 10y agoMaybe I'm not a 'normal' user so it wouldn't have mattered to them, but I used to open loads of tabs at once, and slowly make my way across all the stories, then more often than not, I'd close the browser where all my cache, cookies and history and whatnot are cleared. This would mean none of my votes would ever have made it to the server!
- dspillett 10y ago
- deleted 10y ago[deleted]
- QuercusMax 10y agoIronic timing that Reddit is currently undergoing a major outage right now...
- gooeyblob 10y agoNot really, there were some routing issues with our CDN affecting an extremely small percentage of traffic.
- QuercusMax 10y agoSeemed to be down for lots of people in the bay area: http://downdetector.com/status/reddit http://downdetector.com/status/reddit Maybe only small because most californians were still sleeping? I was only up due to a sick kid. :D
- egonschiele 10y agoWhat's the difference between mcrouter vs something like haproxy?