4 ms·

Nah definitely not confused about the architecture, it's fairly simple to wrap your head around. What I'm advocating for is making the general site semi-dynamic

by pg_bot 8y ago

Nah definitely not confused about the architecture, it's fairly simple to wrap your head around. What I'm advocating for is making the general site semi-dynamic instead of fully dynamic. (Break expensive caches at regular predefined time intervals instead of every time data changes) The key insight is that reddit still functions as you expect if you are seeing stale data from other people so long as your posts/comments/votes are visible when you navigate. Shard the databases by subreddit and you will be able to horizontally scale the load. Use JS to post to a separate subdomain for each service. I.E everything under www.reddit.com/r/funny points to funny.reddit.com which can live as its own independent service. Internally I would treat reddit as a group of individual high traffic sites instead of one massively trafficked site. Then you have another service that queries each site to build the main page.

cagenut 8y ago
Shard the databases by subreddit and you will be able to horizontally scale the load. Use JS to post to a separate subdomain for each service. I.E everything under www.reddit.com/r/funny points to funny.reddit.com which can live as its own independent service. Internally I would treat reddit as a group of individual high traffic sites instead of one massively trafficked site. Then you have another service that queries each site to build the main page. this is terrifying to read, in that this is exactly what the media companies did for the last decade to try, and totally fail, to compete with facebook. having a unified platform is a competitive advantage. also your "sharding and subdomains" trick is straight out of 2010 and a mix of irrelevant (everything's sharded now) and anachronistic (h2 is everywhere now). also-also, you clearly have missed the best use case of reddit, which is to have unsub'd from all/most of the "default" reddits and replaced them with your own customized set. you are optimizing for everyone reading the same thing (newspaper homepage) not everyone reading their custom thing (social network). that has been a proven losing strategy for a solid decade now, you could argue 2.
- pg_bot 8y ago
  Everyone is reading the same thing on reddit, just different combinations of the same thing. Reddit is just a combination of multiple newspapers masquerading as a single custom newspaper. No reason why the internals should leak to what is shown to the user. You could keep the exact same user experience that you have today and guarantee that the site never falls over. You need to make the distinction between UX and Operations. Everything the user sees still lives under the same structure as today but internally it is structured so that you don't overload yourself and if you do fail, those failures are isolated and don't take down the rest of the site.
  - cagenut 8y ago
    Everyone is reading the same thing on reddit, just different combinations of the same thing. this is categorically untrue, you don't seem to understand reddit. Reddit is just a combination of multiple newspapers masquerading as a single custom newspaper. that is exactly the anachronistic thinking that leads you down this path. its movable type vs wordpress all over again. wordpress won. the engineering constraint of pre-calculating caches is worse, in the long run, than the occasional outage for casual/non-logged-in users. its optimizing for the wrong audience, it is a long slow path to failure.
ecto 8y ago
We're taking a sort of hybrid approach to this on https://threadbase.io https://threadbase.io, where you can get a subdomain for your community (or custom domain in the future) and put your own ads in (so moderators can make money). Eventually we'll create a shared homepage for site creators who opt in, and allow you to filter communities you want to see there.

by pg_bot 8y ago

cagenut 8y ago

Shard the databases by subreddit and you will be able to horizontally scale the load. Use JS to post to a separate subdomain for each service. I.E everything under www.reddit.com/r/funny points to funny.reddit.com which can live as its own independent service. Internally I would treat reddit as a group of individual high traffic sites instead of one massively trafficked site. Then you have another service that queries each site to build the main page. this is terrifying to read, in that this is exactly what the media companies did for the last decade to try, and totally fail, to compete with facebook. having a unified platform is a competitive advantage. also your "sharding and subdomains" trick is straight out of 2010 and a mix of irrelevant (everything's sharded now) and anachronistic (h2 is everywhere now). also-also, you clearly have missed the best use case of reddit, which is to have unsub'd from all/most of the "default" reddits and replaced them with your own customized set. you are optimizing for everyone reading the same thing (newspaper homepage) not everyone reading their custom thing (social network). that has been a proven losing strategy for a solid decade now, you could argue 2.

pg_bot 8y ago
Everyone is reading the same thing on reddit, just different combinations of the same thing. Reddit is just a combination of multiple newspapers masquerading as a single custom newspaper. No reason why the internals should leak to what is shown to the user. You could keep the exact same user experience that you have today and guarantee that the site never falls over. You need to make the distinction between UX and Operations. Everything the user sees still lives under the same structure as today but internally it is structured so that you don't overload yourself and if you do fail, those failures are isolated and don't take down the rest of the site.
- cagenut 8y ago
  Everyone is reading the same thing on reddit, just different combinations of the same thing. this is categorically untrue, you don't seem to understand reddit. Reddit is just a combination of multiple newspapers masquerading as a single custom newspaper. that is exactly the anachronistic thinking that leads you down this path. its movable type vs wordpress all over again. wordpress won. the engineering constraint of pre-calculating caches is worse, in the long run, than the occasional outage for casual/non-logged-in users. its optimizing for the wrong audience, it is a long slow path to failure.

ecto 8y ago

We're taking a sort of hybrid approach to this on https://threadbase.io https://threadbase.io, where you can get a subdomain for your community (or custom domain in the future) and put your own ads in (so moderators can make money). Eventually we'll create a shared homepage for site creators who opt in, and allow you to filter communities you want to see there.