11 ms·
Lessons from Building and Scaling Reddit's Ad Serving Platform with Go
- deleted 8y ago[deleted]
- minxomat 8y ago> 12M posts per month, 2B posts per month Curious, which one is it? I suspect the former.
- alkonaut 8y ago2B must be ”pageviews per month”?
- benjaminjackman 8y agoMaybe 12M articles / self posts, 2B comments?
- overcast 8y agoNah, no way on average an article gets 166 comments. That must be page views or something.
- RandallBrown 8y agoWhy not? Most front page posts have several thousand posts, which seems like it might be enough to offset the 0 comments that most submissions will get.
- overcast 8y agoSo what are we talking about, 1% of 12 million articles gets a few thousand comments?
- simias 8y agoThe front page is obviously not representative in any way of 99% of the posts made on the platform. Look at the slew of posts made on reddit: https://www.reddit.com/r/all/new/ https://www.reddit.com/r/all/new/ Most will be lucking to reach 10 comments.
- gtaylor 8y agoThis is correct.
- attfarhan 8y agoLiveblogger here. This post was best-effort liveblogged while the talk was going on, so this typo was my fault. I've fixed it to say "2B votes per month", which reflects what was on the speaker's slides.
- freecodyx 8y agoi will add to not use the default http.Client/Server, poor performance, instead use fasthttp
- ainar-g 8y agoIIRC, fasthttp only supports a subset of HTTP. This may or may not be a deal-breaker for some people.
- ErikDub 8y agoIt serves everything that is required for an ad-server.
- acln 8y agoI like my HTTP package to actually speak as good an approximation of HTTP as it can, so fasthttp wouldn't be my first option. Furthermore, the HTTP stack doesn't show up on most web servers' profiles. net/http is better understood by the community, interoperates with libraries better (e.g. golang.org/x/oauth2), and is better supported. Use net/http.
- ErikDub 8y agoThe main issue with net/http is that it generates a lot of garbage which is definitely going to show up in your profile if you're handling a lot of requests per second.
- sagichmal 8y agoAllocation pressure imposed by net/http has never been a top K source of profile CPU burn in any service I’ve ever written, and I’ve written plenty of high-performance, high-QPS services. fasthttp is rarely if ever a good idea.
- kasey_junk 8y agoThis is terrible advice for something serving broadly to consumers. Fasthttp is great if you can limit your interactions to a minor testable subset of calls but otherwise its not suitable.
- somethingroma 8y agoCool. Maybe these guys should get in touch with the redesign guys and get them fired. Also fire any executive that thought it was a good idea. Because all the hard work to serve ads more efficiently goes to waste when the site is a mess and unusable.
- kokey 8y agoI remember when Digg was bigger than Reddit and then after a certain Digg site redesign they lost a significant number of their users to Reddit and never really recovered.
- kotojo 8y agoI feel like that would have happened this time if there were a viable alternative. I know if they kill the ability to opt out of the new redesign I will be looking a lot harder for one.
- jaipilot747 8y agoold.reddit.com or i.reddit.com on mobile.
- mcny 8y agoI don't have to do that when I'm signed in. It is in the preferences.
- jjoonathan 8y agoThe pref setting regularly breaks for me and many others. I had to install an extension to reliably get old reddit. Most won't bother.
- the_clarence 8y agoHasn’t reddit front page become really bad? When I joined there were a lot more textual posts, now it’s all images and memes. More 9gag-y. Maybe taking askscience and atheism and all these subreddits out of the FP was a really bad idea? I wish there was another category next to hot, all, new: an interesting one with a pre-selection of mostly textual subreddits
- pjc50 8y agoA means for rapidly selecting and loading an advert which will then be blocked by the users' browser. (Although even without adblock, are there actually all that many overt ads on Reddit, other than the sidebar ad which nearly always is an ad for a different subreddit?)
- rawoke083600 8y ago"rapidly blocked" :P
- ufo 8y agoThere are also promoted posts intermingled with user generated content. They are specially visible in the mobile app
- deleted 8y ago[deleted]
- jrockway 8y agoJudging from the types of screenshots Reddit users post, they're probably using the app, not a browser on their PC with an ad blocker.
- ezekg 8y agoUnfortunately, they're probably using the app because they got annoyed by how much the app is forced onto you while using the web version. I however, will not relent. I still use old.reddit.com, even on mobile, because the new React version is slow and constantly tries to get me to download the app so that they can show me ads.
- katzgrau 8y agoAs the founder of a SAAS ad serving company, I can say that as slick as (I think) my platform is, it's not a problem I would want to tackle again. There will be a large volume of technical challenges and required features to support. It'd be like writing Apache or Nginx from scratch. It seems simple on the surface and then you wake up to all kinds of inconvenient and unexpected technical challenges. I'm not sure Reddit's engineering resources are best allocated to building ad serving infrastructure and software. It was going to be an ongoing headache from the start, and worse, it sounds like it was build with a language and stack that was fairly new to the team. My prediction is that it will become a high cost and difficult to maintain project that will also prove difficult to hire for when this engineering team gets tired of working on the (currently new and fun, soon to be old and scary) adserver and moves on to other jobs. My point being that some challenges are best left to third parties who make solving some problem its full time job. I truly do not say that out of bias, just experience building and managing such a complex system.
- somethingroma 8y agoHonestly I think you make a good point and implicitly they could have used those resources instead to improve the user experience (for example, improve the sites efficiency and load times)
- jacquesm 8y agoThey could have gone with Iponweb[1] or something like that but at Reddit's scale it might actually (for a change) make sense to roll their own. They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit. The whole Reddit saga for the last few years is quite the story of turnaround management done well. [1] https://www.iponweb.com/ https://www.iponweb.com/
- katzgrau 8y agoYeah I was thinking that if what they need or are planning is truly unique or just needs to be homegrown (a la FB's ad system tightly integrated with first party data), maybe it really does make sense. Or maybe any alternative third party company they engaged tried to pitch them a huge/prohibitive contract. But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?
- deleted 8y ago[deleted]
- sonnyblarney 8y agoBrilliant, thanks for writing.
- bflesch 8y agoI advertised on reddit for our startup roughly one year ago and it was the worst ad management experience I have ever witnessed. For the size and age of reddit the advertiser backend was a real shame. The kind invoices they provide you is laughable, it feels like the whole backend was built by amateurs. On top of that, the traffic we received from reddit was very bad for our e-sports/gaming-related website.
- jamiequint 8y agoHey bflesch, a year ago we weren’t even running our own ad server and had just released an early v1 of our new ads management interface. It probably felt like the “worst ad management experience [you] have ever witnessed” because the other major ad management platforms have multiple years and hundreds of engineers/designers on us :) We’ve made substantial progress since then, I’d encourage you to not judge us too early and give it another shot, the platform and products are improving rapidly. p.s. If any engineers or PMs are interested in helping us build out a world class, high-scale ad platform please get in touch (email in profile). We are hiring for senior and above positions in NY and SF.
- supernovae 8y agoIf you work in the ad group, my advice is to fix the ad approval process. I've never had such a disappointing experience in my life. Seems like the approvers have editorial control based on personal preference over any objective evaluation. Since reddit is all about the ebs and flows of whats hot and whats not, sure is a shame your advertising platform is a joke at monetizing that because of the terrible approval process.
- jamiequint 8y agoThanks for the feedback! We are completely revamping the approval process over the next few months so I am expecting substantial improvement in this area.
- le-mark 8y agoNo remote? What is the justification for this?
- jdright 8y ago> Thrift has been around since 2007, and is one of the first rpc protocols. Done reading there. Not sure if bad written or just not knowledgeable, please do your research.
- orf 8y agoWhat's more likely: there is an error in transcribing or the developers of one of the worlds largest websites think the first RPC protocol was written in 2007 (over, say, REST, which their entire API is built on).
- Twirrim 8y agoGiven the continual attitudes seen within tech... sadly, the latter.
- kyrra 8y agoI'm not sure what the first RPC protocol was, but could it be the SunRPC[0]? I used that at my first job and thought it was pretty magical. EDIT: duh, search for "rpc" on wikipedia[1]. The first came a few years before SunRPC out of Xerox PARC. [0] https://en.wikipedia.org/wiki/Open_Network_Computing_Remote_Procedure_Call https://en.wikipedia.org/wiki/Open_Network_Computing_Remote_... [1] https://en.wikipedia.org/wiki/Remote_procedure_call https://en.wikipedia.org/wiki/Remote_procedure_call
- attfarhan 8y agoLiveblogger here. This post was best-effort liveblogged while the talk was going on, and this is an error transcribing the talk on my part. I've made a fix to remove this line since it's misleading. Sorry about that!
- tablethnuser 8y agoEnrichment Service, aka surveillance capitalism center
- cdnsteve 8y agoGo-kit looks pretty nice, anyone else have experience with it?
- theplague42 8y agoI've been using it for a few months, and it's pretty great.
- nouney 8y agoIt will lock you down, as it's very hard to migrate from go-kit. IMHO, it only brings problems.
- avdempsey 8y agoIt can be surprising how little code you're actually using from the Go-Kit project, and how it's really more of a structure for your app. It will not save you time typing: each route can have a transport request decoder and response encoder (encoder's are often shared), a request and response type, the actual business logic, a function that takes the request struct, applies it to the business logic function and returns a response object, a function that takes the previous function and connects it with the transport machinery (building the actual endpoint), and possibly middleware implementations. However, by teasing all of these things apart from the very beginning it's very easy to come back later and add a circuit-breaker, or (as the article says) switch your transport from HTTP to grpc (or support both).
- auslander 8y ago> Enrichment service is responsible for getting more data and information about the request, user ... Where do you get info about user and whats in it?
- HyperTalk2 8y agoAds on reddit are incredibly deceptive. I'm not sure why advertisers tolerate it. 98% of the ad clicks on mobile are surely fraudulent through being intentionally accidental. The ads are designed to look exactly the same as real posts at a glance, with a username, post date, upvote/downvote arrows, deceptive title meant to make it look like a real post, comment section, share link, and image thumbnail. Reddit users are all conditioned to believe that they can tap a thumbnail and be shown a larger preview while staying on the site. On these ads, it takes them to the advertiser's site instead. https://i.imgur.com/Vm68wLP.jpg https://i.imgur.com/Vm68wLP.jpg
- bduerst 8y agoHave you actually run ads on reddit, or are you saying this as a user? If so, what stats have seen (impressions, clicks, conversions, etc.) for your ads?
- tapirl 8y agoI agree with OP. My 99% clicks on reddit mobile ad are unintended. It is very annoying, for if I press the back button to early, then the firefox browser will back too the page previous to the page containing the ad I clicked. So generally, I will wait the ad fully loaded then press the back button.
- ma2rten 8y agoYou and OP might not be typical users.
- barbecue_sauce 8y ago* locked comment section
- bitwize 8y agoReddit has become the Something Awful forums without an entry fee. What did you expect? How else are advertisers supposed to capture that millennial and post-millennial dollar through a platform like that?
- acroback 8y agoHalf of the times it just sits there blank. Sometimes it loads in 2-3 seconds. Rest of the times it loads within 1 sec. Rarely, it also shows a default ad( I work in same industry, hence the term). How is that great? PS: Not talking about native ads.