5 ms·

As the founder of a SAAS ad serving company, I can say that as slick as (I think) my platform is, it's not a problem I would want to tackle again. There will be

As the founder of a SAAS ad serving company, I can say that as slick as (I think) my platform is, it's not a problem I would want to tackle again. There will be a large volume of technical challenges and required features to support. It'd be like writing Apache or Nginx from scratch. It seems simple on the surface and then you wake up to all kinds of inconvenient and unexpected technical challenges. I'm not sure Reddit's engineering resources are best allocated to building ad serving infrastructure and software. It was going to be an ongoing headache from the start, and worse, it sounds like it was build with a language and stack that was fairly new to the team. My prediction is that it will become a high cost and difficult to maintain project that will also prove difficult to hire for when this engineering team gets tired of working on the (currently new and fun, soon to be old and scary) adserver and moves on to other jobs. My point being that some challenges are best left to third parties who make solving some problem its full time job. I truly do not say that out of bias, just experience building and managing such a complex system.

somethingroma 8y ago
Honestly I think you make a good point and implicitly they could have used those resources instead to improve the user experience (for example, improve the sites efficiency and load times)
jacquesm 8y ago
They could have gone with Iponweb[1] or something like that but at Reddit's scale it might actually (for a change) make sense to roll their own. They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit. The whole Reddit saga for the last few years is quite the story of turnaround management done well. [1] https://www.iponweb.com/ https://www.iponweb.com/
- katzgrau 8y ago
  Yeah I was thinking that if what they need or are planning is truly unique or just needs to be homegrown (a la FB's ad system tightly integrated with first party data), maybe it really does make sense. Or maybe any alternative third party company they engaged tried to pitch them a huge/prohibitive contract. But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?
ckdarby 8y ago
It seems biased. I had a similar situation with a client of mine. They went out looked at 3rd party companies, and everything looked decent on paper. Internally there were some napkin math and found out while the 3rd parties would save development time the returns of implementing features their customers were asking for now instead of waiting for 3rd party company paid for them to hire 3x the engineering staff the 3rd party company even had. Size matters, Reddit is at the scale of tiny percentage gains, if they control everything and can drop a standard 90ms down to 10ms and that converts to a couple of percentage points on their revenue it is worth it. Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. Reporting and ad setup from the user is much more challenging.
- ChrisSD 8y ago
  Out of interest, how did those option compare with the napkin maths for paying a 3rd party company to include the needed features in their product?
- katzgrau 8y ago
  > if they control everything and can drop a standard 90ms down to 10ms 10ms, after request processing, ad selection logic, and faster-than-light packet travel? I think that's far too optimistic of an improvement. As the post said, 30ms was the goal. My adserver delivers in 20-40ms. But since ads are routinely loaded asynchronously, is speed concern truly valid? The adserver can usually make a selection and deliver before the content is fully rendered in the browser. > Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. If you developed a system that delivers 1BB impressions per day (most large scale ad networks deliver only a modest multiple of that number) and didn't find it challenging, then it's either a very simple system or you're Elon Musk ;) That's 12k requests per second, generally uncached, hitting all sorts of internal services most likely. I'm just saying that I think Reddit's best bet is to offload the task to a company that works on these problems full time. Or acquire one like AOL and Facebook. Unless the use cases are so unique and key to the company's future plans, avoid building in-house. And if you have to, perhaps it would be a larger effort than a 3 person team developing the thing in a language that the company never used in production before.
grey-area 8y ago
They were already using a third party service. It was slow and opaque, so they ditched it. At the scale of reddit, and because this is how they make their money, this is clearly something they want in house and something it would be a competitive advantage to control, if nothing else it gives them a much clearer conduit to their advertisers, and much greater control over things like ad selection.
- supernovae 8y ago
  The problem with Reddit ads isn't their tech, it's their people. The ad approval process is a farce - and human-managed. Oh, and those humans are assholes. You have to go through many denials - for no reason - submit a ticket to get another approver and approved - for no reason stated. It was the most upsetting process I've ever dealt with.
  - yourbandsucks 8y ago
    This is the QA process that yields us precocious-looking stock photo 'MIT grads' trying to sell me a wine subscription on every other pageload?
    - zeveb 8y ago
      reddit has ads? I hadn't noticed … More-pertinently, I wonder how many of the developers of reddit's new ad system themselves have ad blockers installed. Do any of them actually see the results of their work?
      - jedberg 8y ago
        FWIW, when I worked at reddit, I never ran an ad blocker. In fact, I didn't run one for about five years after, until my CPU started spiking on just about every web page I visited. But even then, I whitelisted reddit. So my guess is yes, they see the output of their work.
      - rbobby 8y ago
        Reddit ads were what finally pushed me into install ublock origin about a year ago (redditor since 2007). They had a misbehaving ad (downloaded a ton of "stuff" over and over and over). I reported it on one of the their support subreddits and was told "don't recognize the add urls. scan your system for a virus". It was a pretty good error report too (screen shots, console log, etc). Installed ublock origin that day and haven't looked back.

somethingroma 8y ago

Honestly I think you make a good point and implicitly they could have used those resources instead to improve the user experience (for example, improve the sites efficiency and load times)

jacquesm 8y ago

They could have gone with Iponweb[1] or something like that but at Reddit's scale it might actually (for a change) make sense to roll their own. They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit. The whole Reddit saga for the last few years is quite the story of turnaround management done well. [1] https://www.iponweb.com/ https://www.iponweb.com/

katzgrau 8y ago
Yeah I was thinking that if what they need or are planning is truly unique or just needs to be homegrown (a la FB's ad system tightly integrated with first party data), maybe it really does make sense. Or maybe any alternative third party company they engaged tried to pitch them a huge/prohibitive contract. But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?

ckdarby 8y ago

It seems biased. I had a similar situation with a client of mine. They went out looked at 3rd party companies, and everything looked decent on paper. Internally there were some napkin math and found out while the 3rd parties would save development time the returns of implementing features their customers were asking for now instead of waiting for 3rd party company paid for them to hire 3x the engineering staff the 3rd party company even had. Size matters, Reddit is at the scale of tiny percentage gains, if they control everything and can drop a standard 90ms down to 10ms and that converts to a couple of percentage points on their revenue it is worth it. Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. Reporting and ad setup from the user is much more challenging.

ChrisSD 8y ago
Out of interest, how did those option compare with the napkin maths for paying a 3rd party company to include the needed features in their product?
katzgrau 8y ago
> if they control everything and can drop a standard 90ms down to 10ms 10ms, after request processing, ad selection logic, and faster-than-light packet travel? I think that's far too optimistic of an improvement. As the post said, 30ms was the goal. My adserver delivers in 20-40ms. But since ads are routinely loaded asynchronously, is speed concern truly valid? The adserver can usually make a selection and deliver before the content is fully rendered in the browser. > Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. If you developed a system that delivers 1BB impressions per day (most large scale ad networks deliver only a modest multiple of that number) and didn't find it challenging, then it's either a very simple system or you're Elon Musk ;) That's 12k requests per second, generally uncached, hitting all sorts of internal services most likely. I'm just saying that I think Reddit's best bet is to offload the task to a company that works on these problems full time. Or acquire one like AOL and Facebook. Unless the use cases are so unique and key to the company's future plans, avoid building in-house. And if you have to, perhaps it would be a larger effort than a 3 person team developing the thing in a language that the company never used in production before.

grey-area 8y ago

They were already using a third party service. It was slow and opaque, so they ditched it. At the scale of reddit, and because this is how they make their money, this is clearly something they want in house and something it would be a competitive advantage to control, if nothing else it gives them a much clearer conduit to their advertisers, and much greater control over things like ad selection.

supernovae 8y ago
The problem with Reddit ads isn't their tech, it's their people. The ad approval process is a farce - and human-managed. Oh, and those humans are assholes. You have to go through many denials - for no reason - submit a ticket to get another approver and approved - for no reason stated. It was the most upsetting process I've ever dealt with.
- yourbandsucks 8y ago
  This is the QA process that yields us precocious-looking stock photo 'MIT grads' trying to sell me a wine subscription on every other pageload?
  - zeveb 8y ago
    reddit has ads? I hadn't noticed … More-pertinently, I wonder how many of the developers of reddit's new ad system themselves have ad blockers installed. Do any of them actually see the results of their work?
    - jedberg 8y ago
      FWIW, when I worked at reddit, I never ran an ad blocker. In fact, I didn't run one for about five years after, until my CPU started spiking on just about every web page I visited. But even then, I whitelisted reddit. So my guess is yes, they see the output of their work.
    - rbobby 8y ago
      Reddit ads were what finally pushed me into install ublock origin about a year ago (redditor since 2007). They had a misbehaving ad (downloaded a ton of "stuff" over and over and over). I reported it on one of the their support subreddits and was told "don't recognize the add urls. scan your system for a virus". It was a pretty good error report too (screen shots, console log, etc). Installed ublock origin that day and haven't looked back.