3 ms·

"Under the hood, Apple stopped having each device request a copy of RSS feeds." Yes, because this was creating DDoS levels of traffic to podcast hosting provid

by mikece 4y ago

"Under the hood, Apple stopped having each device request a copy of RSS feeds." Yes, because this was creating DDoS levels of traffic to podcast hosting providers. Now Apple simply pings madly away at all known podcast RSS feeds and updates clients when there is a new episode. What would be far greener -- and efficient! -- would be for them to get on board with PodPing[1] to not only deliver feed updates much faster but to use a tiny fraction of the resources they are using now to get it done. [1] https://podcasting20.substack.com/p/podping https://podcasting20.substack.com/p/podping

bastawhiz 4y ago
> creating DDoS levels of traffic Author here. This has never been a problem, both in terms of volume and cost. I host ~0.25% (maybe more? I haven't checked recently) of all podcasts listed on Apple and up to a few years ago, I wasn't even using a CDN. Two Heroku dynos at ~$250/mo (25 customer subscriptions today) running a Python back-end with no caching at all was able to keep up without trouble. In fact, the only reason I added a CDN was Heroku's infrastructure having spooky issues with that volume of traffic. Podcast hosting providers larger than me running on their own hardware should have (had) exactly zero trouble.
- georgel 4y ago
  The entirety of all the podcasts on iTunes/Apple Podcasts RSS feeds downloaded is around 200-250GB across the 2.5M+ pods. Hosting the RSS feed should be an easy task. Add in caching via ETag and server-side (host the feed as a static file, only re-render when user updates something), and your server load goes down drastically. It gets a bit complex if your hosting company supports dynamic audio/ad insertion depending on how you accomplish that, but as far as retrieving the RSS XML feed, that has no impact.
  - bastawhiz 4y ago
    Audio insertion is actually straightforward: the audio URLs from the feed can redirect!
- ryanbrunner 4y ago
  I agree with this. We actually dynamically render RSS feeds (we have some personalization options) and while we've had to optimize it here and there, it doesn't even register as a real cost or problem (streaming MP3 files to listeners is the real cost of hosting podcasts).
djl0 4y ago
I'm interested to see this podping service, however given the whole model of RSS is that the end-user is requesting feed updates (I would guess daily on average, but maybe hourly?), I have a hard time thinking this was a problem for podcast providers. I have no knowledge of the implementation at scale, but given the feed is static for the most part, wouldn't podcast hosting providers want their users constantly checking in?
- mikece 4y ago
  Depends on the podcast and the users. If I know my favorite afternoon commute podcast drops at 4:30 PM local and I don't see it by 4:45 PM, I'm going to be doing the pull-to-refresh thing frequently, hoping it's available to download before I leave the WiFi at work for my drive home. I suppose Podping plus push notification to all app installs subscribed to the podcast that just updated would be another way to do it.
  - layer8 4y ago
    > I'm going to be doing the pull-to-refresh thing frequently Such refreshs would be using If-Modified-Since, so shouldn’t cause substantial traffic.
whywhywhywhy 4y ago
Would podcast hosting providers rather get an engineer that knows how to handle that traffic and get all the tracking and ID data of a device that they can communicate to their advertisers or have Apple make their lives easier but their service less valuable? Apple reality distortion to claim this is positive for them.
ilyt 4y ago
Why the hell that stupid thing would need blockchain for?
- kevincox 4y ago
  Yeah. WebSub works great. The only listed reason is that subscriptions only last 15 days which isn't true. There is no hard limit in the protocol, it just depends on the hub and clients limits.

by mikece 4y ago

bastawhiz 4y ago

> creating DDoS levels of traffic Author here. This has never been a problem, both in terms of volume and cost. I host ~0.25% (maybe more? I haven't checked recently) of all podcasts listed on Apple and up to a few years ago, I wasn't even using a CDN. Two Heroku dynos at ~$250/mo (25 customer subscriptions today) running a Python back-end with no caching at all was able to keep up without trouble. In fact, the only reason I added a CDN was Heroku's infrastructure having spooky issues with that volume of traffic. Podcast hosting providers larger than me running on their own hardware should have (had) exactly zero trouble.

georgel 4y ago
The entirety of all the podcasts on iTunes/Apple Podcasts RSS feeds downloaded is around 200-250GB across the 2.5M+ pods. Hosting the RSS feed should be an easy task. Add in caching via ETag and server-side (host the feed as a static file, only re-render when user updates something), and your server load goes down drastically. It gets a bit complex if your hosting company supports dynamic audio/ad insertion depending on how you accomplish that, but as far as retrieving the RSS XML feed, that has no impact.
- bastawhiz 4y ago
  Audio insertion is actually straightforward: the audio URLs from the feed can redirect!
ryanbrunner 4y ago
I agree with this. We actually dynamically render RSS feeds (we have some personalization options) and while we've had to optimize it here and there, it doesn't even register as a real cost or problem (streaming MP3 files to listeners is the real cost of hosting podcasts).

djl0 4y ago

I'm interested to see this podping service, however given the whole model of RSS is that the end-user is requesting feed updates (I would guess daily on average, but maybe hourly?), I have a hard time thinking this was a problem for podcast providers. I have no knowledge of the implementation at scale, but given the feed is static for the most part, wouldn't podcast hosting providers want their users constantly checking in?

mikece 4y ago
Depends on the podcast and the users. If I know my favorite afternoon commute podcast drops at 4:30 PM local and I don't see it by 4:45 PM, I'm going to be doing the pull-to-refresh thing frequently, hoping it's available to download before I leave the WiFi at work for my drive home. I suppose Podping plus push notification to all app installs subscribed to the podcast that just updated would be another way to do it.
- layer8 4y ago
  > I'm going to be doing the pull-to-refresh thing frequently Such refreshs would be using If-Modified-Since, so shouldn’t cause substantial traffic.

whywhywhywhy 4y ago

Would podcast hosting providers rather get an engineer that knows how to handle that traffic and get all the tracking and ID data of a device that they can communicate to their advertisers or have Apple make their lives easier but their service less valuable? Apple reality distortion to claim this is positive for them.

ilyt 4y ago

Why the hell that stupid thing would need blockchain for?

kevincox 4y ago
Yeah. WebSub works great. The only listed reason is that subscriptions only last 15 days which isn't true. There is no hard limit in the protocol, it just depends on the hub and clients limits.