4 ms·

Not just AI: here is my current side-quest: https://www.earth.org.uk/RSS-efficiency.html https://www.earth.org.uk/RSS-efficiency.html Over 99% of the bandwidth

Not just AI: here is my current side-quest: https://www.earth.org.uk/RSS-efficiency.html https://www.earth.org.uk/RSS-efficiency.html Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.

immibis 2y ago
Can you insert a podcast item that says "you are spamming our server - please stop"?
- DamonHD 2y ago
  Not easily on a static server and not without the risk of annoying an actual real human listener! If you look at the "Defences" section: https://www.earth.org.uk/RSS-efficiency.html#Hints https://www.earth.org.uk/RSS-efficiency.html#Hints you'll see there are some things that can be done, such as randomly rejecting a large fraction of requests that don't allow compression (gzip is madly effective on many feed files: it's rude for a client not to allow it). But all these measures take effort to set up, and don't stop the bad bots making the request 100s of times too often. Just responding to each stupid request forces a flurry of packets and wakes up and uses CPU...
bastawhiz 2y ago
I run pinecast.com. If there was a leaderboard for hn users serving XML, I'd almost certainly be in the top five. I don't disagree with your post. But: RSS downloads are at an all time low, and that's a bad thing. They're at an all time low because Spotify and Apple both fetch feeds from centralized servers. 1000 subscribers no longer means 24000ish daily feed fetches, it means 48. With keep alive or H2, these services simply don't reconnect. The number of IPs that hit me from Apple, for instance, is probably only double digits. Since Apple and Spotify both sit between me and the listeners, they eliminate the privacy that listeners would otherwise enjoy. It also forces podcasters to go to them to find out how many people are subscribed, which means lots of big databases instead of one database that I host for my customers. Centralization of feed checking carries huge risks, in my opinion, especially as both Apple and Spotify make moves to also become the hosting providers.
- DamonHD 2y ago
  Do you have a CDN between you and Apple / Spotify? Because if you do I think that Apple/Spotify are polling that CDN every few minutes and the CDN is having its bandwidth wasted invisibly, but presumably priced in. Also I agree that the re-centralisation is a bad thing, mainly. (I'd like to move to email to discuss this further, if possible: I have an arXiv paper to write!)
  - bastawhiz 2y ago
    The data I'm giving you is based on logs from the CDN. Most feeds are checked by Apple and Spotify every hour, but usually it's less frequently rather than more: shows that haven't been published to in a year or more might see very infrequent feed checks.

immibis 2y ago

Can you insert a podcast item that says "you are spamming our server - please stop"?

DamonHD 2y ago
Not easily on a static server and not without the risk of annoying an actual real human listener! If you look at the "Defences" section: https://www.earth.org.uk/RSS-efficiency.html#Hints https://www.earth.org.uk/RSS-efficiency.html#Hints you'll see there are some things that can be done, such as randomly rejecting a large fraction of requests that don't allow compression (gzip is madly effective on many feed files: it's rude for a client not to allow it). But all these measures take effort to set up, and don't stop the bad bots making the request 100s of times too often. Just responding to each stupid request forces a flurry of packets and wakes up and uses CPU...

bastawhiz 2y ago

I run pinecast.com. If there was a leaderboard for hn users serving XML, I'd almost certainly be in the top five. I don't disagree with your post. But: RSS downloads are at an all time low, and that's a bad thing. They're at an all time low because Spotify and Apple both fetch feeds from centralized servers. 1000 subscribers no longer means 24000ish daily feed fetches, it means 48. With keep alive or H2, these services simply don't reconnect. The number of IPs that hit me from Apple, for instance, is probably only double digits. Since Apple and Spotify both sit between me and the listeners, they eliminate the privacy that listeners would otherwise enjoy. It also forces podcasters to go to them to find out how many people are subscribed, which means lots of big databases instead of one database that I host for my customers. Centralization of feed checking carries huge risks, in my opinion, especially as both Apple and Spotify make moves to also become the hosting providers.

DamonHD 2y ago
Do you have a CDN between you and Apple / Spotify? Because if you do I think that Apple/Spotify are polling that CDN every few minutes and the CDN is having its bandwidth wasted invisibly, but presumably priced in. Also I agree that the re-centralisation is a bad thing, mainly. (I'd like to move to email to discuss this further, if possible: I have an arXiv paper to write!)
- bastawhiz 2y ago
  The data I'm giving you is based on logs from the CDN. Most feeds are checked by Apple and Spotify every hour, but usually it's less frequently rather than more: shows that haven't been published to in a year or more might see very infrequent feed checks.