5 ms·

I tend to read my RSS feeds on the London Underground, where signal is non existent in the tunnels, and patchy in the stations. My app of choice (newsblur) lets

I tend to read my RSS feeds on the London Underground, where signal is non existent in the tunnels, and patchy in the stations. My app of choice (newsblur) lets you sync your feeds for offline browsing so I do this before heading home. Unfortunately a lot of feeds I subscribe to tend to truncate their articles with a summary or maybe just the first paragraph, with a link to continue reading the full article. I get why they do this, getting people onto the site gives a greater opportunity to show ads etc, sites gotta be funded somehow. A few years ago I wrote a tool that ingests these truncated RSS feeds, visits each new link in the feed, and then uses a text extraction library to pull out the full text. These full text articles are then supplied as a brand new RSS feed for my client to subscribe to. I felt really guilty about it though and wasn't sure about the legal implications of it

MapleWalnut 7y ago
That's a great idea. I would definitely use a service like that.
- djhworld 7y ago
  I think other people in this thread are running services that do a similar thing. My crisis of conscience was around storing the full text of the articles on my own server and then presenting them as a new feed on my domain, e.g. https://mydomain.com/feeds/eurogamer.xml https://mydomain.com/feeds/eurogamer.xml Everything in the feed is that gets generated is exactly the same as the original feed (canonical link to the actual article, publsh date, author etc) the only difference was the text in the body of each entry.
  - WiseWeasel 7y ago
    It's likely that unless you can claim status as a library or some other exception to copyright protections, as a commercial entity providing that service without compensating publishers, such a straightforward violation would get a bunch of takedown requests and eventual lawsuits for failure to respond to those requests, along with more suits if publishers have no easy way of opting out. As a personal tool you built for yourself and your friend(s) nerdy enough to understand it, I wouldn’t worry too much about it, as long as no search engines index your site and no one finds out about it.
- k1m 7y ago
  Posted this in another reply, it's a project I've worked on for a while, works essentially the way djhworld describes: https://fivefilters.org/content-only/ https://fivefilters.org/content-only/ Source code of a slightly older version available at: https://bitbucket.org/fivefilters/full-text-rss/src/master/ https://bitbucket.org/fivefilters/full-text-rss/src/master/
- pvorb 7y ago
  And others would definitely sue a service like that. (I'd use it, too, but I understand why it won't happen.)
AndrewStephens 7y ago
Sorry, I do this (provide only truncated text) in my RSS feed because I couldn't figure out a way of keeping the special formatting my site is designed around. Its a cop out that I still feel bad about. Maybe I will fix it sometime. If it makes you feel better, you can run your extractor against sheep.horse as much as you want.
about_help 7y ago
Could you use a full web engine to scrape the info which would also load the ads?
danShumway 7y ago
As someone who hand-codes their RSS feeds and doesn't include article content at all (just a title/link/description), what's the recommended setup for this? Can I just spit the raw HTML at you? Do I need to inline images? I'm assuming that someone who's consuming a full feed isn't going to want any CSS included. I'm not against this, but I'm also hand-coding all of my blogs, and this seems like a problem that would be more easily and more consistently solved by a user-run service like the one you've made. I don't know what readers currently support, and I'm looking around quickly on DuckDuckGo and not seeing many resources about generating full-text feeds that don't boil down to, "click the button in Wordpress." Again, I have no problem supporting this, but if I'm going to put the time in to change my blogs, I kind of need more prodding than just one or two people saying it annoys them. I don't have any specific reason I wouldn't want to accommodate people like you -- I'm not advertising on my blogs and I don't have any analytics, so I don't care where you read. But I don't want to invest a lot of time into building a completely new content pipeline.
- novok 7y ago
  You can just put raw html and normal img tags. Just think of it as an unstyled version of your normal website, just rendering the article portion. Also look at other pages’ rss feeds that include full articles.
- AndrewStephens 7y ago
  > I'm not advertising on my blogs Perhaps you should, I enjoyed finding stuff like this : https://danshumway.com/blog/gamasutra-vulnerabilities/ https://danshumway.com/blog/gamasutra-vulnerabilities/
  - saagarjha 7y ago
    Perhaps they enjoy not having their website marred by online advertisements?
  - danShumway 7y ago
    I'm really glad you enjoyed the article :) I realize that there are some ways that people serve ads where they vet them and make sure they're not tracking readers. But even if it was easy to set something like that up, I wouldn't feel good running sponsored content or ads on my platforms -- it's not the relationship I want to have with my readers. I take a much more negative view of ads than most other people I know. I don't just dislike ads that track users or that are poorly vetted. I dislike the psychological effect that ads have on people, I dislike the attention that they take, and I am very cautious of monetization schemes that are too closely integrated into the consumption of content. I can't think of a way I could run ads that wouldn't make reading my blog a worse experience. I know that's a somewhat extreme position to take. But it's not that advertising is inherently evil, it's just that it has more consequences and costs than most people realize. It's not only about privacy, security, or money. Given how large the current advertising industry is, where possible I try to block it and harm it indiscriminately. I encourage people to block ads even on websites that they would otherwise support. I think it would be good for society if the advertising industry got a lot smaller, even if that makes funding online content harder.
rchaud 7y ago
Sounds like an automated version of Pocket, which has been around for ~10 years and is now owned by Mozilla; so the legality aspect of it shouldn't be an issue. I use it daily, although recent versions of the app send you to the original article page, ads and all, should you dare to open it with data/wifi active. Airplane mode still pulls up a text-and-image only version of the article.
- deleted 7y ago
  [deleted]
Wowfunhappy 7y ago
FiveFilters.org does this: https://fivefilters.org/content-only/ https://fivefilters.org/content-only/ The free version has some limitations, like only three articles per rss feed. They offer a self-hosted version for a one-time 35euro fee (although software updates only for one year), which I really appreciate.
jeena 7y ago
I also like reading in my feed reader, which is basically like a automatic offline "reader view" with automatic "read this article" marks. Because of that I quite early decided to never subscribe to a feed which doesn't give me the full article. So I don't ever have this problem. I basically feel that there is so much good content to read out there which I can get delivered to my rss reader directly that I don't really need to jump though the hoops and clicking on 'read more' links and adapting to badly designed websites with really small fonts and gray on white text colors.
robk 7y ago
I do this with huginn
kickscondor 7y ago
Rather than extracting the text, could you just render a cached copy of the full web page? I don't see how anyone would have a problem with that. Many times you will end up missing parts of the article anyway if you try to pare it down.
deleted 7y ago
[deleted]

MapleWalnut 7y ago

That's a great idea. I would definitely use a service like that.

djhworld 7y ago
I think other people in this thread are running services that do a similar thing. My crisis of conscience was around storing the full text of the articles on my own server and then presenting them as a new feed on my domain, e.g. https://mydomain.com/feeds/eurogamer.xml https://mydomain.com/feeds/eurogamer.xml Everything in the feed is that gets generated is exactly the same as the original feed (canonical link to the actual article, publsh date, author etc) the only difference was the text in the body of each entry.
- WiseWeasel 7y ago
  It's likely that unless you can claim status as a library or some other exception to copyright protections, as a commercial entity providing that service without compensating publishers, such a straightforward violation would get a bunch of takedown requests and eventual lawsuits for failure to respond to those requests, along with more suits if publishers have no easy way of opting out. As a personal tool you built for yourself and your friend(s) nerdy enough to understand it, I wouldn’t worry too much about it, as long as no search engines index your site and no one finds out about it.
k1m 7y ago
Posted this in another reply, it's a project I've worked on for a while, works essentially the way djhworld describes: https://fivefilters.org/content-only/ https://fivefilters.org/content-only/ Source code of a slightly older version available at: https://bitbucket.org/fivefilters/full-text-rss/src/master/ https://bitbucket.org/fivefilters/full-text-rss/src/master/
pvorb 7y ago
And others would definitely sue a service like that. (I'd use it, too, but I understand why it won't happen.)

AndrewStephens 7y ago

Sorry, I do this (provide only truncated text) in my RSS feed because I couldn't figure out a way of keeping the special formatting my site is designed around. Its a cop out that I still feel bad about. Maybe I will fix it sometime. If it makes you feel better, you can run your extractor against sheep.horse as much as you want.

about_help 7y ago

Could you use a full web engine to scrape the info which would also load the ads?

danShumway 7y ago

As someone who hand-codes their RSS feeds and doesn't include article content at all (just a title/link/description), what's the recommended setup for this? Can I just spit the raw HTML at you? Do I need to inline images? I'm assuming that someone who's consuming a full feed isn't going to want any CSS included. I'm not against this, but I'm also hand-coding all of my blogs, and this seems like a problem that would be more easily and more consistently solved by a user-run service like the one you've made. I don't know what readers currently support, and I'm looking around quickly on DuckDuckGo and not seeing many resources about generating full-text feeds that don't boil down to, "click the button in Wordpress." Again, I have no problem supporting this, but if I'm going to put the time in to change my blogs, I kind of need more prodding than just one or two people saying it annoys them. I don't have any specific reason I wouldn't want to accommodate people like you -- I'm not advertising on my blogs and I don't have any analytics, so I don't care where you read. But I don't want to invest a lot of time into building a completely new content pipeline.

novok 7y ago
You can just put raw html and normal img tags. Just think of it as an unstyled version of your normal website, just rendering the article portion. Also look at other pages’ rss feeds that include full articles.
AndrewStephens 7y ago
> I'm not advertising on my blogs Perhaps you should, I enjoyed finding stuff like this : https://danshumway.com/blog/gamasutra-vulnerabilities/ https://danshumway.com/blog/gamasutra-vulnerabilities/
- saagarjha 7y ago
  Perhaps they enjoy not having their website marred by online advertisements?
- danShumway 7y ago
  I'm really glad you enjoyed the article :) I realize that there are some ways that people serve ads where they vet them and make sure they're not tracking readers. But even if it was easy to set something like that up, I wouldn't feel good running sponsored content or ads on my platforms -- it's not the relationship I want to have with my readers. I take a much more negative view of ads than most other people I know. I don't just dislike ads that track users or that are poorly vetted. I dislike the psychological effect that ads have on people, I dislike the attention that they take, and I am very cautious of monetization schemes that are too closely integrated into the consumption of content. I can't think of a way I could run ads that wouldn't make reading my blog a worse experience. I know that's a somewhat extreme position to take. But it's not that advertising is inherently evil, it's just that it has more consequences and costs than most people realize. It's not only about privacy, security, or money. Given how large the current advertising industry is, where possible I try to block it and harm it indiscriminately. I encourage people to block ads even on websites that they would otherwise support. I think it would be good for society if the advertising industry got a lot smaller, even if that makes funding online content harder.

rchaud 7y ago

Sounds like an automated version of Pocket, which has been around for ~10 years and is now owned by Mozilla; so the legality aspect of it shouldn't be an issue. I use it daily, although recent versions of the app send you to the original article page, ads and all, should you dare to open it with data/wifi active. Airplane mode still pulls up a text-and-image only version of the article.

deleted 7y ago
[deleted]

Wowfunhappy 7y ago

FiveFilters.org does this: https://fivefilters.org/content-only/ https://fivefilters.org/content-only/ The free version has some limitations, like only three articles per rss feed. They offer a self-hosted version for a one-time 35euro fee (although software updates only for one year), which I really appreciate.

jeena 7y ago

I also like reading in my feed reader, which is basically like a automatic offline "reader view" with automatic "read this article" marks. Because of that I quite early decided to never subscribe to a feed which doesn't give me the full article. So I don't ever have this problem. I basically feel that there is so much good content to read out there which I can get delivered to my rss reader directly that I don't really need to jump though the hoops and clicking on 'read more' links and adapting to badly designed websites with really small fonts and gray on white text colors.

robk 7y ago

I do this with huginn

kickscondor 7y ago

Rather than extracting the text, could you just render a cached copy of the full web page? I don't see how anyone would have a problem with that. Many times you will end up missing parts of the article anyway if you try to pare it down.

deleted 7y ago

[deleted]