4 ms·

Hacker News RSS Feed with Full Text Articles

32 points by talison 17y ago

knieveltech 17y ago
Counting the seconds until someone flings a takedown order. Aggregating full content without permission definitely exceeds the bounds of "fair use".
- jncraton 17y ago
  It's been working at least for a few months without getting shut down.
  - knieveltech 17y ago
    So it's under the radar. That doesn't change the fact that aggregating full content without permission exceeds the bounds of "fair use".
    - akkartik 17y ago
      How is this different from their own RSS feed? I think it's just saving them bandwidth.
      - petercooper 17y ago
        It might not feature their advertising, their FeedFlare (if on FeedBurner - a great way to drive people to comments sections), special formatting, or tracking graphics that they use for statistics. Before the inevitable "But I don't care!" responses, whether or not these things are interesting to us as readers is not relevant - they're still differences that could cause publishers to be unhappy and cry foul. On the plus side, since the HN news is just an aggregation of many sources, it could encourage people to find new sources and subscribe to them directly in the long run. As a content provider myself, I'd be happy with this aggregator if it was indicated in each post what the original source is.. if that's missing, it needs to be added. We want attribution even if we're being syndicated ;-)
    - jncraton 17y ago
      I agree completely that the service is infringing on the intellectual property of the authors of stories that HN links to. I just don't think that it is likely to get shut down, because, as you said, it is under the radar.
- anigbrowl 17y ago
  By whom? Nirmal, who created the tool, but whose work could be easily copied? Google, who have no control over which feeds people choose to subscribe to? Me, for just installing it? Actually, I might well prefer just the first paragraph and/or the top 3 three comments - I don't know yet whether I want the full articles. However, I don't think it's as cut-and-dried as you say. These debates will continue until online publishers come up with a convenient and affordable micropayment system. I'm not opposed to advertising per se, but I am against visual the clutter it often entails. And sorry to HN designers, but the same applies to a lot of websites. Less is more.
  - carbon8 17y ago
    "By whom?" Obviously by anyone hosting the feed for republishing full content without copy permission. I'm not happy with the current trend in copyright law, either, but it is was it is. "I don't think it's as cut-and-dried as you say." It really is. Edit: your downvotes and personal opinions won't stop you from getting takedown notices and losing lawsuits. Ignore the legal reality at your own peril.
    - anigbrowl 17y ago
      Obviously by Nirmal [...] Really? But Nirmal has no way of knowing what articles people will see fit to post to HN. How can he negotiate permission with all possible content providers, given that I may link tomorrow to a brand new blog created later today? I question whether he scan truly be said to be publishing, when it appears that the feed in question simply reformats without asserting any ownership over the content or monetizing it in any way, and involves no editorial selection on his part, something I consider a defining feature of publication.
      - carbon8 17y ago
        "Really?" Yes, really. And it's necessary to have a more thorough understanding of the current issues in copyright law and recognize that it's based on statutes and case law before speculating about what may or may not violate fair use. Words also already have definitions (eg, http://www.answers.com/publication http://www.answers.com/publication and http://www.answers.com/publish http://www.answers.com/publish) and, in a legal setting, those are the definitions you'll need to work with whether or not you have your own. I realize that this sounds somewhat ungracious (sorry! I mean no harm), but it's a legal issue, not a purely speculative philosophical issue.
- RiderOfGiraffes 17y ago
  For what it's worth I agree with you, but that also makes me wonder how it is that scribd can copy an entire copyrighted article without asking permission, and then say that to take it down they require a request that "meets all criteria for validity as a DMCA affidavit." That seems wrong to me. If it's copyrighted, and they copy it the entire thing without permission, how is that not wrong? EDIT: I see this has been discussed before many times, but this time I ask - scribd are apparently effectively immune because users upload things to them. They then claim that if it's copyrighted, it's not their fault. HN has uploaded content to them without the author's permission. Does that make HN a copyright violator?
  - carbon8 17y ago
    HN only publishes links and, often but not always, headlines. Most publishers are OK with this, but not all. Recently the AP has reaffirmed that they consider copying headlines to be copyright violations and are threatening to again go after sites (notably google).
  - qeorge 17y ago
    IANAL, but here's my understanding of the DMCA's Safe Harbour provision, which is why scribd and other UGC sites (e.g. YouTube, isohunt, flickr) aren't held responsible for copyrighted material appearing on their sites. UGC websites can be protected by the Safe Harbour provision under the following conditions: - The site provides a legal, legitimate service (i.e. fair use) - The amount of UGC on the site is too much to reasonably police - They have an employee registered with the US govt as the agent for handling all copyright claims. Simply providing an email address, contact form, or even a mailing address doesn't count. - When they receive legitimate removal requests from the copyright holders they remove the content in a timely manner. As long as these conditions are met they can't be held liable for copyright violations which occur through the use of their product. HN is a different case, because the amount of content copied from the source site (the headline, and only in some cases) is a small portion of the original work. Furthermore, HN is adding value to the content through ranking and community, which makes it a distinct piece of intellectual property. Similarly, you can legally make a collage which contains copyrighted material because the collage itself is a distinct entity (more than the sum of its parts). How much of the page content can scraped and presented, or how large the pieces of a collage can be, is vividly debated. For me personally, its a bit of a Potter Stewart situation, and this feed is absolutely copyright infringement.
dawie 17y ago
I love it! I have been looking for a RSS feed like this for quite a while now. Good Job!
robotrout 17y ago
I love that you did this. Good work. I intend to use it, and look at the code as well. It does raise ethical concerns. Obviously, the folks aren't selling any advertising, if we're accessing their site content another way... Which, as you correctly state, is exactly what RSS already allows. I've never understood why people put RSS feeds on their blogs. It's insanity. Give people a way to NOT come to my site and view my ads. Fine if ads are not your model, but for 95% of blogs, ads are the model. RSS is a fad, and eventually, saner heads will prevail. Until that time comes, as you say, you're presenting the same info that an RSS feed does, but with the benefit of having been ranked and vetted by the HN community first.
- Jebdm 17y ago
  RSS may go away, but something better will take its place. People put RSS on their blogs for multiple reasons. I use it on mine because it greatly increases the usability of my site. Another way it helps is that it makes sites "stickier"; without RSS, people would not have something automatically telling them "Hey, that site you liked updated! Go check it out!". They would have to either remember the URL, or remember a link path to the site, or bookmark it, and they'd have to periodically check back. (Not that I particularly care—while it is nice to have readers, I write mainly for my own amusement, and my site is not commercialized at all.) And I think that these two factors are a net win for most sites, whether commercial or not. There are ways around the advertising issue—offer truncated feeds, put ads in your feeds, provoke comments, etc. Plus, I thought that the known wisdom was that regular readers don't use ads much, it's the ones that stumble onto your site from Google, etc.? I haven't seen a study, though. Also, for the record, I basically don't check sites that don't offer RSS. If it's particularly good, it'll get bookmarked, and maybe I'll come back to it in a few months when I've got some free time. But most sites don't even get bookmarked. On the other hand, if I find a site that had a good article, I'll often add it to my RSS reader on a trial basis.
trezor 17y ago
Either it is a bug in Firefox 3.5 RC or the mime-type provided is wrong. It just renders like text, and not like an RSS-feed I can just click to subscribe to. Since RSS feeds seems to work on other sites, I'll assume the problem is with the feed. Correct mimetype should be application/rss+xml. Confirmed: $ wget -O /dev/null http://nirmalpatel.com/fcgi/hn_feed.fcgi --2009-06-22 21:01:56-- http://nirmalpatel.com/fcgi/hn_feed.fcgi Resolving nirmalpatel.com... 207.210.105.86 Connecting to nirmalpatel.com|207.210.105.86|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 211754 (207K) [text/plain] Saving to: `/dev/null' 100%[==================================================>] 211,754 254K/s in 0.8s 2009-06-22 21:01:57 (254 KB/s) - `/dev/null' saved [211754/211754]
talison 17y ago
I didn't realize this had been submitted before (by nirmal). It was hosted on a different domain in the previous submission which is why this wasn't caught as a dupe. Anyway, you can see some of the author's comments in this thread: http://news.ycombinator.com/item?id=542334 http://news.ycombinator.com/item?id=542334
jjames 17y ago
I brought up some concerns about full content scraping last time this was discussed: http://news.ycombinator.com/item?id=542552 http://news.ycombinator.com/item?id=542552 I used the feed for a few months but found the feed-readerized articles lifeless, frozen in time and free of discussion. I'm also not a fan of the title-only RSS feed. Some sites simply lose too much in feed form imo.
schwanksta 17y ago
Thanks, I was looking for this the other day!

32 points by talison 17y ago

Counting the seconds until someone flings a takedown order. Aggregating full content without permission definitely exceeds the bounds of "fair use".

jncraton 17y ago
It's been working at least for a few months without getting shut down.
- knieveltech 17y ago
  So it's under the radar. That doesn't change the fact that aggregating full content without permission exceeds the bounds of "fair use".
  - akkartik 17y ago
    How is this different from their own RSS feed? I think it's just saving them bandwidth.
    - petercooper 17y ago
      It might not feature their advertising, their FeedFlare (if on FeedBurner - a great way to drive people to comments sections), special formatting, or tracking graphics that they use for statistics. Before the inevitable "But I don't care!" responses, whether or not these things are interesting to us as readers is not relevant - they're still differences that could cause publishers to be unhappy and cry foul. On the plus side, since the HN news is just an aggregation of many sources, it could encourage people to find new sources and subscribe to them directly in the long run. As a content provider myself, I'd be happy with this aggregator if it was indicated in each post what the original source is.. if that's missing, it needs to be added. We want attribution even if we're being syndicated ;-)
  - jncraton 17y ago
    I agree completely that the service is infringing on the intellectual property of the authors of stories that HN links to. I just don't think that it is likely to get shut down, because, as you said, it is under the radar.
anigbrowl 17y ago
By whom? Nirmal, who created the tool, but whose work could be easily copied? Google, who have no control over which feeds people choose to subscribe to? Me, for just installing it? Actually, I might well prefer just the first paragraph and/or the top 3 three comments - I don't know yet whether I want the full articles. However, I don't think it's as cut-and-dried as you say. These debates will continue until online publishers come up with a convenient and affordable micropayment system. I'm not opposed to advertising per se, but I am against visual the clutter it often entails. And sorry to HN designers, but the same applies to a lot of websites. Less is more.
- carbon8 17y ago
  "By whom?" Obviously by anyone hosting the feed for republishing full content without copy permission. I'm not happy with the current trend in copyright law, either, but it is was it is. "I don't think it's as cut-and-dried as you say." It really is. Edit: your downvotes and personal opinions won't stop you from getting takedown notices and losing lawsuits. Ignore the legal reality at your own peril.
  - anigbrowl 17y ago
    Obviously by Nirmal [...] Really? But Nirmal has no way of knowing what articles people will see fit to post to HN. How can he negotiate permission with all possible content providers, given that I may link tomorrow to a brand new blog created later today? I question whether he scan truly be said to be publishing, when it appears that the feed in question simply reformats without asserting any ownership over the content or monetizing it in any way, and involves no editorial selection on his part, something I consider a defining feature of publication.
    - carbon8 17y ago
      "Really?" Yes, really. And it's necessary to have a more thorough understanding of the current issues in copyright law and recognize that it's based on statutes and case law before speculating about what may or may not violate fair use. Words also already have definitions (eg, http://www.answers.com/publication http://www.answers.com/publication and http://www.answers.com/publish http://www.answers.com/publish) and, in a legal setting, those are the definitions you'll need to work with whether or not you have your own. I realize that this sounds somewhat ungracious (sorry! I mean no harm), but it's a legal issue, not a purely speculative philosophical issue.
RiderOfGiraffes 17y ago
For what it's worth I agree with you, but that also makes me wonder how it is that scribd can copy an entire copyrighted article without asking permission, and then say that to take it down they require a request that "meets all criteria for validity as a DMCA affidavit." That seems wrong to me. If it's copyrighted, and they copy it the entire thing without permission, how is that not wrong? EDIT: I see this has been discussed before many times, but this time I ask - scribd are apparently effectively immune because users upload things to them. They then claim that if it's copyrighted, it's not their fault. HN has uploaded content to them without the author's permission. Does that make HN a copyright violator?
- carbon8 17y ago
  HN only publishes links and, often but not always, headlines. Most publishers are OK with this, but not all. Recently the AP has reaffirmed that they consider copying headlines to be copyright violations and are threatening to again go after sites (notably google).
- qeorge 17y ago
  IANAL, but here's my understanding of the DMCA's Safe Harbour provision, which is why scribd and other UGC sites (e.g. YouTube, isohunt, flickr) aren't held responsible for copyrighted material appearing on their sites. UGC websites can be protected by the Safe Harbour provision under the following conditions: - The site provides a legal, legitimate service (i.e. fair use) - The amount of UGC on the site is too much to reasonably police - They have an employee registered with the US govt as the agent for handling all copyright claims. Simply providing an email address, contact form, or even a mailing address doesn't count. - When they receive legitimate removal requests from the copyright holders they remove the content in a timely manner. As long as these conditions are met they can't be held liable for copyright violations which occur through the use of their product. HN is a different case, because the amount of content copied from the source site (the headline, and only in some cases) is a small portion of the original work. Furthermore, HN is adding value to the content through ranking and community, which makes it a distinct piece of intellectual property. Similarly, you can legally make a collage which contains copyrighted material because the collage itself is a distinct entity (more than the sum of its parts). How much of the page content can scraped and presented, or how large the pieces of a collage can be, is vividly debated. For me personally, its a bit of a Potter Stewart situation, and this feed is absolutely copyright infringement.

dawie 17y ago

I love it! I have been looking for a RSS feed like this for quite a while now. Good Job!

robotrout 17y ago

I love that you did this. Good work. I intend to use it, and look at the code as well. It does raise ethical concerns. Obviously, the folks aren't selling any advertising, if we're accessing their site content another way... Which, as you correctly state, is exactly what RSS already allows. I've never understood why people put RSS feeds on their blogs. It's insanity. Give people a way to NOT come to my site and view my ads. Fine if ads are not your model, but for 95% of blogs, ads are the model. RSS is a fad, and eventually, saner heads will prevail. Until that time comes, as you say, you're presenting the same info that an RSS feed does, but with the benefit of having been ranked and vetted by the HN community first.

Jebdm 17y ago
RSS may go away, but something better will take its place. People put RSS on their blogs for multiple reasons. I use it on mine because it greatly increases the usability of my site. Another way it helps is that it makes sites "stickier"; without RSS, people would not have something automatically telling them "Hey, that site you liked updated! Go check it out!". They would have to either remember the URL, or remember a link path to the site, or bookmark it, and they'd have to periodically check back. (Not that I particularly care—while it is nice to have readers, I write mainly for my own amusement, and my site is not commercialized at all.) And I think that these two factors are a net win for most sites, whether commercial or not. There are ways around the advertising issue—offer truncated feeds, put ads in your feeds, provoke comments, etc. Plus, I thought that the known wisdom was that regular readers don't use ads much, it's the ones that stumble onto your site from Google, etc.? I haven't seen a study, though. Also, for the record, I basically don't check sites that don't offer RSS. If it's particularly good, it'll get bookmarked, and maybe I'll come back to it in a few months when I've got some free time. But most sites don't even get bookmarked. On the other hand, if I find a site that had a good article, I'll often add it to my RSS reader on a trial basis.

trezor 17y ago

Either it is a bug in Firefox 3.5 RC or the mime-type provided is wrong. It just renders like text, and not like an RSS-feed I can just click to subscribe to. Since RSS feeds seems to work on other sites, I'll assume the problem is with the feed. Correct mimetype should be application/rss+xml. Confirmed: $ wget -O /dev/null http://nirmalpatel.com/fcgi/hn_feed.fcgi --2009-06-22 21:01:56-- http://nirmalpatel.com/fcgi/hn_feed.fcgi Resolving nirmalpatel.com... 207.210.105.86 Connecting to nirmalpatel.com|207.210.105.86|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 211754 (207K) [text/plain] Saving to: `/dev/null' 100%[==================================================>] 211,754 254K/s in 0.8s 2009-06-22 21:01:57 (254 KB/s) - `/dev/null' saved [211754/211754]

talison 17y ago

I didn't realize this had been submitted before (by nirmal). It was hosted on a different domain in the previous submission which is why this wasn't caught as a dupe. Anyway, you can see some of the author's comments in this thread: http://news.ycombinator.com/item?id=542334 http://news.ycombinator.com/item?id=542334

jjames 17y ago

I brought up some concerns about full content scraping last time this was discussed: http://news.ycombinator.com/item?id=542552 http://news.ycombinator.com/item?id=542552 I used the feed for a few months but found the feed-readerized articles lifeless, frozen in time and free of discussion. I'm also not a fan of the title-only RSS feed. Some sites simply lose too much in feed form imo.

schwanksta 17y ago

Thanks, I was looking for this the other day!