5 ms·

Show HN: gofeed, a robust RSS and Atom Parser for Go

68 points by drakenot 11y ago

Jemaclus 11y ago
This is pretty rad. Is there a way to customize this for non-RSS feeds, or is this pretty much tailored for RSS/Atom feeds?
- drakenot 11y ago
  What is another feed format you'd like to see supported? If you have some custom thing you could take a look goxpp[0] which is the XML Pull Parser I wrote to use in gofeed. [0] https://github.com/mmcdole/goxpp https://github.com/mmcdole/goxpp
  - Jemaclus 11y ago
    I deal with a lot of custom feeds, so not a lot of standard formats. I'll take a look at goxpp! Thank you!
  - code_research 11y ago
    The selfoss reader [0] has a nice concept of extending the format of the incoming data feeds, would be also a very nice feature for your code. Even nicer it would be if new incoming data formats could be configured in yaml files without having to recompile. the selfoss author mentions these kind of data: * RSS Feeds * Images from a RSS Feed * Images from deviantArt Users * Images from tumblr * Your twitter timeline * Tweets of a twitter user * heise News with full content * golem News with full content * MMOSpy News with full content * RSS Feeds with readability certainly not the most important sources, but it gives an idea about how diverse data sources are on the net today, so limiting to rss and atom is not a good idea. maybe some people would like to add facebook streams or other "social" sources. Also please take a look at the extensive testing suite of feedparser [1] - lots of feeds there, that will break any naive assumption about how clean real life data will be. [0] http://selfoss.aditu.de/ http://selfoss.aditu.de/ [1] https://github.com/kurtmckee/feedparser/tree/develop/tests https://github.com/kurtmckee/feedparser/tree/develop/tests
    - drakenot 11y ago
      I've actually already ported hundreds of test cases from Mark Pilgrim's feedparser.
      - code_research 11y ago
        ah, that is great! I think feedparser has the biggest collection of obscure feeds, it is an awesome collection of what to expect in the wild. If you know some more sources of weird feeds produced by "modern web software", please tell me, thanks!
dzdt 11y ago
I read it as "goofed" at first.
pkulak 11y ago
Wow, this is great. I built a web scraper in Go a while back that had feed support, and this is exactly what you end up writing. You need the title, description, items, etc, but you really don't care if it's RSS or ATOM. So you translate all that crazy into some common format before you start working with it. I love how comprehensive this is. I didn't even know a lot of the ways you list to get the different RSS properties. :D

68 points by drakenot 11y ago

This is pretty rad. Is there a way to customize this for non-RSS feeds, or is this pretty much tailored for RSS/Atom feeds?

drakenot 11y ago
What is another feed format you'd like to see supported? If you have some custom thing you could take a look goxpp[0] which is the XML Pull Parser I wrote to use in gofeed. [0] https://github.com/mmcdole/goxpp https://github.com/mmcdole/goxpp
- Jemaclus 11y ago
  I deal with a lot of custom feeds, so not a lot of standard formats. I'll take a look at goxpp! Thank you!
- code_research 11y ago
  The selfoss reader [0] has a nice concept of extending the format of the incoming data feeds, would be also a very nice feature for your code. Even nicer it would be if new incoming data formats could be configured in yaml files without having to recompile. the selfoss author mentions these kind of data: * RSS Feeds * Images from a RSS Feed * Images from deviantArt Users * Images from tumblr * Your twitter timeline * Tweets of a twitter user * heise News with full content * golem News with full content * MMOSpy News with full content * RSS Feeds with readability certainly not the most important sources, but it gives an idea about how diverse data sources are on the net today, so limiting to rss and atom is not a good idea. maybe some people would like to add facebook streams or other "social" sources. Also please take a look at the extensive testing suite of feedparser [1] - lots of feeds there, that will break any naive assumption about how clean real life data will be. [0] http://selfoss.aditu.de/ http://selfoss.aditu.de/ [1] https://github.com/kurtmckee/feedparser/tree/develop/tests https://github.com/kurtmckee/feedparser/tree/develop/tests
  - drakenot 11y ago
    I've actually already ported hundreds of test cases from Mark Pilgrim's feedparser.
    - code_research 11y ago
      ah, that is great! I think feedparser has the biggest collection of obscure feeds, it is an awesome collection of what to expect in the wild. If you know some more sources of weird feeds produced by "modern web software", please tell me, thanks!

dzdt 11y ago

I read it as "goofed" at first.

pkulak 11y ago

Wow, this is great. I built a web scraper in Go a while back that had feed support, and this is exactly what you end up writing. You need the title, description, items, etc, but you really don't care if it's RSS or ATOM. So you translate all that crazy into some common format before you start working with it. I love how comprehensive this is. I didn't even know a lot of the ways you list to get the different RSS properties. :D