8 ms·
I think I've come full circle on this... (I was one of the creators of RSS). My current company, Datastreamer: http://www.datastreamer.io/ http://www.datastr
by burtonator 8y ago
I think I've come full circle on this...
(I was one of the creators of RSS).
My current company, Datastreamer:
http://www.datastreamer.io/ http://www.datastreamer.io/
Provides social media data streams for companies and search engines wanting full torrents of web content.
We deprecated RSS a LONG time ago, which was for me, like abandoning your baby.
I think RSS is dead in many ways but it's also still around in a sense.
Mostly because of Twitter and Facebook metadata. You can accomplish 90% of what you want with RSS just by parsing the metadata on an HTML page.
Because we've added NLP and content extraction algorithms on top of the content we're able to re-construct feeds that are better than the original RSS.
With another app I'm working on:
https://getpolarized.io/ https://getpolarized.io/
I might actually use something similar to build in data feeds similar to a feed reader.
But man.. I can't believe I'm thinking about building in a feed reader again :-P
- tgough 8y agoIs data streamer available for hobbyists? I saw the contact us page, but can’t tell if you provide any access for people who are just tinkering. Also, what is your pricing like? I understand you can’t give details, but are we talking a few hundred a month? A few thousand? Thanks!
- burtonator 8y agoWe provide discounts for startups and researchers. Definitely contact us. Pricing is based on query throughput but it's definitely affordable.
- tgough 8y agoThank you. I just submitted a query via your website.
- ip26 8y agoThanks for your work on RSS. I can't stand the current news environment and have gone back to RSS for tuning into the news I care about. It's the only way I have to curate a list of sources- cutting popular sources I don't like and including super-niche sources I love. (I also like how data-lite it tends to be!) Shout-out to one of those niche sources, for example: https://www.newsdeeply.com/water https://www.newsdeeply.com/water
- MrJagil 8y agoWhat other news sources do you like?
- ip26 8y agoHigh Country News, CityLab, Alpinist, Adventure Journal, Hyperlight Mountain Gear. Some regional stuff. Some local stuff. On certain sites like The Guardian, you can RSS follow certain authors, which I also like. Open to other great niche sources.
- mindcrime 8y agoI think I've come full circle on this... Keep circling!
- quotemstr 8y agoNLP and content extraction is hard and unreliable except when carefully and continually maintained. RSS is simple for both publishers and clients and allows for simple, decentralized implementations of both roles. Any shift toward heavyweight processing is a shift toward centralization, which is the last thing the web needs right now.
- Natela 8y agowith RSS it was decentralized, you could "follow" bloggers and website directly. Now news are own by big companies like FB or Twitter and it's kind of sad. If you want to follow someone you have to comply to FB or Twitter
- scarface74 8y agoRSS is far from dead as far as content producers are concern. Every publishing site still supports it. It’s not popular among users but that doesn’t affect me.
- newman8r 8y agoI hope you do build one. The majority of the top newspapers still do offer RSS (as I'm sure you know) - there's still time to revive it IMO.
- capdeck 8y ago> You can accomplish 90% of what you want with RSS just by parsing the metadata on an HTML page. I am not sure you can. Metadata is quite inconsistent and many websites that I have the RSS feed for block crawlers and scrapers. Advantages of RSS are common standard and design to be consumed by a bot. What we've got instead are curated fb and twitter feeds that people get hooked on and never look back. Openness is the aspect of RSS that I'll miss the most.
- irrational 8y agoHow do they block crawlers and scrapers? If I grab the page using headless chrome, how can it tell what I'm doing with the data on my end, especially if I have the script grab the pages at a rate that would be more consistent with a human?
- tubbs 8y ago> at a rate that would be more consistent with a human? I think that's the hard part. Some content providers put out content way quicker than a human could ever keep up with. I wrote a web crawler once to download high resolution pictures of cars from a Russian website at a quick but modest pace and I got IP banned within a couple hours.
- kbenson 8y agoIt depends on the site. If the Russian site wasn't used to the amount of traffic you generated through your downloads, it's easy to see. If a site is updating content so fast that crawling it all would get you blocked, you use multiple crawlers. It's trivial and cheap to get 100 different ips for crawling for your headless chromes, instances. At that point you could get something new once a minute and not reuse an ip for close to two hours. A more aggressive reuse schedule would make that sufficient for almost any site that it's worth crawling.
- shabbyrobe 8y agoHow do you get 100 different IPs trivially and cheaply? I suppose we might have different definitions of cheap; 100 EC2 t3.nanos would be a bit out of my price range, but that might be small change for the next person.
- the_clarence 8y agoYou'd be surprised how many of us mostly ingest news through RSS feeds.
- kitsunesoba 8y agoIt’s the only way to distill the unending flow into something scannable and digestible, especially if you want multiple sources for each story. There’s no way I’d use twitter or even news site home pages for these things… the signal to noise ratios on those are far too low.
- arendtio 8y agoConcerning getpolarized.io: I just watched the first video on the front page, and at 50% I was like, okay 'reading-progress tracker' I don't need that and was about to close it, but instead, I skipped through to the end, and then I heard 'Anki'. Boom, spot on. Being able to create Anki flashcards while you are reading new content is pretty cool, and much more useful than the whole 'non-linear reading progress' (most documents are meant to be read linearly, and my PDF reader opens a file at the position I closed it). Now I will check it out ;-)
- newacctjhro 8y agoDude, Wordpress supports RSS, and it powers the web. For those that use it, RSS works just fine.
- Semiapies 8y agoRSS is the norm and about as obsolete as HTML. It only fails to exist in half-assed greenfield projects and sites like Twitter where the management has decided you must follow and interact in very limited ways.
- tim44 8y agoWhat companies want torrent form? You have them host your data?
- NicoJuicy 8y agoLol, thanks. Gotta love HN for meeting people that invent tech i use :-P. One of my webapps is using rss and I like it a lot. I currently use it mostly for the "post to hn" function and as rss reader ofc :p It just needs some curation and the example I'm showing is more for testing with "more than average data" -> http://handlr.sapico.me/Home/Newest http://handlr.sapico.me/Home/Newest Ps. Rss is good enough, just like SQL is good enough untill you hit the limits Actually today I thought about transforming schema org data to rss depending on the configuration ( eg. price changes), which seems to be what you are doing ( sort of)
- forapurpose 8y ago> Mostly because of Twitter and Facebook metadata. You can accomplish 90% of what you want with RSS just by parsing the metadata on an HTML page. Very few end users can parse the metadata on an HTML page, and of those who can do it, few have time to do it for every page they'd like to 'subscribe to'.
- tokyodude 8y ago> Mostly because of Twitter and Facebook metadata. sounds like your saying a site has to use Facebook and Twitter if they want to achieve the same thing as an rss feed. how is having two for profit companies who record ever click be the gateway to a feed the right solution?
- Semiapies 8y agoThree; you're forgetting Datastreamer itself. I'm the last person to want to begrudge making a profit, but the move away from RSS is a blatant, deliberate move away from openness/user control and toward walled gardens. And ever since Twitter dropped RSS, I realized I'd rather do without the sites entirely than do without their feeds. Meanwhile, huge swathes of the internet still use RSS and Atom or even the age-old approach of mailing lists. (And readers like Feedbin gives you an address to provide to mailing lists so you can follow those just like any other feed.) Every time I see someone say RSS is dead, I go and look at my unread article count.
- rhizome 8y agosounds like your saying a site has to use Facebook and Twitter if they want to achieve the same thing as an rss feed. Worse: a site has to scrape FB and TWTR to achieve the same thing. I'm baffled how anybody who even knows what page-scraping is could think of it as an improvement over RSS.
- kgwxd 8y agoThe thing you're working on now sounds like it's way out of the RSS wheelhouse. Maybe you just gave your baby a job it didn't want :) Filters (never see topics or even entire sites, you don't want to); mark-as-read (never rescan, HUGE time saver); single place to monitor multiple sources; standard. All these things make RSS far superior to any subscribe/follow systems I've seen anywhere else. I know RSS was supposed to be about the content too, but just as a notification system, it's crushes anything else out there. Some sites still include the content, but obviously you can't get that from aggregate and multimedia sites. RSS still provides a better way to consume the general web, even if most people don't realize it. Thank you for your work.
- askmike 8y ago> I know RSS was supposed to be about the content too, but just as a notification system, it's crushes anything else out there. The sad truth is that RSS is a product designed for an internet that doesn't really exist anymore (blogosphere on top of the open web). The reason everyone is dropping support for RSS is because not enough people are using it. Besides the consumers not using it anymore you also see producers moving behind paywalls or pushing people to their website so they can click ads (media), and everyone else uses a few closed platforms (twitter, facebook). EDIT: I personally would choose an RSS feed over Facebook's newsfeed any day of the week. Including having to go through the hassle of keeping my subscriptions up to date and going out of my way to find new sources. Unfortunately the reality is that mass consumers don't agree and FB has over 2 billion users whereas RSS readers are dropping left and right because no one is using them.
- emn13 8y agoMonopolies suck - perhaps they'll be broken open someday, but it doesn't appear to be soon.
- kgwxd 8y agoRSS readers are fine, there's tons of them available. There's no need for it to be built into a browser or email client and there's definitely no need for it to be a hosted service. Firefox did the right thing. Open source, dedicated RSS client software is readily available and likely needs very little maintenance at this point, so we don't have to worry about useless middle-men getting bored. Every site I've cared to "subscribe" to has an RSS feed somewhere, so that end of things is still alive and well. I don't see any reason for them to stop either, I still end up at their site (assuming they only publish links) and that's all they can hope for.
- sjapkee 8y ago>I think RSS is dead This is the most effective way to consume content. It's not dead and won't be until more effective one.
- catach 8y agoI interpret "dead" in this context as "very few people are using it, and usage is shrinking".
- rstuart4133 8y ago> I interpret "dead" in this context as "very few people are using it, and usage is shrinking". Is it shrinking? My podcast apps use RSS feeds to locate episodes. Debian uses sourceforge's RSS feeds to locate updated sources. A published, chronological list of events or things is not an uncommon thing to want, and as the parent said unless something better comes along RSS/Atom is it.
- ksec 8y agoWhen usage excluding software update and podcast. How many people are using RSS for consuming "news"?
- deleted 8y ago[deleted]
- igravious 8y ago> I think I've come full circle on this... > (I was one of the creators of RSS). To come full circle figuratively means to be back where you started. What you meant to say is that you've done a 180 (a U-turn) on the idea. I think it is a curious thing that even though all major browsers are written and built by geeks their corporate entities have ditched a tech that geeks are very vocal about liking. Google famously killed Google Reader, a much loved product. That decision made me for the first time look at Google in a different light (when Reader was around I practically lived in Reader – this is perhaps what Google didn't like, I know it's a paranoid/conspiratorial viewpoint but …) [0] Microsoft killed RSS support when moving from IE to Edge. [1] Apple removed RSS support from Safari in Mountain Lion only to add it back in half-heartedly in Yosemite but I don't know enough about the MacOS/iOS platform to say for certain where its support is at. [2] And today – champion of the open web, Mozilla – kills support a short while after acquiring Pocket, a non-open feature of their browser that for most people sits uselessly on the toolbar taking up valuable screen real estate and also bloats the code-base. That's all of them then. So what's up? Why is it that browser creators don't want us to consume content in an open, uniform and standard way using RSS when it works fine and clearly geek like consuming content that way? What's so awful about: “I like that site (or sub-site), please subscribe to it and notify me about new content.”? Anybody who has any inside info on this topic please let us know. I think it mirrors why large tech companies have failed to deliver open and federated social media. From now on, given that Chromium is also open source, I know longer consider Mozilla to be the champion of the open web. Not if they kill useful open standards and don't champion other ones even if it's not popular. Freedom has never been about popularity. Firefox should have been the first browser ages ago to ship with a built in ad-blocking and anti-tracking tech but they never did. Why? I thought Persona was pretty cool tech, but nope, they had to shut it down. And Firefox Hello? D.O.A. In certain optics it looks like Mozilla only exist to enrich themselves and not rock the ecosystem too much. [0] “ Google Reader shutting down 1926 points knurdle 6 years ago 704 comments (http://googleblog.blogspot.com/2013/03/a-second-spring-of-cleaning.html http://googleblog.blogspot.com/2013/03/a-second-spring-of-cl...) “ https://news.ycombinator.com/item?id=5371725 https://news.ycombinator.com/item?id=5371725 (I have rarely seen such a uniform outcry on HN) [1] https://answers.microsoft.com/en-us/insider/forum/insider_internet-insider_spartan-insiderplat_pc/will-microsoft-edge-browser-support-rss-feeds/778b6dd2-346e-44d3-a1c7-2e9350f4c958 https://answers.microsoft.com/en-us/insider/forum/insider_in... [2] https://thesweetsetup.com/apps/best-rss-reader-os-x/ https://thesweetsetup.com/apps/best-rss-reader-os-x/
- alangibson 8y agoDatastreamer actually looks pretty nice, but the lack of pricing info gives me that "if you have to ask you can't afford it" vibe.
- deleted 8y ago[deleted]
- atomi 8y agoI'm using miniflux to read this article. RSS is dead. Long live RSS. I know this much, if RSS was easier to monetize. Firefox wouldn't have any problems maintaining it on their browser.
- orbital-decay 8y agoDoesn't miniflux use RSS as the backend though? I'm using Inoreader, but it's still RSS under the hood. >if RSS was easier to monetize There's no problems with monetizing RSS feeds. It's just a notification system.
- brian_herman 8y agoGet polarized looks awesome nice work man!
- rhizome 8y agoRefusing to provide an RSS feed is one thing, but what Firefox is doing is removing "reader-y" code that other people have written better, plus axing live bookmarks which I never liked. That is to say, what Firefox and Datastreamer are deciding about RSS appears to be unrelated. However, as a cocreator of RSS, you probably know that HTML -- which is the source of "metadata" for you -- is hard to parse correctly. Much harder and error-prone than parsing XML for RSS. Your solution serves only to push complexity downstream. Providing RSS feeds is practically free everywhere. I wouldn't be surprised if many more resources are spent deciding to turn it off than feeds would consume for all eternity.