4 ms·

Interesting. It relies on a central service that's polling Reddit, though. Why not a distributed, peer-to-peer system instead? By combining the power of hundr

Interesting. It relies on a central service that's polling Reddit, though. Why not a distributed, peer-to-peer system instead? By combining the power of hundreds of individual nodes polling the site, you could have fast visibility of new content without being too obvious to the site being monitored.

ChristianBundy 11y ago
I'm not OP, but then you'd have to solve for consensus. Solvable, but annoying.
- jondubois 11y ago
  Achieving consensus using browsers as nodes is unpractical for many reasons. Firstly, there are hard limits on localStorage, then there is the issue that nodes go offline really often (whenever someone closes their browser/tab) and therefore it's highly volatile - You would have to replicate data across nodes like crazy to reduce the odds of data loss (the bandwidth use would be massive with loads of duplicate data all over the place). Plus you still need a server to do the signaling to coordinate consensus across all the nodes. Also, there are security implications when storing data inside other peoples' browsers (though this isn't relevant in this specific case). There has been some hype around using WebRTC for storing data only on the client-side using consensus algorithms like RAFT, but given the constraints which most browsers have and the massive amount of complexity required to get this working, it's really not worthwhile. I don't think these browser constraints will be lifted anytime soon - There are some very good reasons for having them in the first place - For one, most people don't want their computers' CPU, memory and their internet bandwidth to be used up to process other people's stuff.
rtheunissen 11y ago
It's not against the rules of reddit, and the allowed request rate is enough to grab everything even during busy periods. You're right that it relies on a central service that's polling reddit. I've done all I can to make it as stable and scalable as possible. I've been running, monitoring and testing it for a week and it's been keeping up just fine.
- StavrosK 11y ago
  Given that many people have bots doing the exact same thing, I don't think you'd have any trouble. Hell, total load on reddit would decrease because people would use your service. I don't know why reddit didn't implement something like this themselves.
  - rtheunissen 11y ago
    Given the small amount of people that would use it (relative to the entire userbase), I don't think it would ever be on their roadmap. It would take many developer hours and constant maintenance, which wouldn't be worth it for them as a company. They have more important things to build and fix.
MasterScrat 11y ago
I made a proof of concept like this some years ago. It was only synchronising the scores, and was was running as a Chrome extension. The idea is that each user sees the scores updated in realtime when he display a Reddit page, but he also broadcasts the updated scores back to the server every time he loads a page. That way no polling is involved, and it doesn't generate any additional load on Reddit servers since the updated data comes from people who were loading tbose pages anyway. This means im not bound to any API restriction (it's basically distributed scrapping). https://github.com/MasterScrat/LiveReddit https://github.com/MasterScrat/LiveReddit
- rtheunissen 11y ago
  That's a very cool idea. An unintended side-effect of rockets could be that it reduces the load on reddit's server. My theory is that there are N bots that rely on monitoring new content, and they're all polling for the same data. If those bots just hook into something like this, that's N less requests per second sent to reddit.com.
  - TeMPOraL 11y ago
    There's of course a side effect of data being untrusted. While an individual couldn't game this if the system would depend on multiple sources, an organized group (like 4chan) could likely make the system show whatever they want.
    - rtheunissen 11y ago
      How might they achieve that?
      - TeMPOraL 11y ago
        I was referring to MasterScrat's idea of a browser extension. In that case, it's only a matter of saturating the service with crafted data, something that 4chan excels at doing.

ChristianBundy 11y ago

I'm not OP, but then you'd have to solve for consensus. Solvable, but annoying.

jondubois 11y ago
Achieving consensus using browsers as nodes is unpractical for many reasons. Firstly, there are hard limits on localStorage, then there is the issue that nodes go offline really often (whenever someone closes their browser/tab) and therefore it's highly volatile - You would have to replicate data across nodes like crazy to reduce the odds of data loss (the bandwidth use would be massive with loads of duplicate data all over the place). Plus you still need a server to do the signaling to coordinate consensus across all the nodes. Also, there are security implications when storing data inside other peoples' browsers (though this isn't relevant in this specific case). There has been some hype around using WebRTC for storing data only on the client-side using consensus algorithms like RAFT, but given the constraints which most browsers have and the massive amount of complexity required to get this working, it's really not worthwhile. I don't think these browser constraints will be lifted anytime soon - There are some very good reasons for having them in the first place - For one, most people don't want their computers' CPU, memory and their internet bandwidth to be used up to process other people's stuff.

rtheunissen 11y ago

It's not against the rules of reddit, and the allowed request rate is enough to grab everything even during busy periods. You're right that it relies on a central service that's polling reddit. I've done all I can to make it as stable and scalable as possible. I've been running, monitoring and testing it for a week and it's been keeping up just fine.

StavrosK 11y ago
Given that many people have bots doing the exact same thing, I don't think you'd have any trouble. Hell, total load on reddit would decrease because people would use your service. I don't know why reddit didn't implement something like this themselves.
- rtheunissen 11y ago
  Given the small amount of people that would use it (relative to the entire userbase), I don't think it would ever be on their roadmap. It would take many developer hours and constant maintenance, which wouldn't be worth it for them as a company. They have more important things to build and fix.

MasterScrat 11y ago

I made a proof of concept like this some years ago. It was only synchronising the scores, and was was running as a Chrome extension. The idea is that each user sees the scores updated in realtime when he display a Reddit page, but he also broadcasts the updated scores back to the server every time he loads a page. That way no polling is involved, and it doesn't generate any additional load on Reddit servers since the updated data comes from people who were loading tbose pages anyway. This means im not bound to any API restriction (it's basically distributed scrapping). https://github.com/MasterScrat/LiveReddit https://github.com/MasterScrat/LiveReddit

rtheunissen 11y ago
That's a very cool idea. An unintended side-effect of rockets could be that it reduces the load on reddit's server. My theory is that there are N bots that rely on monitoring new content, and they're all polling for the same data. If those bots just hook into something like this, that's N less requests per second sent to reddit.com.
- TeMPOraL 11y ago
  There's of course a side effect of data being untrusted. While an individual couldn't game this if the system would depend on multiple sources, an organized group (like 4chan) could likely make the system show whatever they want.
  - rtheunissen 11y ago
    How might they achieve that?
    - TeMPOraL 11y ago
      I was referring to MasterScrat's idea of a browser extension. In that case, it's only a matter of saturating the service with crafted data, something that 4chan excels at doing.