7 ms·

Show HN: Rockets – Reddit and websockets

98 points by rtheunissen 11y ago

julien 11y ago
Brilliant! Do you poll Reddit for new messages?
- rtheunissen 11y ago
  Yup, every second. There's roughly ~30 models created per second on average.
  - deleted 11y ago
    [deleted]
  - crindy 11y ago
    This is absolutely amazing. I'll definitely be using it extensively. I'm also incredibly curious about how it's implemented though. As far as I know, reddit doesn't have an API point for all new comments, do they?
    - rtheunissen 11y ago
      Thanks! Let me know if you have any problems or questions. Re: the implementation, take a look at the task that fetches new comments: http://git.io/vOGT3 http://git.io/vOGT3. reddit's primary 'new comments' endpoint let's you fetch at most the 100 newest comments. But you have to account for times when the previously 'newest' model is more than 100 comments in the past, ie. you have a gap because there were too many comments or another request took longer than expected. The endpoints themselves are simple, but the task to make sure you don't miss any or fall behind is tricky.
chrissnell 11y ago
Interesting. It relies on a central service that's polling Reddit, though. Why not a distributed, peer-to-peer system instead? By combining the power of hundreds of individual nodes polling the site, you could have fast visibility of new content without being too obvious to the site being monitored.
- ChristianBundy 11y ago
  I'm not OP, but then you'd have to solve for consensus. Solvable, but annoying.
  - jondubois 11y ago
    Achieving consensus using browsers as nodes is unpractical for many reasons. Firstly, there are hard limits on localStorage, then there is the issue that nodes go offline really often (whenever someone closes their browser/tab) and therefore it's highly volatile - You would have to replicate data across nodes like crazy to reduce the odds of data loss (the bandwidth use would be massive with loads of duplicate data all over the place). Plus you still need a server to do the signaling to coordinate consensus across all the nodes. Also, there are security implications when storing data inside other peoples' browsers (though this isn't relevant in this specific case). There has been some hype around using WebRTC for storing data only on the client-side using consensus algorithms like RAFT, but given the constraints which most browsers have and the massive amount of complexity required to get this working, it's really not worthwhile. I don't think these browser constraints will be lifted anytime soon - There are some very good reasons for having them in the first place - For one, most people don't want their computers' CPU, memory and their internet bandwidth to be used up to process other people's stuff.
- rtheunissen 11y ago
  It's not against the rules of reddit, and the allowed request rate is enough to grab everything even during busy periods. You're right that it relies on a central service that's polling reddit. I've done all I can to make it as stable and scalable as possible. I've been running, monitoring and testing it for a week and it's been keeping up just fine.
  - StavrosK 11y ago
    Given that many people have bots doing the exact same thing, I don't think you'd have any trouble. Hell, total load on reddit would decrease because people would use your service. I don't know why reddit didn't implement something like this themselves.
fasteo 11y ago
The animated logo immediately caught my attention. Kudos to Ken, the illustrator [1]. Great portfolio. [1] http://cargocollective.com/kensamonte/ http://cargocollective.com/kensamonte/
- rtheunissen 11y ago
  Yeah, Ken is very talented. I'll extend your appreciation.
  - deleted 11y ago
    [deleted]
StavrosK 11y ago
I wanted this for a script I was thinking of writing, nice! Unfortunately, I'm a total convenience victim these days, and any node-based service is very unlikely to enter my stack. Python is okay, Go is best. I just love only deploying a single binary. I think it speaks volumes that I like writing Python but like deploying Go.
- rmetzler 11y ago
  This is why I like Docker. I don't need to think about installing the whole stack on the server.
- rtheunissen 11y ago
  You can still make full use of this. All you need is basic websocket support. The node client [1] is small, so wouldn't take long to port to Python or Go. [1] http://git.io/vOOZo http://git.io/vOOZo
onli 11y ago
That seems to work really well, thanks for your work. I might use it. One thing I'm struggling with is the data model, of what exactly is returned on an event. I see that right that rockets is sending data like the API would've returned them, and that is why they are not defined in the code? I'm new to Reddits Api and was wondering how to get a comments url. Just realizing that I should just echo the full object and find keys that way. I'm leaving the comment, maybe you want to add some documentation/links to the proper Api documentation in that direction.
- rtheunissen 11y ago
  > maybe you want to add some documentation/links to the proper Api documentation That's an excellent idea, added in http://git.io/vOOKi http://git.io/vOOKi. > how to get a comments url A comment permalink is a strange combination of the comment itself and its parent post. You can build a comment permalink using the following template: reddit.com/r/{subreddit}/comments/{link_id[3:]}/_/{id}
  - onli 11y ago
    Thanks a lot! I discovered link_url and that would have worked for the moment, but the real comment permalink would have taken a lot of time to find out. The link to the json-wiki page is perfect as well, that is exactly what I searched and did not find. Thanks again.
rgbrgb 11y ago
So what's Reddit's take on something like this?
- rtheunissen 11y ago
  The company? I don't think they mind, as long as you're following their API rules [1], which I am. The community? Surprisingly disinterested. Not sure if I did something wrong. I thought they'd love it, but only got 2 points in r/programming and 7 points in r/node. It's not about the points but it's still discouraging. Maybe I timed it badly, maybe I didn't bait lazy leaders well enough, maybe it's actually not something cool, maybe the ranking algorithm just wasn't in my favour. ¯\_(ツ)_/¯ [1] https://github.com/reddit/reddit/wiki/API https://github.com/reddit/reddit/wiki/API
fiatjaf 11y ago
Websockets? That means the Reddit bot will be listening from a webpage? Is that how Reddit bots operate? I'm confused, I thought Reddit bots would be a normal computer process, listening/polling for content and making HTTP calls to the API, nothing from a web browser. Someone please explain it to me!
- rtheunissen 11y ago
  Not necessarily from a webpage. Web browsers support [1] what's called the WebSocket API [2], which allows browsers to create socket connections to a socket server. But many programming languages also provide a library to create socket connections on the server. This is what most reddit bots would be doing, as demonstrated in the rockets-client [3], which uses a Node.js websocket library called 'ws'. So it's incredibly flexible. You could wrap a UI around it in the browser, or use it server-side with any mainstream programming language that supports websockets. [1] http://caniuse.com/#feat=websockets http://caniuse.com/#feat=websockets [2] http://www.websocket.org/aboutwebsocket.html http://www.websocket.org/aboutwebsocket.html [3] https://github.com/rtheunissen/rockets-client https://github.com/rtheunissen/rockets-client
  - fiatjaf 11y ago
    Thank you very much.

98 points by rtheunissen 11y ago

julien 11y ago

Brilliant! Do you poll Reddit for new messages?

rtheunissen 11y ago
Yup, every second. There's roughly ~30 models created per second on average.
- deleted 11y ago
  [deleted]
- crindy 11y ago
  This is absolutely amazing. I'll definitely be using it extensively. I'm also incredibly curious about how it's implemented though. As far as I know, reddit doesn't have an API point for all new comments, do they?
  - rtheunissen 11y ago
    Thanks! Let me know if you have any problems or questions. Re: the implementation, take a look at the task that fetches new comments: http://git.io/vOGT3 http://git.io/vOGT3. reddit's primary 'new comments' endpoint let's you fetch at most the 100 newest comments. But you have to account for times when the previously 'newest' model is more than 100 comments in the past, ie. you have a gap because there were too many comments or another request took longer than expected. The endpoints themselves are simple, but the task to make sure you don't miss any or fall behind is tricky.

chrissnell 11y ago

Interesting. It relies on a central service that's polling Reddit, though. Why not a distributed, peer-to-peer system instead? By combining the power of hundreds of individual nodes polling the site, you could have fast visibility of new content without being too obvious to the site being monitored.

ChristianBundy 11y ago
I'm not OP, but then you'd have to solve for consensus. Solvable, but annoying.
- jondubois 11y ago
  Achieving consensus using browsers as nodes is unpractical for many reasons. Firstly, there are hard limits on localStorage, then there is the issue that nodes go offline really often (whenever someone closes their browser/tab) and therefore it's highly volatile - You would have to replicate data across nodes like crazy to reduce the odds of data loss (the bandwidth use would be massive with loads of duplicate data all over the place). Plus you still need a server to do the signaling to coordinate consensus across all the nodes. Also, there are security implications when storing data inside other peoples' browsers (though this isn't relevant in this specific case). There has been some hype around using WebRTC for storing data only on the client-side using consensus algorithms like RAFT, but given the constraints which most browsers have and the massive amount of complexity required to get this working, it's really not worthwhile. I don't think these browser constraints will be lifted anytime soon - There are some very good reasons for having them in the first place - For one, most people don't want their computers' CPU, memory and their internet bandwidth to be used up to process other people's stuff.
rtheunissen 11y ago
It's not against the rules of reddit, and the allowed request rate is enough to grab everything even during busy periods. You're right that it relies on a central service that's polling reddit. I've done all I can to make it as stable and scalable as possible. I've been running, monitoring and testing it for a week and it's been keeping up just fine.
- StavrosK 11y ago
  Given that many people have bots doing the exact same thing, I don't think you'd have any trouble. Hell, total load on reddit would decrease because people would use your service. I don't know why reddit didn't implement something like this themselves.

fasteo 11y ago

The animated logo immediately caught my attention. Kudos to Ken, the illustrator [1]. Great portfolio. [1] http://cargocollective.com/kensamonte/ http://cargocollective.com/kensamonte/

rtheunissen 11y ago
Yeah, Ken is very talented. I'll extend your appreciation.
- deleted 11y ago
  [deleted]

StavrosK 11y ago

I wanted this for a script I was thinking of writing, nice! Unfortunately, I'm a total convenience victim these days, and any node-based service is very unlikely to enter my stack. Python is okay, Go is best. I just love only deploying a single binary. I think it speaks volumes that I like writing Python but like deploying Go.

rmetzler 11y ago
This is why I like Docker. I don't need to think about installing the whole stack on the server.
rtheunissen 11y ago
You can still make full use of this. All you need is basic websocket support. The node client [1] is small, so wouldn't take long to port to Python or Go. [1] http://git.io/vOOZo http://git.io/vOOZo

onli 11y ago

That seems to work really well, thanks for your work. I might use it. One thing I'm struggling with is the data model, of what exactly is returned on an event. I see that right that rockets is sending data like the API would've returned them, and that is why they are not defined in the code? I'm new to Reddits Api and was wondering how to get a comments url. Just realizing that I should just echo the full object and find keys that way. I'm leaving the comment, maybe you want to add some documentation/links to the proper Api documentation in that direction.

rtheunissen 11y ago
> maybe you want to add some documentation/links to the proper Api documentation That's an excellent idea, added in http://git.io/vOOKi http://git.io/vOOKi. > how to get a comments url A comment permalink is a strange combination of the comment itself and its parent post. You can build a comment permalink using the following template: reddit.com/r/{subreddit}/comments/{link_id[3:]}/_/{id}
- onli 11y ago
  Thanks a lot! I discovered link_url and that would have worked for the moment, but the real comment permalink would have taken a lot of time to find out. The link to the json-wiki page is perfect as well, that is exactly what I searched and did not find. Thanks again.

rgbrgb 11y ago

So what's Reddit's take on something like this?

rtheunissen 11y ago
The company? I don't think they mind, as long as you're following their API rules [1], which I am. The community? Surprisingly disinterested. Not sure if I did something wrong. I thought they'd love it, but only got 2 points in r/programming and 7 points in r/node. It's not about the points but it's still discouraging. Maybe I timed it badly, maybe I didn't bait lazy leaders well enough, maybe it's actually not something cool, maybe the ranking algorithm just wasn't in my favour. ¯\_(ツ)_/¯ [1] https://github.com/reddit/reddit/wiki/API https://github.com/reddit/reddit/wiki/API

fiatjaf 11y ago

Websockets? That means the Reddit bot will be listening from a webpage? Is that how Reddit bots operate? I'm confused, I thought Reddit bots would be a normal computer process, listening/polling for content and making HTTP calls to the API, nothing from a web browser. Someone please explain it to me!

rtheunissen 11y ago
Not necessarily from a webpage. Web browsers support [1] what's called the WebSocket API [2], which allows browsers to create socket connections to a socket server. But many programming languages also provide a library to create socket connections on the server. This is what most reddit bots would be doing, as demonstrated in the rockets-client [3], which uses a Node.js websocket library called 'ws'. So it's incredibly flexible. You could wrap a UI around it in the browser, or use it server-side with any mainstream programming language that supports websockets. [1] http://caniuse.com/#feat=websockets http://caniuse.com/#feat=websockets [2] http://www.websocket.org/aboutwebsocket.html http://www.websocket.org/aboutwebsocket.html [3] https://github.com/rtheunissen/rockets-client https://github.com/rtheunissen/rockets-client
- fiatjaf 11y ago
  Thank you very much.