6 ms·

Show HN: Heartbeat – Transform REST Endpoints to Streaming APIs

111 points by sidi 10y ago

LeonM 10y ago
Looks neat, but I can't really think of a real world application for it other than some debugging of API's. It's basically polling-as-a-service. The hashtag about the real time web is a joke, right? Please, don't turn it into some kind of SAAS, that would ruin everything. I'd be willing to pay a license fee on a stand-alone app, not a monthly fee on a tool I'd use once every few months.
- mobiuscog 10y ago
  I still can't get over how many 'technologies' are still just 'something else does the polling so you don't have to'. Trying to arrange a polling-in-sixty-seconds universal-protocol in a .. well.
- forgotpwtomain 10y ago
  The socket.io example server is 6 lines: https://github.com/socketio/socket.io https://github.com/socketio/socket.io A polling loop is 4-5? I guess that's everything minus the WEB Gui.
- sidi 10y ago
  As other comments point out, there are significant benefits to having a streaming endpoint over a polling logic. It conserves resources, provides a more elegant representation with an events based interface to act on incoming data. A latent benefit of Heartbeat (which we will start emphasizing soon) is an ability to filter the incoming stream of events by the JSON data properties, so one can build nice IFTTT style code using any REST based APIs as sources. We are leaning towards creating a monthly and a longer-term pricing plan, but if your use-case is casual, then it will always be free. We are able support the current load of requests with a single worker process.
dvcc 10y ago
Looks cool but a minimum of 1 minute polling intervals? That's a pretty long time for what should be streaming data, no? If the data has no need to be real-time then why stream it? Might want to fix the Heartbeat icon link by the way, it currently links to https://heartbeat.appbase.io/heartbeat.appbase.io https://heartbeat.appbase.io/heartbeat.appbase.io
- drdaeman 10y ago
  > If the data has no need to be real-time then why stream it? As I get it, it's basically a way to keep request rates low. Rather than every client polling your server every minute (which can be quite heavy), only one (Heartbeat) does, and the rest just subscribe to Heartbeat streams. Totally makes sense if you haven't planned a pubsub-based API but notice that your RESTful one is close to being hammered into a pulp - so you need something to give to your clients, fast. But I don't think it make much sense as a long-term solution.
- sidi 10y ago
  We would like to enable a lower polling interval, but wanted to see the general response before doing that. In fact, we would like to enable: 1. A lambda function support to transform APIs (diffing to only append new data, adding meta data, removing unnecessary fields) before they get streamed. 2. Polling frequencies of up to 1s. 3. Custom domains for the resulting streaming endpoints.
  - mathiasrw 10y ago
    Now - THAT - sounds awesome
altharaz 10y ago
Hey, that's a cool service. Is that different from http://streamdata.io/ http://streamdata.io/ ?
simonpure 10y ago
Reminds me of PubSubHubbub but using streams instead of webhooks for updates - https://en.m.wikipedia.org/wiki/PubSubHubbub https://en.m.wikipedia.org/wiki/PubSubHubbub
doublerebel 10y ago
Well, I can certainly see the usefulness. I just had a need this past week, but don't mind running my own Node instance. Here's sseries-of-tubes, a Nodejs library I wrote to turn any Express/Connect route (or anything you can poll) into a Server Sent Events stream: https://github.com/nextorigin/sseries-of-tubes https://github.com/nextorigin/sseries-of-tubes Pretty fast for you guys to turn it into a service already. What are you using on the backend? Node makes it really easy.
- jkarneges 10y ago
  Appbase uses Pushpin (https://github.com/fanout/pushpin https://github.com/fanout/pushpin) to handle streaming.
PCaponetti 10y ago
Why don't more web guys use MQTT for pub/sub live data? I saw the PubSubHubBub in the comments already, but most devs are still shoe-horning HTTP into solutions that really aren't request/response based
- doublerebel 10y ago
  SSE makes HTTP streaming dead simple. MQTT requires a server/broker and for data to be marshaled/unmarshaled at each end. HTTP streaming can be just text. Not to mention HTTP identifies the resource by path in the URI, and easily routes right through proxies. TCP is not nearly that accessible.
- jkarneges 10y ago
  In general, the API community has consolidated around HTTP for realtime push, mainly because it's what developers know and it's super easy to use (you can just curl a stream for example). Another thing to know about APIs is that the endpoints tend to be resource-oriented, and resources are often abstractions. For example, when you interact with Twitter's streaming API, no doubt there are some pubsub mechanisms going on behind the scenes, but pubsub concepts such as publishers, subscribers, brokers, topics, messages, store-and-forward, etc are hidden away by the API. You just fetch a resource and get tweets. My feeling is that the most successful protocols used in APIs are going to be those that expose very little about the inner workings or topology of the server and other client entities.
  - niftich 10y ago
    This; and HTTP has a massive install base, readily available clients on every platform, casual debuggability from tools many people already possess (browser address bar, browser js console, browser addons, curl). Specific protocols have their merits, but they can only really compete in the backend-to-backend space, and even there they have to compete against HTTP endpoints. In anything that can be construed as vaguely front-end, HTTP has a massive first mover advantage with the ecosystem it brings.
    - 10y ago
SmellTheGlove 10y ago
This is very cool, thank you for sharing. I wish I could demo, but my current work is such that I really can't expose my endpoints/data to a 3rd party (financial services/PII type work). That said, pretend I could provided we had appropriate security - tell me about the security involved.
- sidi 10y ago
  There are two parts: 1. Source API endpoint - Heartbeat fetches the data using the authentication supported by the source API (basic auth, security token in headers, URL query strings for example). 2. Heartbeat transformed endpoint - This is accessible over an HTTPS endpoint and is secured with a basic auth username:password key. As long as you use it within a secure environment (server code, or an internal network), there should be no problems. If you use it within a web app that's distributed to other users, they can see the key. However, this key is read-only and we will allow generating a new key from within the dashboard shortly.
deleted 10y ago
[deleted]

111 points by sidi 10y ago

LeonM 10y ago

Looks neat, but I can't really think of a real world application for it other than some debugging of API's. It's basically polling-as-a-service. The hashtag about the real time web is a joke, right? Please, don't turn it into some kind of SAAS, that would ruin everything. I'd be willing to pay a license fee on a stand-alone app, not a monthly fee on a tool I'd use once every few months.

mobiuscog 10y ago
I still can't get over how many 'technologies' are still just 'something else does the polling so you don't have to'. Trying to arrange a polling-in-sixty-seconds universal-protocol in a .. well.
forgotpwtomain 10y ago
The socket.io example server is 6 lines: https://github.com/socketio/socket.io https://github.com/socketio/socket.io A polling loop is 4-5? I guess that's everything minus the WEB Gui.
sidi 10y ago
As other comments point out, there are significant benefits to having a streaming endpoint over a polling logic. It conserves resources, provides a more elegant representation with an events based interface to act on incoming data. A latent benefit of Heartbeat (which we will start emphasizing soon) is an ability to filter the incoming stream of events by the JSON data properties, so one can build nice IFTTT style code using any REST based APIs as sources. We are leaning towards creating a monthly and a longer-term pricing plan, but if your use-case is casual, then it will always be free. We are able support the current load of requests with a single worker process.

dvcc 10y ago

Looks cool but a minimum of 1 minute polling intervals? That's a pretty long time for what should be streaming data, no? If the data has no need to be real-time then why stream it? Might want to fix the Heartbeat icon link by the way, it currently links to https://heartbeat.appbase.io/heartbeat.appbase.io https://heartbeat.appbase.io/heartbeat.appbase.io

drdaeman 10y ago
> If the data has no need to be real-time then why stream it? As I get it, it's basically a way to keep request rates low. Rather than every client polling your server every minute (which can be quite heavy), only one (Heartbeat) does, and the rest just subscribe to Heartbeat streams. Totally makes sense if you haven't planned a pubsub-based API but notice that your RESTful one is close to being hammered into a pulp - so you need something to give to your clients, fast. But I don't think it make much sense as a long-term solution.
sidi 10y ago
We would like to enable a lower polling interval, but wanted to see the general response before doing that. In fact, we would like to enable: 1. A lambda function support to transform APIs (diffing to only append new data, adding meta data, removing unnecessary fields) before they get streamed. 2. Polling frequencies of up to 1s. 3. Custom domains for the resulting streaming endpoints.
- mathiasrw 10y ago
  Now - THAT - sounds awesome

altharaz 10y ago

Hey, that's a cool service. Is that different from http://streamdata.io/ http://streamdata.io/ ?

simonpure 10y ago

Reminds me of PubSubHubbub but using streams instead of webhooks for updates - https://en.m.wikipedia.org/wiki/PubSubHubbub https://en.m.wikipedia.org/wiki/PubSubHubbub

doublerebel 10y ago

Well, I can certainly see the usefulness. I just had a need this past week, but don't mind running my own Node instance. Here's sseries-of-tubes, a Nodejs library I wrote to turn any Express/Connect route (or anything you can poll) into a Server Sent Events stream: https://github.com/nextorigin/sseries-of-tubes https://github.com/nextorigin/sseries-of-tubes Pretty fast for you guys to turn it into a service already. What are you using on the backend? Node makes it really easy.

jkarneges 10y ago
Appbase uses Pushpin (https://github.com/fanout/pushpin https://github.com/fanout/pushpin) to handle streaming.

PCaponetti 10y ago

Why don't more web guys use MQTT for pub/sub live data? I saw the PubSubHubBub in the comments already, but most devs are still shoe-horning HTTP into solutions that really aren't request/response based

doublerebel 10y ago
SSE makes HTTP streaming dead simple. MQTT requires a server/broker and for data to be marshaled/unmarshaled at each end. HTTP streaming can be just text. Not to mention HTTP identifies the resource by path in the URI, and easily routes right through proxies. TCP is not nearly that accessible.
jkarneges 10y ago
In general, the API community has consolidated around HTTP for realtime push, mainly because it's what developers know and it's super easy to use (you can just curl a stream for example). Another thing to know about APIs is that the endpoints tend to be resource-oriented, and resources are often abstractions. For example, when you interact with Twitter's streaming API, no doubt there are some pubsub mechanisms going on behind the scenes, but pubsub concepts such as publishers, subscribers, brokers, topics, messages, store-and-forward, etc are hidden away by the API. You just fetch a resource and get tweets. My feeling is that the most successful protocols used in APIs are going to be those that expose very little about the inner workings or topology of the server and other client entities.
- niftich 10y ago
  This; and HTTP has a massive install base, readily available clients on every platform, casual debuggability from tools many people already possess (browser address bar, browser js console, browser addons, curl). Specific protocols have their merits, but they can only really compete in the backend-to-backend space, and even there they have to compete against HTTP endpoints. In anything that can be construed as vaguely front-end, HTTP has a massive first mover advantage with the ecosystem it brings.
  - 10y ago

SmellTheGlove 10y ago

This is very cool, thank you for sharing. I wish I could demo, but my current work is such that I really can't expose my endpoints/data to a 3rd party (financial services/PII type work). That said, pretend I could provided we had appropriate security - tell me about the security involved.

sidi 10y ago
There are two parts: 1. Source API endpoint - Heartbeat fetches the data using the authentication supported by the source API (basic auth, security token in headers, URL query strings for example). 2. Heartbeat transformed endpoint - This is accessible over an HTTPS endpoint and is secured with a basic auth username:password key. As long as you use it within a secure environment (server code, or an internal network), there should be no problems. If you use it within a web app that's distributed to other users, they can see the key. However, this key is read-only and we will allow generating a new key from within the dashboard shortly.

deleted 10y ago

[deleted]