6 ms·

Building a TCP API: How we built our realtime app, Filmgrain, part 2

18 points by Guzba 13y ago

gobengo 13y ago
I get that there's more than request-response. Why not use Websockets instead? What do you plan to do if your mobile users want a browser-based experience?
- Guzba 13y ago
  Websockets are the natural choice when/if we bring Filmgrain to the browser. I actually think Websockets are really exciting, but for a mobile app just going straight socket made more sense.
- treeform 13y ago
  You get what many people dont! When we will need a webapp we would write a WebSocket protocol then. Why write stuff you don't need too? TCP with json was just surprisingly simple and effective for our realtime twitter+move app.
bhouston 13y ago
Wouldn't web sockets have worked here nicely?
- Guzba 13y ago
  Web sockets are great for getting TCP in the browser but since we're only building mobile apps right now, we don't need the HTTP layer, instead we're using straight sockets.
johnrob 13y ago
HTTP lets you inherit a large and diverse body of infrastructure - load balancing, monitoring, testing, logging, etc etc. Also, some combination of websockets and/or keep-alive would yield most of the benefits of using just TCP.
- Guzba 13y ago
  This is all true, and HTTP really is a great choice, but that's the point I'm trying to make--it's a choice and there are alternatives. Specifically for mobile apps, those alternative are really attractive.
- pbsdp 13y ago
  Most of the complex HTTP infrastructure exists to support a the complex HTTP protocol. Get rid of HTTP, and pretty much all of the complexity of the HTTP stack evaporates.
  - treeform 13y ago
    Exactly! You said what I could not say, but thats exactly how I feel.
jallmann 13y ago
Props for trying a new approach, Just Because. Here's some feedback: > newlines make sense as indicators for the end of a message Until someone decides to inject a newline into your JSON, then you've got a problem [1]. Just use a length-prefixed encoding, there are tons: bencode, tnetstrings, etc. Also, JSON objects delimit theselves, although you could use a proper stream parser to handle fragmented messages better. A secondary transport like websockets (as others have mentioned) would also give you framing for free. > Load balancing Just use DNS. You're already making a DNS request for S3, so here you have a redundant HTTP request, and leak network-level operational details in application logic. [1] Ignoring HTTP-style newlines here for moment; common implementations are battle-hardened, but it doesn't mean newline-terminated protocols are really a good idea...
- treeform 13y ago
  You know how to give feed back! Thanks! +1 Would length-prefixed also cause issue if malicious client send numbers that are too big, just like malicious clients now can send json with newlines? I been using websockets in my editor for a while, but for a mobile app they seamed like they just had extra stuff we did not need. But I am warming up to them now. Honestly we thought about it, did not use DNS because its scary. The errors and propagation slowness can cost you a ton of uptime. I feel safer with a static DNS.
  - jallmann 13y ago
    Set a upper bound on the length of the message, and everything within it -- constrain string lengths, numeric ranges, nesting depth, etc. This should be built into the parser. Discard messages that are truncated, malformed or don't conform to the schema (don't forget to check types...). Load balancing is ultimately your call, and HTTP might be the simplest solution if very quick turnaround is critical. But don't be afraid of the Internet's plumbing, understanding it will make you a better engineer. For example, you can still have fail-over to any of the DNS A records (completely transparent with any decent HTTP/websocket client), or if you're seriously concerned about availability, anycast/multihoming. But I'd put money on you having more downtime due to errors on your end or with your provider, than with DNS. Basically, these problems have been studied for a long time, so it's valuable to be comfortable with the various approaches. FWIW, adding extra A records is just a couple clicks on Linode, and they don't make you pay for each request to boot, unlike S3.
nitrogen 13y ago
After ages fighting slowness and lag on an HTTP-based API for some hardware (Hue lights) that could save 98% of its bandwidth using a TCP interface, I was thinking just last week that HTTP obsession is getting out of hand, and that I'd like to see more developers using TCP directly. Good on you.
- bradleyland 13y ago
  HTTP is TCP. Or rather, HTTP occurs over HTTP. There's a lot of confusion of terms flying around in this thread. TCP is a transport layer technology. HTTP provides signaling and data transport via TCP (actually, HTTP doesn't care what the transport layer is; e.g., you can do HTTP over IPX/SPX if you'd like). What's really being said is that HTTP is stateless, and stateless HTTP isn't always the best solution. When you say "an HTTP-based API for some hardware (Hue lights) that could save 98% of its bandwidth using a TCP interface", I really raised an eyebrow. You're already using a TCP interface. I think what you meant to say was that you it could save bandwidth by using a stateful protocol instead of HTTP, or even a custom binary protocol.
  - nitrogen 13y ago
    Yes, I know HTTP runs over TCP. I expected the context to be sufficient to infer that I meant a non-HTTP, single-connection, non-JSON protocol over TCP/IP. Being able to send one TCP packet with a payload of "SET 1,hsb,280,255,0" is faster than a complete TCP handshake, followed by an HTTP PUT to a lengthy URL of a JSON-formatted message. The CPU load on the puny processor in the Hue bridge would be significantly reduced as well. Has anybody done HTTP over IPX?
    - bradleyland 13y ago
      In an engineering context, it's necessary to be specific. Otherwise, someone unfamiliar with the terms being used might come away with the wrong idea. IPX is virtually dead, but it would be entirely possible to do HTTP over IPX if you had a web server and client that could bind to an IPX interface.
icedog 13y ago
Sounds like you re-invented the wheel. Google Cloud Messaging and (to a lesser extent) websockets would have been better solutions.

18 points by Guzba 13y ago

I get that there's more than request-response. Why not use Websockets instead? What do you plan to do if your mobile users want a browser-based experience?

Guzba 13y ago
Websockets are the natural choice when/if we bring Filmgrain to the browser. I actually think Websockets are really exciting, but for a mobile app just going straight socket made more sense.
treeform 13y ago
You get what many people dont! When we will need a webapp we would write a WebSocket protocol then. Why write stuff you don't need too? TCP with json was just surprisingly simple and effective for our realtime twitter+move app.

bhouston 13y ago

Wouldn't web sockets have worked here nicely?

Guzba 13y ago
Web sockets are great for getting TCP in the browser but since we're only building mobile apps right now, we don't need the HTTP layer, instead we're using straight sockets.

johnrob 13y ago

HTTP lets you inherit a large and diverse body of infrastructure - load balancing, monitoring, testing, logging, etc etc. Also, some combination of websockets and/or keep-alive would yield most of the benefits of using just TCP.

Guzba 13y ago
This is all true, and HTTP really is a great choice, but that's the point I'm trying to make--it's a choice and there are alternatives. Specifically for mobile apps, those alternative are really attractive.
pbsdp 13y ago
Most of the complex HTTP infrastructure exists to support a the complex HTTP protocol. Get rid of HTTP, and pretty much all of the complexity of the HTTP stack evaporates.
- treeform 13y ago
  Exactly! You said what I could not say, but thats exactly how I feel.

jallmann 13y ago

Props for trying a new approach, Just Because. Here's some feedback: > newlines make sense as indicators for the end of a message Until someone decides to inject a newline into your JSON, then you've got a problem [1]. Just use a length-prefixed encoding, there are tons: bencode, tnetstrings, etc. Also, JSON objects delimit theselves, although you could use a proper stream parser to handle fragmented messages better. A secondary transport like websockets (as others have mentioned) would also give you framing for free. > Load balancing Just use DNS. You're already making a DNS request for S3, so here you have a redundant HTTP request, and leak network-level operational details in application logic. [1] Ignoring HTTP-style newlines here for moment; common implementations are battle-hardened, but it doesn't mean newline-terminated protocols are really a good idea...

treeform 13y ago
You know how to give feed back! Thanks! +1 Would length-prefixed also cause issue if malicious client send numbers that are too big, just like malicious clients now can send json with newlines? I been using websockets in my editor for a while, but for a mobile app they seamed like they just had extra stuff we did not need. But I am warming up to them now. Honestly we thought about it, did not use DNS because its scary. The errors and propagation slowness can cost you a ton of uptime. I feel safer with a static DNS.
- jallmann 13y ago
  Set a upper bound on the length of the message, and everything within it -- constrain string lengths, numeric ranges, nesting depth, etc. This should be built into the parser. Discard messages that are truncated, malformed or don't conform to the schema (don't forget to check types...). Load balancing is ultimately your call, and HTTP might be the simplest solution if very quick turnaround is critical. But don't be afraid of the Internet's plumbing, understanding it will make you a better engineer. For example, you can still have fail-over to any of the DNS A records (completely transparent with any decent HTTP/websocket client), or if you're seriously concerned about availability, anycast/multihoming. But I'd put money on you having more downtime due to errors on your end or with your provider, than with DNS. Basically, these problems have been studied for a long time, so it's valuable to be comfortable with the various approaches. FWIW, adding extra A records is just a couple clicks on Linode, and they don't make you pay for each request to boot, unlike S3.

nitrogen 13y ago

After ages fighting slowness and lag on an HTTP-based API for some hardware (Hue lights) that could save 98% of its bandwidth using a TCP interface, I was thinking just last week that HTTP obsession is getting out of hand, and that I'd like to see more developers using TCP directly. Good on you.

bradleyland 13y ago
HTTP is TCP. Or rather, HTTP occurs over HTTP. There's a lot of confusion of terms flying around in this thread. TCP is a transport layer technology. HTTP provides signaling and data transport via TCP (actually, HTTP doesn't care what the transport layer is; e.g., you can do HTTP over IPX/SPX if you'd like). What's really being said is that HTTP is stateless, and stateless HTTP isn't always the best solution. When you say "an HTTP-based API for some hardware (Hue lights) that could save 98% of its bandwidth using a TCP interface", I really raised an eyebrow. You're already using a TCP interface. I think what you meant to say was that you it could save bandwidth by using a stateful protocol instead of HTTP, or even a custom binary protocol.
- nitrogen 13y ago
  Yes, I know HTTP runs over TCP. I expected the context to be sufficient to infer that I meant a non-HTTP, single-connection, non-JSON protocol over TCP/IP. Being able to send one TCP packet with a payload of "SET 1,hsb,280,255,0" is faster than a complete TCP handshake, followed by an HTTP PUT to a lengthy URL of a JSON-formatted message. The CPU load on the puny processor in the Hue bridge would be significantly reduced as well. Has anybody done HTTP over IPX?
  - bradleyland 13y ago
    In an engineering context, it's necessary to be specific. Otherwise, someone unfamiliar with the terms being used might come away with the wrong idea. IPX is virtually dead, but it would be entirely possible to do HTTP over IPX if you had a web server and client that could bind to an IPX interface.

icedog 13y ago

Sounds like you re-invented the wheel. Google Cloud Messaging and (to a lesser extent) websockets would have been better solutions.