3 ms·

Props for trying a new approach, Just Because. Here's some feedback: > newlines make sense as indicators for the end of a message Until someone decides to inj

Props for trying a new approach, Just Because. Here's some feedback: > newlines make sense as indicators for the end of a message Until someone decides to inject a newline into your JSON, then you've got a problem [1]. Just use a length-prefixed encoding, there are tons: bencode, tnetstrings, etc. Also, JSON objects delimit theselves, although you could use a proper stream parser to handle fragmented messages better. A secondary transport like websockets (as others have mentioned) would also give you framing for free. > Load balancing Just use DNS. You're already making a DNS request for S3, so here you have a redundant HTTP request, and leak network-level operational details in application logic. [1] Ignoring HTTP-style newlines here for moment; common implementations are battle-hardened, but it doesn't mean newline-terminated protocols are really a good idea...

treeform 13y ago
You know how to give feed back! Thanks! +1 Would length-prefixed also cause issue if malicious client send numbers that are too big, just like malicious clients now can send json with newlines? I been using websockets in my editor for a while, but for a mobile app they seamed like they just had extra stuff we did not need. But I am warming up to them now. Honestly we thought about it, did not use DNS because its scary. The errors and propagation slowness can cost you a ton of uptime. I feel safer with a static DNS.
- jallmann 13y ago
  Set a upper bound on the length of the message, and everything within it -- constrain string lengths, numeric ranges, nesting depth, etc. This should be built into the parser. Discard messages that are truncated, malformed or don't conform to the schema (don't forget to check types...). Load balancing is ultimately your call, and HTTP might be the simplest solution if very quick turnaround is critical. But don't be afraid of the Internet's plumbing, understanding it will make you a better engineer. For example, you can still have fail-over to any of the DNS A records (completely transparent with any decent HTTP/websocket client), or if you're seriously concerned about availability, anycast/multihoming. But I'd put money on you having more downtime due to errors on your end or with your provider, than with DNS. Basically, these problems have been studied for a long time, so it's valuable to be comfortable with the various approaches. FWIW, adding extra A records is just a couple clicks on Linode, and they don't make you pay for each request to boot, unlike S3.
chrisdew 13y ago
You can still accept multi-line JSON with line delimited JSON. http://en.wikipedia.org/wiki/Line_Delimited_JSON http://en.wikipedia.org/wiki/Line_Delimited_JSON

treeform 13y ago

You know how to give feed back! Thanks! +1 Would length-prefixed also cause issue if malicious client send numbers that are too big, just like malicious clients now can send json with newlines? I been using websockets in my editor for a while, but for a mobile app they seamed like they just had extra stuff we did not need. But I am warming up to them now. Honestly we thought about it, did not use DNS because its scary. The errors and propagation slowness can cost you a ton of uptime. I feel safer with a static DNS.

jallmann 13y ago
Set a upper bound on the length of the message, and everything within it -- constrain string lengths, numeric ranges, nesting depth, etc. This should be built into the parser. Discard messages that are truncated, malformed or don't conform to the schema (don't forget to check types...). Load balancing is ultimately your call, and HTTP might be the simplest solution if very quick turnaround is critical. But don't be afraid of the Internet's plumbing, understanding it will make you a better engineer. For example, you can still have fail-over to any of the DNS A records (completely transparent with any decent HTTP/websocket client), or if you're seriously concerned about availability, anycast/multihoming. But I'd put money on you having more downtime due to errors on your end or with your provider, than with DNS. Basically, these problems have been studied for a long time, so it's valuable to be comfortable with the various approaches. FWIW, adding extra A records is just a couple clicks on Linode, and they don't make you pay for each request to boot, unlike S3.

chrisdew 13y ago

You can still accept multi-line JSON with line delimited JSON. http://en.wikipedia.org/wiki/Line_Delimited_JSON http://en.wikipedia.org/wiki/Line_Delimited_JSON