7 ms·

A similar complaint came up here recently on a thread called Next steps for Gmane [1], starting with "I really miss the newsgroups that focused on just the mess

A similar complaint came up here recently on a thread called Next steps for Gmane [1], starting with "I really miss the newsgroups that focused on just the messages, and could be consumed by any NNTP client, stored offline, searched, etc." Quoting a portion of a great answer [2] from HN user niftich, that feels very appropriate here: > "Not enough people make new running-on-TCP or running-on-UDP protocols because new protocols are hard to design, they don't work with the one application where everyone spends 70+% of their time (the web browser), and they probably get blocked on a middlebox except if you use port 80 or 443 and fake being HTTP anyway. For all but very specialized use-cases, vomiting blobs of JSON (or if you want to feel extra good, some custom binary serialization format like protobuf or Thrift or Cap'nProto or MessagePack) across HTTP endpoints is pretty okay." [1] https://news.ycombinator.com/item?id=12440230 https://news.ycombinator.com/item?id=12440230 [2] https://news.ycombinator.com/item?id=12440783 https://news.ycombinator.com/item?id=12440783

batbomb 10y ago
Running over 80 and 443 using an arbitrary protocol often doesn't even work with next gen firewalls because they don't recognize the protocol as HTTP.
- nostrademons 10y ago
  Curious how that would work on 443, given that the whole point of SSL is that you can't recognize the payload as...well, anything.
  - laumars 10y ago
    Payload aside, SSL is identifiable as SSL. The host name is transmitted in clear text (SNI) as well as the protocol version and cipher negotiation. Just because the traffic is encrypted, it doesn't mean the connection type isn't identifiable.
    - nostrademons 10y ago
      Right, but presumably any arbitrary protocol that plans to tunnel over port 443 would use a standard SSL negotiation, standard HTTPS connection, and then send the application-level protocol as part of the encrypted payload. My very first job (back in 2000) actually did exactly this - we tunneled arbitrary TCP/IP traffic over HTTPS. The startup died over various management & investor relations fuck-ups, but the product was working - you could do things like make a SourceSafe connection across a firewall without exposing the server to the Internet. There was some concern about whether firewalls would kill the connection if it was open for too long, but it turned out that none of them actually did this (which unfortunately we didn't figure out until we were almost out of money and had wasted several months handling this case; see again re: management fuck-ups).
      - laumars 10y ago
        That's a different point though as you're now effectively using HTTP as your transport layer rather than using your proprietary protocol "naked" on port 443. So whether you send your protocol in a proprietary binary format over HTTP or via JSON API's, it's still just HTTP over TLS. You're still sending HTTP headers, just with our own proprietary content as the body. The real problem is if you want persistent types of connections. As websockets have taught us: a great many equipment that supports HTTP don't support websockets and thus why even websites are still often stuck with RESTful APIs even when that doesn't always fit their problem quite right (that situation is improving however). If one is using HTTP to transport your protocol (and the aforementioned websockets point isn't an issue) then your protocol should work over port 80 as well as 443. Barring any corporate / nationwide web filtering; but as others have already pointed out, that could also be an issue with 443 even with advantage of TLS. By the way, was it Socks2HTTP you worked on? I remember that project back around the time you described and was quite fascinated by it.
        sly010 10y ago
        > That's a different point though as you're now effectively using HTTP as your transport layer rather than using your proprietary protocol "naked" on port 443. HTTPS is built on top of TLS, not the other way around. You can't (passively) tell if a 443 TLS connection is HTTPS or a proprietary stream. You can take a guess based on statistics (which is what the Chinese firewall does to detect tunneling) but that's about it.
  - mjevans 10y ago
    Depends if the network your on is state level and/or corporate ownership of device level MitM or not.
_pmf_ 10y ago
> "and they probably get blocked on a middlebox except if you use port 80 or 443 and fake being HTTP anyway" Curiously, games using UDP still work fine. Stop justifying bad protocols with the proxy straw man.
niftich 10y ago
And as my view in that post foretells, I believe Twitter is already a protocol, we just call it the Twitter API. It's not an open protocol, of course, and its domain model is fairly anemic [1], especially compared to that of Facebook [2][3][4]. But in my opinion, this is what the author is trying to say: Twitter could have been a place where people interact with structured resources created and consumed by other apps. It's a possibility, especially since Facebook became that place instead. Ultimately, Twitter didn't because they wanted to focus on driving more traffic through their first-party app (presumably as a captive client on which they can eventually display ads), and because they focused on cultivating a community (like Medium, Tumblr, LiveJournal) rather than a resource ecosystem. [1] https://dev.twitter.com/rest/public https://dev.twitter.com/rest/public [2] https://developers.facebook.com/docs/graph-api/reference https://developers.facebook.com/docs/graph-api/reference [3] https://developers.facebook.com/docs/sharing/opengraph/using-objects https://developers.facebook.com/docs/sharing/opengraph/using... [4] https://developers.facebook.com/docs/sharing/opengraph/object-api/ https://developers.facebook.com/docs/sharing/opengraph/objec...

batbomb 10y ago

Running over 80 and 443 using an arbitrary protocol often doesn't even work with next gen firewalls because they don't recognize the protocol as HTTP.

nostrademons 10y ago
Curious how that would work on 443, given that the whole point of SSL is that you can't recognize the payload as...well, anything.
- laumars 10y ago
  Payload aside, SSL is identifiable as SSL. The host name is transmitted in clear text (SNI) as well as the protocol version and cipher negotiation. Just because the traffic is encrypted, it doesn't mean the connection type isn't identifiable.
  - nostrademons 10y ago
    Right, but presumably any arbitrary protocol that plans to tunnel over port 443 would use a standard SSL negotiation, standard HTTPS connection, and then send the application-level protocol as part of the encrypted payload. My very first job (back in 2000) actually did exactly this - we tunneled arbitrary TCP/IP traffic over HTTPS. The startup died over various management & investor relations fuck-ups, but the product was working - you could do things like make a SourceSafe connection across a firewall without exposing the server to the Internet. There was some concern about whether firewalls would kill the connection if it was open for too long, but it turned out that none of them actually did this (which unfortunately we didn't figure out until we were almost out of money and had wasted several months handling this case; see again re: management fuck-ups).
    - laumars 10y ago
      That's a different point though as you're now effectively using HTTP as your transport layer rather than using your proprietary protocol "naked" on port 443. So whether you send your protocol in a proprietary binary format over HTTP or via JSON API's, it's still just HTTP over TLS. You're still sending HTTP headers, just with our own proprietary content as the body. The real problem is if you want persistent types of connections. As websockets have taught us: a great many equipment that supports HTTP don't support websockets and thus why even websites are still often stuck with RESTful APIs even when that doesn't always fit their problem quite right (that situation is improving however). If one is using HTTP to transport your protocol (and the aforementioned websockets point isn't an issue) then your protocol should work over port 80 as well as 443. Barring any corporate / nationwide web filtering; but as others have already pointed out, that could also be an issue with 443 even with advantage of TLS. By the way, was it Socks2HTTP you worked on? I remember that project back around the time you described and was quite fascinated by it.
      - sly010 10y ago
        > That's a different point though as you're now effectively using HTTP as your transport layer rather than using your proprietary protocol "naked" on port 443. HTTPS is built on top of TLS, not the other way around. You can't (passively) tell if a 443 TLS connection is HTTPS or a proprietary stream. You can take a guess based on statistics (which is what the Chinese firewall does to detect tunneling) but that's about it.
- mjevans 10y ago
  Depends if the network your on is state level and/or corporate ownership of device level MitM or not.

_pmf_ 10y ago

> "and they probably get blocked on a middlebox except if you use port 80 or 443 and fake being HTTP anyway" Curiously, games using UDP still work fine. Stop justifying bad protocols with the proxy straw man.

niftich 10y ago

And as my view in that post foretells, I believe Twitter is already a protocol, we just call it the Twitter API. It's not an open protocol, of course, and its domain model is fairly anemic [1], especially compared to that of Facebook [2][3][4]. But in my opinion, this is what the author is trying to say: Twitter could have been a place where people interact with structured resources created and consumed by other apps. It's a possibility, especially since Facebook became that place instead. Ultimately, Twitter didn't because they wanted to focus on driving more traffic through their first-party app (presumably as a captive client on which they can eventually display ads), and because they focused on cultivating a community (like Medium, Tumblr, LiveJournal) rather than a resource ecosystem. [1] https://dev.twitter.com/rest/public https://dev.twitter.com/rest/public [2] https://developers.facebook.com/docs/graph-api/reference https://developers.facebook.com/docs/graph-api/reference [3] https://developers.facebook.com/docs/sharing/opengraph/using-objects https://developers.facebook.com/docs/sharing/opengraph/using... [4] https://developers.facebook.com/docs/sharing/opengraph/object-api/ https://developers.facebook.com/docs/sharing/opengraph/objec...