4 ms·

Having media out of band causes some problems but eases others. Having a separate tcp window for media and messages is pretty valuable. Additionally, if media i

by toast0 12d ago

Having media out of band causes some problems but eases others. Having a separate tcp window for media and messages is pretty valuable. Additionally, if media is separate, you might run it in different locations than chat. HTTP may not have native resumption, but it's not hard to do partial upload and resumption over HTTP, you just need agreement on parameters / a protocol to determine where to resume from.

aboardRat4 11d ago
>Having a separate tcp window for media and messages is pretty valuable What for? >t's not hard to LOOOOOOOOL
- ValdikSS 11d ago
  >What for? To prevent head-of-line blocking.
  - aboardRat4 11d ago
    >To prevent head-of-line blocking. Of course using TCP has nothing to do with head-of-line blocking. Only if you blindly do write(socket, fileptr, sizeof(file)), it starts to matter, but, you know, people did not start writing for TCP/IP yesterday. If you have a decent protocol, it splits files (and texts as well) into chunks, and sends them in prioritized order, text chunks having higher priority than binary chunks. The server re-assembles the chunks then. The fact that such a simple way of multiplexing data is a discovery for XMPP fanboys is very revealing.
    - toast0 11d ago
      You can certainly chunk things and interleave/multiplex interactive and bulk data. But it's easy to end up with low throughput, because you limit how much unacked bulk data you send, because you don't want to overqueue bulk data and not be able to immediately write interactive messages. And it's easy to end up with high latency, because despite the limits above, you queued too much and interactive messages need to wait. Or you sent bulk data and there was a burst of packet loss and new data can't be received until the missing packets are resent and received. > people did not start writing for TCP/IP yesterday. And yet you find resuming uploads over http to be too hard?
      - aboardRat4 11d ago
        >But it's easy to end up with low throughput, >And it's easy to end up with high latency, And yet the OS is doing just that when you open two TCP streams. It packs them into ethernet frames and keeps track of the acks. Only the OS does not know the context, so it prioritises both streams equally. (Unless some evil magic is involved.) >And yet you find resuming uploads over http to be too hard? Me? No. Xmpp developers? Apparently yes, as it's neither in any of the XEPs nor in any of the clients.
        toast0 11d ago
        > Only the OS does not know the context, so it prioritises both streams equally. Of course there's prioritization options on socket APIs since forever (DSCP), but usage is limited, because it requires coordination and coordination is hard. Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. If you're lucky the chat connection didn't see any of the loss, and even if you're not, you've likely got a smaller queue on your chat connection than your media connection. This also doesn't prohibit client driven prioritization --- when connecting, get caught up on chat before processing media. Maybe pause media while sending chat messages or if server acks are slow. Server driven prioritization is hard, because probably your http media servers don't have information on the chat connection --- but if a client stops reading from the media connection, the server will get the message eventually.
        aboardRat4 11d ago
        >Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. Well, again, it's merely an implementation detail. You could just open a second TCP connection using the in-band IM protocol, not some external HTTP. Or make your IM protocol UDP based. But really, that's what quic was designed for, which is also not new.
- toast0 11d ago
  >>t's not hard to > LOOOOOOOOL ? I've done it a few times. It worked fine on Nokia S40 and the version of J2ME they run can't even seek backwards in files. It's a three part recipe: a) send a request to upload with whatever auth you need, some stable identifier for the file and the file size. Ideally a nice checksum to confirm the file was not corrupted in transit; TLS should protect you, but I've seen things, a 32-byte sha256 checksum offers protection from a lot of things. b) if the upload is unfinished, you'll get a url (or whatever) to post to and a starting offset (0 on the first time); if the upload is complete, you'll get a download url to send to your correspondent. (or an error like file too big, try again later, go away, whatever) c) upload the file to the url. Maybe get a status if the upload finishes and you're still online to receive it. Status could indicate error or a download url. If you timeout or get a retriable error, go back to part a. It's not rocket science or anything. Maybe it would be hard to get done in the XMPP ecosystem, but it's simple enough to do with any stack that's got big enough media that resuming uploads is relevant.
  - aboardRat4 11d ago
    yeah, yeah, yeah. Now tell your girlfriend's mother to do the same.

by toast0 12d ago

aboardRat4 11d ago

>Having a separate tcp window for media and messages is pretty valuable What for? >t's not hard to LOOOOOOOOL

ValdikSS 11d ago
>What for? To prevent head-of-line blocking.
- aboardRat4 11d ago
  >To prevent head-of-line blocking. Of course using TCP has nothing to do with head-of-line blocking. Only if you blindly do write(socket, fileptr, sizeof(file)), it starts to matter, but, you know, people did not start writing for TCP/IP yesterday. If you have a decent protocol, it splits files (and texts as well) into chunks, and sends them in prioritized order, text chunks having higher priority than binary chunks. The server re-assembles the chunks then. The fact that such a simple way of multiplexing data is a discovery for XMPP fanboys is very revealing.
  - toast0 11d ago
    You can certainly chunk things and interleave/multiplex interactive and bulk data. But it's easy to end up with low throughput, because you limit how much unacked bulk data you send, because you don't want to overqueue bulk data and not be able to immediately write interactive messages. And it's easy to end up with high latency, because despite the limits above, you queued too much and interactive messages need to wait. Or you sent bulk data and there was a burst of packet loss and new data can't be received until the missing packets are resent and received. > people did not start writing for TCP/IP yesterday. And yet you find resuming uploads over http to be too hard?
    - aboardRat4 11d ago
      >But it's easy to end up with low throughput, >And it's easy to end up with high latency, And yet the OS is doing just that when you open two TCP streams. It packs them into ethernet frames and keeps track of the acks. Only the OS does not know the context, so it prioritises both streams equally. (Unless some evil magic is involved.) >And yet you find resuming uploads over http to be too hard? Me? No. Xmpp developers? Apparently yes, as it's neither in any of the XEPs nor in any of the clients.
      - toast0 11d ago
        > Only the OS does not know the context, so it prioritises both streams equally. Of course there's prioritization options on socket APIs since forever (DSCP), but usage is limited, because it requires coordination and coordination is hard. Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. If you're lucky the chat connection didn't see any of the loss, and even if you're not, you've likely got a smaller queue on your chat connection than your media connection. This also doesn't prohibit client driven prioritization --- when connecting, get caught up on chat before processing media. Maybe pause media while sending chat messages or if server acks are slow. Server driven prioritization is hard, because probably your http media servers don't have information on the chat connection --- but if a client stops reading from the media connection, the server will get the message eventually.
        aboardRat4 11d ago
        >Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. Well, again, it's merely an implementation detail. You could just open a second TCP connection using the in-band IM protocol, not some external HTTP. Or make your IM protocol UDP based. But really, that's what quic was designed for, which is also not new.
toast0 11d ago
>>t's not hard to > LOOOOOOOOL ? I've done it a few times. It worked fine on Nokia S40 and the version of J2ME they run can't even seek backwards in files. It's a three part recipe: a) send a request to upload with whatever auth you need, some stable identifier for the file and the file size. Ideally a nice checksum to confirm the file was not corrupted in transit; TLS should protect you, but I've seen things, a 32-byte sha256 checksum offers protection from a lot of things. b) if the upload is unfinished, you'll get a url (or whatever) to post to and a starting offset (0 on the first time); if the upload is complete, you'll get a download url to send to your correspondent. (or an error like file too big, try again later, go away, whatever) c) upload the file to the url. Maybe get a status if the upload finishes and you're still online to receive it. Status could indicate error or a download url. If you timeout or get a retriable error, go back to part a. It's not rocket science or anything. Maybe it would be hard to get done in the XMPP ecosystem, but it's simple enough to do with any stack that's got big enough media that resuming uploads is relevant.
- aboardRat4 11d ago
  yeah, yeah, yeah. Now tell your girlfriend's mother to do the same.