4 ms·

Would be great to have an rsync like protocol running in the webbrowser ideally based on webrtc purely P2P. No install needed. I hate all those custom clients f

by riedel 5y ago

Would be great to have an rsync like protocol running in the webbrowser ideally based on webrtc purely P2P. No install needed. I hate all those custom clients for all those cloud storage services. Particularly was looking for an option for a client with a lot of IT restrictions to upload a large number of large files. Embarrassingly often we end up shipping encrypted hard drives.

mkl95 5y ago
I hate cloud clients so much. Every company seems to be using a different one and I haven't seen a single one with good UX.
tyingq 5y ago
The changes you would need to make for HTTP to support a rsync style pull protocol are pretty minimal. Two of them, really. 1) A checksum header in the HEAD response 2) Altering the HEAD method to optionally accept the same range request headers that a GET does, and have the checksum header return the checksum for that range. I'm mildly surprised it hasn't happened. Pretty sure you could do it with vanilla php or the nginx lua module.
- ComputerGuru 5y ago
  The first is already there in the form of a strong etag. Http conditional requests with if-range support etag validation but that is a document checksum not a chunk checksum.
  - tyingq 5y ago
    >The first is already there in the form of a strong etag Unfortunately, there's no standard as to how that's computed, so it wouldn't be useful in an rsync style setup unless you controlled both ends.
    - ComputerGuru 5y ago
      Or stored it as opaque metadata on first retrieval, regardless of how it was computed, paired with the actual hash. When one is invalidated, the other is too.
  - zie 5y ago
    There is also Content-MD5: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.15 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1...
    - riedel 5y ago
      Thanks. Looking up support for this I found dcache.org which supports rfc3230 which supports even more digests.
- zie 5y ago
  You have smattering of disasters, that let you sort of fake it till you make it? :) For 1) You have SRI: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres... You have Etag and you have Content-MD5: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.15 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1... Like you already mentioned etag sucks, because you just get a guarantee X hasn't changed but no way to now how it was calculated. i.e. it is opaque. SRI is only useful for everything outside of the original document(i.e. for <script> tags and the like. Content-MD5 actually works.. except it's using the broken MD5 digest.. so it might work for the use case, depending on the details of the use-case. :) 2 is a no-go as far as I'm aware. Though you could perhaps do an HTML document where you link to each "chunk" of the file using SRI.. though that would be a total bastardization of HTML and a total hack.. and a bad one at that :) Also the chunks would be decided well ahead of time that way, so not the same as a RANGE..
spinax 5y ago
This may be of interest to you: https://rclone.org/ https://rclone.org/ rclone is basically rsync with cloud native endpoints, it's not browser based but does what needs to be done very well. It does have an experimental web GUI but it's not a feature I've tried personally. https://github.com/rclone/rclone-webui-react https://github.com/rclone/rclone-webui-react
- thegagne 5y ago
  Just don’t use it to keep two systems in sync when you pay for transactions as it can be very costly.
hinkley 5y ago
Never underestimate the bandwidth of a station wagon full of data tapes, barreling down the highway.

by riedel 5y ago

mkl95 5y ago

I hate cloud clients so much. Every company seems to be using a different one and I haven't seen a single one with good UX.

tyingq 5y ago

The changes you would need to make for HTTP to support a rsync style pull protocol are pretty minimal. Two of them, really. 1) A checksum header in the HEAD response 2) Altering the HEAD method to optionally accept the same range request headers that a GET does, and have the checksum header return the checksum for that range. I'm mildly surprised it hasn't happened. Pretty sure you could do it with vanilla php or the nginx lua module.

ComputerGuru 5y ago
The first is already there in the form of a strong etag. Http conditional requests with if-range support etag validation but that is a document checksum not a chunk checksum.
- tyingq 5y ago
  >The first is already there in the form of a strong etag Unfortunately, there's no standard as to how that's computed, so it wouldn't be useful in an rsync style setup unless you controlled both ends.
  - ComputerGuru 5y ago
    Or stored it as opaque metadata on first retrieval, regardless of how it was computed, paired with the actual hash. When one is invalidated, the other is too.
- zie 5y ago
  There is also Content-MD5: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.15 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1...
  - riedel 5y ago
    Thanks. Looking up support for this I found dcache.org which supports rfc3230 which supports even more digests.
zie 5y ago
You have smattering of disasters, that let you sort of fake it till you make it? :) For 1) You have SRI: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres... You have Etag and you have Content-MD5: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.15 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1... Like you already mentioned etag sucks, because you just get a guarantee X hasn't changed but no way to now how it was calculated. i.e. it is opaque. SRI is only useful for everything outside of the original document(i.e. for <script> tags and the like. Content-MD5 actually works.. except it's using the broken MD5 digest.. so it might work for the use case, depending on the details of the use-case. :) 2 is a no-go as far as I'm aware. Though you could perhaps do an HTML document where you link to each "chunk" of the file using SRI.. though that would be a total bastardization of HTML and a total hack.. and a bad one at that :) Also the chunks would be decided well ahead of time that way, so not the same as a RANGE..

spinax 5y ago

This may be of interest to you: https://rclone.org/ https://rclone.org/ rclone is basically rsync with cloud native endpoints, it's not browser based but does what needs to be done very well. It does have an experimental web GUI but it's not a feature I've tried personally. https://github.com/rclone/rclone-webui-react https://github.com/rclone/rclone-webui-react

thegagne 5y ago
Just don’t use it to keep two systems in sync when you pay for transactions as it can be very costly.

hinkley 5y ago

Never underestimate the bandwidth of a station wagon full of data tapes, barreling down the highway.