3 ms·

Show HN: Syq – copy files between machines fast (better than rsync)

41 points by greaber 23d ago

I got frustrated with the slowness of rsync and made an alternative that works faster by using multiple parallel connections, direct encrypted TCP when available, and other optimizations. I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs, the ability to download to your laptop while working in an ssh shell on a server, and the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). It is also designed to be more robustly/flexibly scriptable than alternatives.

mika6996 23d ago
Why is this better than rsync?
- greaber 23d ago
  There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.
  - amelius 23d ago
    There's something vastly superior over rsync still: "btrfs send" and "btrfs receive" (but require you to have btrfs on source and destination).
    - greaber 23d ago
      Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
      - gcr 23d ago
        'btrfs send' is more like diff/patch than it is like rsync. It requires both source and destination to agree about the source snapshot.
      - amelius 23d ago
        btrfs send is useful for sending incremental updates; this is used a LOT for backups; and it saves a lot of time because it does not need to traverse files/directories that were not changed You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.
  - craftkiller 23d ago
    One instance where rsync's delta merge comes up for me: For all of my virtual machines, I build a bootable live ISO so that way I can cleanly swap out the full ISO with a new one when there are software updates. This means that every time I want to change a config file inside the virtual machine, I end up building a new ISO and rsyncing it to my server. rsync speeds this up significantly since most (but not all) of the ISO is unchanged.
    - dwedge 23d ago
      How do you handle updates this way? Do you mount the iso, update and keep it as an iso? Are all of the virtual machines mostly read only?
      - craftkiller 23d ago
        The iso itself is read-only. When I need to update software I build a whole new iso and replace the existing one. For any folders/files that need persistence across reboots, I store that data separately from the iso, either in a virtual nvme drive or a mounted 9pfs folder from the host. The iso mounts the persistence drive/9pfs via /etc/fstab. So in the end, the software and config files are read-only (until I replace them with a new iso) but my databases and data files are read-write. Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence. I build it all via NixOS and I use Impermanence to manage the bind mounts to the persist drive: https://wiki.nixos.org/wiki/Impermanence https://wiki.nixos.org/wiki/Impermanence
    - greaber 23d ago
      Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)
  - Someone 23d ago
    > Note that syq doesn't currently implement rsync's delta merge algorithm That, and still claiming it’s “better than rsync”? Seems like you shouldn’t compare it with rsync (yet, if doing delta merges is planned)
    - greaber 23d ago
      Yeah, I kind of regret writing "better than rsync" in the tagline. Maybe I should have written "better than rsync in some respects".
- nimih 23d ago
  Browsing the repo, I can see a couple of advantages: - Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in. - Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance. - UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
  - greaber 23d ago
    - No need to be snarky. It's a new project, while rsync is time tested, and that is certainly an advantage for rsync and all old software. - You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me. - I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
    - ProphetOfParado 23d ago
      How do you hope to convince sys-admins to install this? I work with clusters where I do not have superuser access, so this is a no go for me.
cynicalsecurity 23d ago
I'm eager to replace battle-tested rsync with a new half-baked tool that no one uses for a tiny speed improvement. I'm also going to enable automatic updates of that tool too.
- xz18r 23d ago
  You live up to your username!
- greaber 23d ago
  I'm seeing 5x speedups on my real workloads, and you can see some synthetic benchmarks here or run your own https://greaber.github.io/syq-bench/ https://greaber.github.io/syq-bench/
  - throw1234567891 23d ago
    what's a "synthetic benchmark", claude came up with those numbers by looking at the code?
    - greaber 23d ago
      They are benchmarks on real hardware, and I tried to provide enough details so that anyone can reproduce them (but some of the benchmarks are on servers I have that you can't order on public cloud -- these ones are clearly marked). "Synthetic" means that the data is synthetic, not an actual production workload. For instance, some of the benchmarks use files filled with random bytes.
      - beepbooptheory 23d ago
        Do non-randomly ordered bytes produce different results??
        greaber 23d ago
        Syq has compression turned on by default. Also, if you are updating a file that already exists on the other end, syq will only send blocks of the file that changed.
hikarudo 23d ago
Very cool! How does the direct TCP mode work?
- greaber 23d ago
  Currently, there is just a fixed range of ports that it will use if they are open and you don't pass `--no-tcp`. It does its own encryption over the TCP connections. Also worth knowing is that for some connections (e.g. if there are any dropped packets), you may get much better performance if you can enable BBR congestion control. https://greaber.github.io/syq/server-tuning.html#test-congestion-control https://greaber.github.io/syq/server-tuning.html#test-conges...
hxseven 23d ago
I thought I'd share a more positive comment. I work on a tool in a similar area (moving stuff around) but on a higher level. I currently use rsync or rclone for the file moving parts. So to me your tool looks interesting and promising, thanks for sharing it :) Some suggestions for improvement for your website/Github: - I wanted to understand how it differs from rsync. A comparison page would be helpful. - A comparison to rclone would also be interesting, as it also uses multiple parallel connections. - As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.
- greaber 23d ago
  Thanks! Yeah, I was a little surprised (maybe I shouldn't have been) at how negative some of the comments were. I will think about what additional explanations I can provide that will help people. Regarding rclone, I have also spent a bunch of time looking at faster transfers to and from S3-compatible storage, but that problem is somewhat different, and I didn't look in depth at how rclone is implemented. FWIW, in my experience, s5cmd is generally a bit faster than rclone even if you tune rclone options. (But rclone is more flexible.) My guess is that s5cmd is already close to the performance ceiling, but I don't really know. As for edge cases, I will also try to document that more. In general, when I'm not sure what semantics to go for, I try to either copy what rsync does or do something safer. For instance, one thing I am looking at now is the best way to handle cases where a source path cannot be represented on the target filesystem or where two source paths would collide (e.g. due to unicode normalization or case insensitivity). Tentatively, my inclination is to try to fail before copying any files when possible. Currently, syq doesn't do this (but neither does rsync).
  - rdbl27 23d ago
    Don't despair; new tool rollouts are always a hard sell -- especially when replacing a beloved classic. For me, the killer feature would be "ease of use." Sure, I _can_ make rsync do any number of rarely-used optimizations, if I feel like studying the manpage for half an hour and figuring out how to fit it to my exact use case. If your tool has the same features but is automatically adaptive -- I'd use it. Copying thousands of tiny files in deeply nested subdirs? Just works. Copying a huge file that's already encrypted? Just works. Copying a mix? Just works. No special flags to set, zero config. It just works, optimally, every time. "Reducing the user's cognitive load" is the killer feature here.
unsnap_biceps 23d ago
> I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's. > the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.
- greaber 23d ago
  Yeah, ControlMaster is cool and was an inspiration, but since the syq client is always talking to the same receiver on the server, we can save a few round trips that ControlMaster still needs, so syq persist feels faster. This architecture is also what enables reverse mode, where you can download to your laptop while working on the server. ProxyJump doesn't help when you want to copy files from server A to server B without giving server A an agent that can do arbitrary things on server B. There is really no alternative to using a restricted authorized key on server B for this scenario.
AceJohnny2 23d ago
Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to. As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?
- greaber 23d ago
  I entirely agree that multiple connections shouldn't actually be necessary for speed, but they help in multiple situations (long-distance transfers, same DC between servers, NFS), and it's not just about amortizing per-file setup overhead. There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful. I will work on one. On the other hand, I also tried hard to make it just go fast without needing the user to understand why it is fast or tune anything. For instance, the number of connections is auto-tuned by default.
  - AceJohnny2 23d ago
    > There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful First, thank you for taking the feedback :) To be clear, I'm focusing on the first thing you should be telling users. It seems that, as a CLI program, the target audience of this tool is fairly technical users who are likely already familiar with SSH, rsync, and everything. As demonstrated in this thread, your first communication task is to convince them why it's worth their time to use your tool, considering they're probably satisfied with the pre-existing ones (or a replacement would already exist!). My point is, explaining why it's better/faster than rsync shouldn't be buried in the docs. It should be a paragraph on the front page. Hope this helps, and keep it up. HN is a rough audience to go public on ^^;
  - fmajid 22d ago
    Multiple connections totally help. At a previous job, we drove rsync with GNU parallel, then did a final catch-all pass with rsync, and that divided transfer time for about 600GB by a factor of 4.
antii 23d ago
[dead]
jttnr 23d ago
Open page, read headline, read first sentence, emdash, close page.
Zenul_Abidin 23d ago
Wow, interesting! How is the performance compared to rclone though?
- greaber 22d ago
  I did some benchmarks against rclone today. https://greaber.github.io/syq-bench/rclone.html https://greaber.github.io/syq-bench/rclone.html Syq was sometimes much faster and never more than a few percent slower. I didn't do any tuning of syq since it is meant to perform well without any special options, but I did tune a couple of rclone options. It's possible that more extensive rclone tuning would yield better results though. Please let me know if there is anything I need to try!
QuiCasseRien 22d ago
rclone has been the real fully featured ultra power alternative to rsync over the past 4-5 years. How do you compare Syq to rclone ?
- greaber 22d ago
  As for speed, see https://greaber.github.io/syq-bench/rclone.html https://greaber.github.io/syq-bench/rclone.html. As for other stuff, it depends what you use rclone for.
vancekai 22d ago
[dead]

41 points by greaber 23d ago

mika6996 23d ago

Why is this better than rsync?

greaber 23d ago
There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.
- amelius 23d ago
  There's something vastly superior over rsync still: "btrfs send" and "btrfs receive" (but require you to have btrfs on source and destination).
  - greaber 23d ago
    Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
    - gcr 23d ago
      'btrfs send' is more like diff/patch than it is like rsync. It requires both source and destination to agree about the source snapshot.
    - amelius 23d ago
      btrfs send is useful for sending incremental updates; this is used a LOT for backups; and it saves a lot of time because it does not need to traverse files/directories that were not changed You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.
- craftkiller 23d ago
  One instance where rsync's delta merge comes up for me: For all of my virtual machines, I build a bootable live ISO so that way I can cleanly swap out the full ISO with a new one when there are software updates. This means that every time I want to change a config file inside the virtual machine, I end up building a new ISO and rsyncing it to my server. rsync speeds this up significantly since most (but not all) of the ISO is unchanged.
  - dwedge 23d ago
    How do you handle updates this way? Do you mount the iso, update and keep it as an iso? Are all of the virtual machines mostly read only?
    - craftkiller 23d ago
      The iso itself is read-only. When I need to update software I build a whole new iso and replace the existing one. For any folders/files that need persistence across reboots, I store that data separately from the iso, either in a virtual nvme drive or a mounted 9pfs folder from the host. The iso mounts the persistence drive/9pfs via /etc/fstab. So in the end, the software and config files are read-only (until I replace them with a new iso) but my databases and data files are read-write. Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence. I build it all via NixOS and I use Impermanence to manage the bind mounts to the persist drive: https://wiki.nixos.org/wiki/Impermanence https://wiki.nixos.org/wiki/Impermanence
  - greaber 23d ago
    Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)
- Someone 23d ago
  > Note that syq doesn't currently implement rsync's delta merge algorithm That, and still claiming it’s “better than rsync”? Seems like you shouldn’t compare it with rsync (yet, if doing delta merges is planned)
  - greaber 23d ago
    Yeah, I kind of regret writing "better than rsync" in the tagline. Maybe I should have written "better than rsync in some respects".
nimih 23d ago
Browsing the repo, I can see a couple of advantages: - Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in. - Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance. - UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
- greaber 23d ago
  - No need to be snarky. It's a new project, while rsync is time tested, and that is certainly an advantage for rsync and all old software. - You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me. - I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
  - ProphetOfParado 23d ago
    How do you hope to convince sys-admins to install this? I work with clusters where I do not have superuser access, so this is a no go for me.

cynicalsecurity 23d ago

I'm eager to replace battle-tested rsync with a new half-baked tool that no one uses for a tiny speed improvement. I'm also going to enable automatic updates of that tool too.

xz18r 23d ago
You live up to your username!
greaber 23d ago
I'm seeing 5x speedups on my real workloads, and you can see some synthetic benchmarks here or run your own https://greaber.github.io/syq-bench/ https://greaber.github.io/syq-bench/
- throw1234567891 23d ago
  what's a "synthetic benchmark", claude came up with those numbers by looking at the code?
  - greaber 23d ago
    They are benchmarks on real hardware, and I tried to provide enough details so that anyone can reproduce them (but some of the benchmarks are on servers I have that you can't order on public cloud -- these ones are clearly marked). "Synthetic" means that the data is synthetic, not an actual production workload. For instance, some of the benchmarks use files filled with random bytes.
    - beepbooptheory 23d ago
      Do non-randomly ordered bytes produce different results??
      - greaber 23d ago
        Syq has compression turned on by default. Also, if you are updating a file that already exists on the other end, syq will only send blocks of the file that changed.

hikarudo 23d ago

Very cool! How does the direct TCP mode work?

greaber 23d ago
Currently, there is just a fixed range of ports that it will use if they are open and you don't pass `--no-tcp`. It does its own encryption over the TCP connections. Also worth knowing is that for some connections (e.g. if there are any dropped packets), you may get much better performance if you can enable BBR congestion control. https://greaber.github.io/syq/server-tuning.html#test-congestion-control https://greaber.github.io/syq/server-tuning.html#test-conges...

hxseven 23d ago

I thought I'd share a more positive comment. I work on a tool in a similar area (moving stuff around) but on a higher level. I currently use rsync or rclone for the file moving parts. So to me your tool looks interesting and promising, thanks for sharing it :) Some suggestions for improvement for your website/Github: - I wanted to understand how it differs from rsync. A comparison page would be helpful. - A comparison to rclone would also be interesting, as it also uses multiple parallel connections. - As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.

greaber 23d ago
Thanks! Yeah, I was a little surprised (maybe I shouldn't have been) at how negative some of the comments were. I will think about what additional explanations I can provide that will help people. Regarding rclone, I have also spent a bunch of time looking at faster transfers to and from S3-compatible storage, but that problem is somewhat different, and I didn't look in depth at how rclone is implemented. FWIW, in my experience, s5cmd is generally a bit faster than rclone even if you tune rclone options. (But rclone is more flexible.) My guess is that s5cmd is already close to the performance ceiling, but I don't really know. As for edge cases, I will also try to document that more. In general, when I'm not sure what semantics to go for, I try to either copy what rsync does or do something safer. For instance, one thing I am looking at now is the best way to handle cases where a source path cannot be represented on the target filesystem or where two source paths would collide (e.g. due to unicode normalization or case insensitivity). Tentatively, my inclination is to try to fail before copying any files when possible. Currently, syq doesn't do this (but neither does rsync).
- rdbl27 23d ago
  Don't despair; new tool rollouts are always a hard sell -- especially when replacing a beloved classic. For me, the killer feature would be "ease of use." Sure, I _can_ make rsync do any number of rarely-used optimizations, if I feel like studying the manpage for half an hour and figuring out how to fit it to my exact use case. If your tool has the same features but is automatically adaptive -- I'd use it. Copying thousands of tiny files in deeply nested subdirs? Just works. Copying a huge file that's already encrypted? Just works. Copying a mix? Just works. No special flags to set, zero config. It just works, optimally, every time. "Reducing the user's cognitive load" is the killer feature here.

unsnap_biceps 23d ago

> I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's. > the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.

greaber 23d ago
Yeah, ControlMaster is cool and was an inspiration, but since the syq client is always talking to the same receiver on the server, we can save a few round trips that ControlMaster still needs, so syq persist feels faster. This architecture is also what enables reverse mode, where you can download to your laptop while working on the server. ProxyJump doesn't help when you want to copy files from server A to server B without giving server A an agent that can do arbitrary things on server B. There is really no alternative to using a restricted authorized key on server B for this scenario.

AceJohnny2 23d ago

Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to. As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?

greaber 23d ago
I entirely agree that multiple connections shouldn't actually be necessary for speed, but they help in multiple situations (long-distance transfers, same DC between servers, NFS), and it's not just about amortizing per-file setup overhead. There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful. I will work on one. On the other hand, I also tried hard to make it just go fast without needing the user to understand why it is fast or tune anything. For instance, the number of connections is auto-tuned by default.
- AceJohnny2 23d ago
  > There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful First, thank you for taking the feedback :) To be clear, I'm focusing on the first thing you should be telling users. It seems that, as a CLI program, the target audience of this tool is fairly technical users who are likely already familiar with SSH, rsync, and everything. As demonstrated in this thread, your first communication task is to convince them why it's worth their time to use your tool, considering they're probably satisfied with the pre-existing ones (or a replacement would already exist!). My point is, explaining why it's better/faster than rsync shouldn't be buried in the docs. It should be a paragraph on the front page. Hope this helps, and keep it up. HN is a rough audience to go public on ^^;
- fmajid 22d ago
  Multiple connections totally help. At a previous job, we drove rsync with GNU parallel, then did a final catch-all pass with rsync, and that divided transfer time for about 600GB by a factor of 4.

antii 23d ago

[dead]

jttnr 23d ago

Open page, read headline, read first sentence, emdash, close page.

Zenul_Abidin 23d ago

Wow, interesting! How is the performance compared to rclone though?

greaber 22d ago
I did some benchmarks against rclone today. https://greaber.github.io/syq-bench/rclone.html https://greaber.github.io/syq-bench/rclone.html Syq was sometimes much faster and never more than a few percent slower. I didn't do any tuning of syq since it is meant to perform well without any special options, but I did tune a couple of rclone options. It's possible that more extensive rclone tuning would yield better results though. Please let me know if there is anything I need to try!

QuiCasseRien 22d ago

rclone has been the real fully featured ultra power alternative to rsync over the past 4-5 years. How do you compare Syq to rclone ?

greaber 22d ago
As for speed, see https://greaber.github.io/syq-bench/rclone.html https://greaber.github.io/syq-bench/rclone.html. As for other stuff, it depends what you use rclone for.

vancekai 22d ago

[dead]