9 ms·

Openrsync imported into the tree

138 points by protomyth 8y ago

hawski 8y ago
From a comment on the site: "(...) its (original rsync's) compressed manual page is almost as big as the compressed openrsync sources (...)" It's license (ISC ofc.) and size makes it great resource to study rsync. I would like to have Dropbox on my phone as legendary combination of rsync and cron. It may be nice to have a port to Java so it would work without JNI, but maybe that's only my fetish.
- ComputerGuru 8y ago
  I just want to point out that rsync is, in fact, no longer ISC licensed but rather GPL (v3, at that), which is likely a big part of the reason this new implementation even exists.
  - meruru 8y ago
    rsync was never ISC licensed afaik. The parent is referring to openrsync's license.
    - tinus_hn 8y ago
      Rsync was developed by the Samba people, it is under the same license (GPL).
amaccuish 8y ago
For those wondering what this is, see https://github.com/kristapsdz/openrsync https://github.com/kristapsdz/openrsync
- benatkin 8y ago
  I'll try explaining it. It's a new implementation, from scratch (clean room) of rsync, which will become the new rsync in OpenBSD. The tree that it's been imported into is the openbsd cvs tree that contains openbsd, openssh, opencvs, and other major projects.
geofft 8y ago
> The actual work of porting, however, is matching the security features provided by OpenBSD's pledge(2) and unveil(2). These are critical elements to the functionality of the system. Without them, your system accepts arbitrary data from the public network. ... rsync has specific running modes for the super-user. It also pumps arbitrary data from the network onto your file-system. Do you want that running without specific mitigation in place? This is a confusing claim. What exactly does "accepts arbitrary data from the public network" mean? (Most servers do that, they just choose not to process the data without additional validation.) And in what way is it critical to the functionality of the system? Is the claim that, after calling pledge() and unveil(), the openrsync process is happy to satisfy arbitrary read/write requests from the other side of the connection, and so without them it is insecure? Does openrsync view peer-induced memory corruption after pledge() or unveil() as a vulnerability? Or is the idea that the attacker can already "pump arbitrary data from the network onto your filesystem" and that the attacker gaining control flow is not a meaningful escalation of privileges? My impression is that pledge() and unveil() are hardening tools, intended to limit the damage from a process that has already gotten out of control (in the same way that e.g. running Apache as non-root does not mean that you're actively fine letting attackers run code as www-data). Is that impression wrong? Is openrsync using them for the basic functionality of making sure that a file is only being rsynced to the filename given on the command line?
- admax88q 8y ago
  You're not wrong but the point being made is that wouldn't you want a tool which writes data from the network to disk to have those mitigations enabled?
- aidenn0 8y ago
  It is possible to read the readme[1] to imply that unveil is the only protection from escaping the root (e.g. with a ".." directory). The only way to know for sure is to dig through the code though. https://github.com/kristapsdz/openrsync https://github.com/kristapsdz/openrsync
- anjbe 8y ago
  I’m trying to understand your question. Typically, a process once compromised can do all sorts of things: touch files, access the network, execute programs, and so on. Among other things, OpenBSD’s security culture focuses on mitigating the damage done by compromised code through development practices such as privilege separation. Traditionally this was done by splitting functionality into multiple processes, each serving a specific purpose such as doing network communication or parsing configuration, and dropping privileges in any way possible such as chrooting and switching to a dedicated user. Thus the attack surface is reduced, and the potential damage done by a compromised (sub‐)process is reduced as well. pledge() and unveil() are the latest evolution in OpenBSD’s technique. pledge() whitelists syscalls, and unveil whitelists files that can be accessed. So your process reads this arbitrary data from the public network. You validate it through some function and pass the data on to the next stage of your program. But what if there’s a bug in your validator, and your process gets compromised? If your process hasn’t had its capabilities reduced, the attacker can do practically anything, especially if the process has superuser privileges. But if the program uses a multi‐process privilege‐separated architecture, your validation process can’t access the filesystem or the network and isn’t running as root. If it tries, the kernel will kill it for violating its pledge. All the compromised process can do is pass malicious data through whatever interface you’ve provided between your validator and filesystem processes, hopefully an interface that is simple, well‐defined, and well‐audited. What if your filesystem process gets compromised? With pledge() it can’t access the network or execute external code. With unveil(), even its file accesses are limited to the files whitelisted earlier in the program. It can’t read your SSH keys or delete your photos. Certainly, if the process can be compromised that’s a bug that needs to be fixed. But we see new bugs constantly in the software we use every day. It’s a safe bet to say we will encounter more. By using a secure architecture, the damage these bugs can cause is drastically reduced. There’s a really good description and demonstration of privilege separation in another project by Kristaps, acme-client (a Let’s Encrypt/certbot alternative): https://kristaps.bsd.lv/acme-client/ https://kristaps.bsd.lv/acme-client/ Another such project is Google Chrome, which uses pledge() and unveil() on OpenBSD.
theamk 8y ago
It is interesting that "open" part of openrsync refers to license -- BSD, vs original rsync's GPL It's not often I see "open" to mean "non-GPL" in software :)
- zdw 8y ago
  rsync went GPLv3 a while ago, and many businesses don't trust some of the newer clauses that were added. Similarly, the more strict strict BSD crowd has issues with the Apache2 license clauses regarding revocation - see here: https://www.openbsd.org/policy.html https://www.openbsd.org/policy.html
- protomyth 8y ago
  I get the feeling the "open" part was because they were hoping to get it included in OpenBSD like OpenSMTPD, etc.
  - meruru 8y ago
    Yeah, that was my assumption too. It's coming from the OpenBSD community, so openrsync it is.
- enneff 8y ago
  Stallman himself makes a big fuss about "Open Source" not being equivalent to "Free Software". See: https://www.gnu.org/philosophy/open-source-misses-the-point.en.html https://www.gnu.org/philosophy/open-source-misses-the-point....
- anjbe 8y ago
  Fun bit of history: the “Open” here comes from OpenBSD; but the “Open” in OpenBSD came from the development process, not the license. Before Git and SVN, we had CVS, and to check out code from a CVS repository you needed to have an account on the CVS server. If you wanted to contribute but didn’t already have a developer account, you were limited to writing patches against release tarballs or whatever alternative method upstream supplied. One of OpenBSD’s major projects in the mid 90s was creating anonymous CVS, where anyone could check out code without any account. This came from Theo’s experience after losing his NetBSD account, where he found himself unable to make meaningful contributions anymore without the ability to cvs checkout, cvs diff, etc. So when he started OpenBSD, he had in mind to open up the development process to everyone, account or not. This is described in the commentary for the OpenBSD 6.1 release song: https://www.openbsd.org/lyrics.html#61 https://www.openbsd.org/lyrics.html#61
AdmiralAsshat 8y ago
Interesting. This is the first project I can think of where a clean-room implementation was done so that a project could use a less free license ("free" as defined by the FSF). Does anyone else know of instances where a company did a clean-room implementation of a previously FOSS tool so that they could make a paid/proprietary version? Usually it goes the other way.
- protomyth 8y ago
  How is the ISC (version of BSD license used by OpenBSD) less free than the GPL3? This is very far from a "paid/proprietary" version.
  - mouldysammich 8y ago
    Its much closer to a proprietary version than a GPL version would or could be however.
    - kevin_thibedeau 8y ago
      Proprietary-friendly is not less-free.
      - StavrosK 8y ago
        How not?
        GalacticDomin8r 8y ago
        Because you are actually free to do what you want with it, instead of free to do what someone else wants you to do with it.
        StavrosK 8y ago
        Yes, you are free to close it up and sell it, but everyone else then isn't free to use your changes. "More friendly to proprietary purposes" is "less free". It's kind of like arguing that a country where anyone can steal from anyone else with impunity is more free. Not when you consider the rights of the person being stolen from.
meruru 8y ago
I hope this ends up being a lot simpler and easier to understand than the original rsync. The rsync manpage is way too long.
- gmueckl 8y ago
  Rsunc solved a complex problem that comes in many nuanced variants. It may seem trivial at the outset, but it is actually not. So I don't think that rsync has many features that are somehow unnecessary or bloat.
  - meruru 8y ago
    Well, the manpage for this is looking really good and it already has almost everything that I care about. The -a option isn't in yet, but it's in the TODO. https://github.com/kristapsdz/openrsync/blob/master/openrsync.1 https://github.com/kristapsdz/openrsync/blob/master/openrsyn... https://github.com/kristapsdz/openrsync/blob/master/TODO.md https://github.com/kristapsdz/openrsync/blob/master/TODO.md I hope the -c and maybe -X option make it.
- m0nty 8y ago
  > The rsync manpage is way too long I see its thoroughness as a feature, not a bug. It's very well written and I can just ignore the bits I'm not interested in. I wish more man pages were "too long" like this one is.
accrual 8y ago
Very cool news. rsync(1) is one of the first things I install on a new OpenBSD instance. Tangentially related, I've been using Time Machine-like wrapper [0] around rsync(1) for a few years. It's very helpful for maintaining snapshots of my home directory. [0] https://blog.interlinked.org/tutorials/rsync_time_machine.html https://blog.interlinked.org/tutorials/rsync_time_machine.ht...
- davewongillies 8y ago
  I use rsnapshot [0] for the same thing. [0] https://rsnapshot.org/ https://rsnapshot.org/
gerdesj 8y ago
Is it any better than rsync?
- yjftsjthsd-h 8y ago
  It is better in some ways, and worse in some, both largely subjective. It has a different license, is smaller, less battle tested, from different developers, designed with different goals in mind. It Depends™ on how you judge.
- rstuart4133 8y ago
  All openrsync implements is the equivalent of a fast "cp -a" across the network, plus it can also remove files if they don't exist. rsync does much more and over the years I've used most of it, so there is no way I would use openrsync. The upside is the manpage of openrsync isn't that much more complex than cp, which is a definite bonus if that's all you are doing. The only thing I would change about rsync is it's default, which IMO should be to copy all meta data supported by both sides. Ie, the default should be to make the destination as similar to the source as possible. It's default is to only copy the data, and you must add options to say what else you want copied. To make matters worse you can just add every option because if you say you want to copy something not supported by one side of the other it errors out. I may have missed it as I am reading the man page source, but openrsync didn't seem to change that.
  - kristapsdz 8y ago
    No. openrsync implements the rsync protocol. It doesn't have all of its options, but the protocol is what it is. Do you have any idea what you're talking about?
CaliforniaKarl 8y ago
I would not be surprised if, in a few years, this becomes one of the CLI tools installed on macOS, either as part of the default install or as part of the Xcode CLI tools.
- riffraff 8y ago
  Why? MacOS has a bunch of GPL stuff, such as bash, IIRC.
  - __david__ 8y ago
    And it already has actual rsync.
    - avar 8y ago
      It has 12 year old rsync due to Apple not wanting to ship anything that's GPLv3: https://bayton.org/2018/07/how-to-update-rsync-on-mac-os-high-sierra/ https://bayton.org/2018/07/how-to-update-rsync-on-mac-os-hig...
  - perbu 8y ago
    GPL2, I doubt you'll find any GPL3 code in there. Which is why bash on MacOS is from 2007.
- gpvos 8y ago
  Does macOS have any security features similar to pledge/unveil or any of the Linux hardening packages?
  - Fnoord 8y ago
    It has a port of PF.
    - gpvos 8y ago
      I'm more interested to know about system call and filesystem access restrictors. I think pf is only a packet filter.
joppy 8y ago
What does a "clean-room implementation" mean?
- anjbe 8y ago
  Not derived from the existing code. The reason it’s mentioned is to assert that openrsync is not subject to the original rsync’s GPL.
- Tor3 8y ago
  The first (well-known) 'clean-room' implementation was when Phoenix implemented an IBM PC-compatible BIOS by having one team studying the IBM source (which was available), then writing up a specification for how it worked, handing that specification over to somebody else (they were Phoenix' legal team, IIRC), which then handed the specs over to another team that had never seen the IBM source. They sat down in their "clean room" (b/c it wasn't tainted by actual IBM source) and implemented a BIOS from specs only. In that way Phoenix was protected from any claims of copyright infringement: Nothing was copied, and the people writing the code had never seen the original source. In that particular case the specs were reverse-engineered from actual source, but that's not a necessary part of the process. It's more common to have one team study the protocol, data going over the wire, disassembling, etc, then use the knowledge gained to write specs, and then another team implements the equivalent functionality from specifications only.

138 points by protomyth 8y ago

From a comment on the site: "(...) its (original rsync's) compressed manual page is almost as big as the compressed openrsync sources (...)" It's license (ISC ofc.) and size makes it great resource to study rsync. I would like to have Dropbox on my phone as legendary combination of rsync and cron. It may be nice to have a port to Java so it would work without JNI, but maybe that's only my fetish.

ComputerGuru 8y ago
I just want to point out that rsync is, in fact, no longer ISC licensed but rather GPL (v3, at that), which is likely a big part of the reason this new implementation even exists.
- meruru 8y ago
  rsync was never ISC licensed afaik. The parent is referring to openrsync's license.
  - tinus_hn 8y ago
    Rsync was developed by the Samba people, it is under the same license (GPL).

amaccuish 8y ago

For those wondering what this is, see https://github.com/kristapsdz/openrsync https://github.com/kristapsdz/openrsync

benatkin 8y ago
I'll try explaining it. It's a new implementation, from scratch (clean room) of rsync, which will become the new rsync in OpenBSD. The tree that it's been imported into is the openbsd cvs tree that contains openbsd, openssh, opencvs, and other major projects.

geofft 8y ago

> The actual work of porting, however, is matching the security features provided by OpenBSD's pledge(2) and unveil(2). These are critical elements to the functionality of the system. Without them, your system accepts arbitrary data from the public network. ... rsync has specific running modes for the super-user. It also pumps arbitrary data from the network onto your file-system. Do you want that running without specific mitigation in place? This is a confusing claim. What exactly does "accepts arbitrary data from the public network" mean? (Most servers do that, they just choose not to process the data without additional validation.) And in what way is it critical to the functionality of the system? Is the claim that, after calling pledge() and unveil(), the openrsync process is happy to satisfy arbitrary read/write requests from the other side of the connection, and so without them it is insecure? Does openrsync view peer-induced memory corruption after pledge() or unveil() as a vulnerability? Or is the idea that the attacker can already "pump arbitrary data from the network onto your filesystem" and that the attacker gaining control flow is not a meaningful escalation of privileges? My impression is that pledge() and unveil() are hardening tools, intended to limit the damage from a process that has already gotten out of control (in the same way that e.g. running Apache as non-root does not mean that you're actively fine letting attackers run code as www-data). Is that impression wrong? Is openrsync using them for the basic functionality of making sure that a file is only being rsynced to the filename given on the command line?

admax88q 8y ago
You're not wrong but the point being made is that wouldn't you want a tool which writes data from the network to disk to have those mitigations enabled?
aidenn0 8y ago
It is possible to read the readme[1] to imply that unveil is the only protection from escaping the root (e.g. with a ".." directory). The only way to know for sure is to dig through the code though. https://github.com/kristapsdz/openrsync https://github.com/kristapsdz/openrsync
anjbe 8y ago
I’m trying to understand your question. Typically, a process once compromised can do all sorts of things: touch files, access the network, execute programs, and so on. Among other things, OpenBSD’s security culture focuses on mitigating the damage done by compromised code through development practices such as privilege separation. Traditionally this was done by splitting functionality into multiple processes, each serving a specific purpose such as doing network communication or parsing configuration, and dropping privileges in any way possible such as chrooting and switching to a dedicated user. Thus the attack surface is reduced, and the potential damage done by a compromised (sub‐)process is reduced as well. pledge() and unveil() are the latest evolution in OpenBSD’s technique. pledge() whitelists syscalls, and unveil whitelists files that can be accessed. So your process reads this arbitrary data from the public network. You validate it through some function and pass the data on to the next stage of your program. But what if there’s a bug in your validator, and your process gets compromised? If your process hasn’t had its capabilities reduced, the attacker can do practically anything, especially if the process has superuser privileges. But if the program uses a multi‐process privilege‐separated architecture, your validation process can’t access the filesystem or the network and isn’t running as root. If it tries, the kernel will kill it for violating its pledge. All the compromised process can do is pass malicious data through whatever interface you’ve provided between your validator and filesystem processes, hopefully an interface that is simple, well‐defined, and well‐audited. What if your filesystem process gets compromised? With pledge() it can’t access the network or execute external code. With unveil(), even its file accesses are limited to the files whitelisted earlier in the program. It can’t read your SSH keys or delete your photos. Certainly, if the process can be compromised that’s a bug that needs to be fixed. But we see new bugs constantly in the software we use every day. It’s a safe bet to say we will encounter more. By using a secure architecture, the damage these bugs can cause is drastically reduced. There’s a really good description and demonstration of privilege separation in another project by Kristaps, acme-client (a Let’s Encrypt/certbot alternative): https://kristaps.bsd.lv/acme-client/ https://kristaps.bsd.lv/acme-client/ Another such project is Google Chrome, which uses pledge() and unveil() on OpenBSD.

theamk 8y ago

It is interesting that "open" part of openrsync refers to license -- BSD, vs original rsync's GPL It's not often I see "open" to mean "non-GPL" in software :)

zdw 8y ago
rsync went GPLv3 a while ago, and many businesses don't trust some of the newer clauses that were added. Similarly, the more strict strict BSD crowd has issues with the Apache2 license clauses regarding revocation - see here: https://www.openbsd.org/policy.html https://www.openbsd.org/policy.html
protomyth 8y ago
I get the feeling the "open" part was because they were hoping to get it included in OpenBSD like OpenSMTPD, etc.
- meruru 8y ago
  Yeah, that was my assumption too. It's coming from the OpenBSD community, so openrsync it is.
enneff 8y ago
Stallman himself makes a big fuss about "Open Source" not being equivalent to "Free Software". See: https://www.gnu.org/philosophy/open-source-misses-the-point.en.html https://www.gnu.org/philosophy/open-source-misses-the-point....
anjbe 8y ago
Fun bit of history: the “Open” here comes from OpenBSD; but the “Open” in OpenBSD came from the development process, not the license. Before Git and SVN, we had CVS, and to check out code from a CVS repository you needed to have an account on the CVS server. If you wanted to contribute but didn’t already have a developer account, you were limited to writing patches against release tarballs or whatever alternative method upstream supplied. One of OpenBSD’s major projects in the mid 90s was creating anonymous CVS, where anyone could check out code without any account. This came from Theo’s experience after losing his NetBSD account, where he found himself unable to make meaningful contributions anymore without the ability to cvs checkout, cvs diff, etc. So when he started OpenBSD, he had in mind to open up the development process to everyone, account or not. This is described in the commentary for the OpenBSD 6.1 release song: https://www.openbsd.org/lyrics.html#61 https://www.openbsd.org/lyrics.html#61

AdmiralAsshat 8y ago

Interesting. This is the first project I can think of where a clean-room implementation was done so that a project could use a less free license ("free" as defined by the FSF). Does anyone else know of instances where a company did a clean-room implementation of a previously FOSS tool so that they could make a paid/proprietary version? Usually it goes the other way.

protomyth 8y ago
How is the ISC (version of BSD license used by OpenBSD) less free than the GPL3? This is very far from a "paid/proprietary" version.
- mouldysammich 8y ago
  Its much closer to a proprietary version than a GPL version would or could be however.
  - kevin_thibedeau 8y ago
    Proprietary-friendly is not less-free.
    - StavrosK 8y ago
      How not?
      - GalacticDomin8r 8y ago
        Because you are actually free to do what you want with it, instead of free to do what someone else wants you to do with it.
        StavrosK 8y ago
        Yes, you are free to close it up and sell it, but everyone else then isn't free to use your changes. "More friendly to proprietary purposes" is "less free". It's kind of like arguing that a country where anyone can steal from anyone else with impunity is more free. Not when you consider the rights of the person being stolen from.

meruru 8y ago

I hope this ends up being a lot simpler and easier to understand than the original rsync. The rsync manpage is way too long.

gmueckl 8y ago
Rsunc solved a complex problem that comes in many nuanced variants. It may seem trivial at the outset, but it is actually not. So I don't think that rsync has many features that are somehow unnecessary or bloat.
- meruru 8y ago
  Well, the manpage for this is looking really good and it already has almost everything that I care about. The -a option isn't in yet, but it's in the TODO. https://github.com/kristapsdz/openrsync/blob/master/openrsync.1 https://github.com/kristapsdz/openrsync/blob/master/openrsyn... https://github.com/kristapsdz/openrsync/blob/master/TODO.md https://github.com/kristapsdz/openrsync/blob/master/TODO.md I hope the -c and maybe -X option make it.
m0nty 8y ago
> The rsync manpage is way too long I see its thoroughness as a feature, not a bug. It's very well written and I can just ignore the bits I'm not interested in. I wish more man pages were "too long" like this one is.

accrual 8y ago

Very cool news. rsync(1) is one of the first things I install on a new OpenBSD instance. Tangentially related, I've been using Time Machine-like wrapper [0] around rsync(1) for a few years. It's very helpful for maintaining snapshots of my home directory. [0] https://blog.interlinked.org/tutorials/rsync_time_machine.html https://blog.interlinked.org/tutorials/rsync_time_machine.ht...

davewongillies 8y ago
I use rsnapshot [0] for the same thing. [0] https://rsnapshot.org/ https://rsnapshot.org/

gerdesj 8y ago

Is it any better than rsync?

yjftsjthsd-h 8y ago
It is better in some ways, and worse in some, both largely subjective. It has a different license, is smaller, less battle tested, from different developers, designed with different goals in mind. It Depends™ on how you judge.
rstuart4133 8y ago
All openrsync implements is the equivalent of a fast "cp -a" across the network, plus it can also remove files if they don't exist. rsync does much more and over the years I've used most of it, so there is no way I would use openrsync. The upside is the manpage of openrsync isn't that much more complex than cp, which is a definite bonus if that's all you are doing. The only thing I would change about rsync is it's default, which IMO should be to copy all meta data supported by both sides. Ie, the default should be to make the destination as similar to the source as possible. It's default is to only copy the data, and you must add options to say what else you want copied. To make matters worse you can just add every option because if you say you want to copy something not supported by one side of the other it errors out. I may have missed it as I am reading the man page source, but openrsync didn't seem to change that.
- kristapsdz 8y ago
  No. openrsync implements the rsync protocol. It doesn't have all of its options, but the protocol is what it is. Do you have any idea what you're talking about?

CaliforniaKarl 8y ago

I would not be surprised if, in a few years, this becomes one of the CLI tools installed on macOS, either as part of the default install or as part of the Xcode CLI tools.

riffraff 8y ago
Why? MacOS has a bunch of GPL stuff, such as bash, IIRC.
- __david__ 8y ago
  And it already has actual rsync.
  - avar 8y ago
    It has 12 year old rsync due to Apple not wanting to ship anything that's GPLv3: https://bayton.org/2018/07/how-to-update-rsync-on-mac-os-high-sierra/ https://bayton.org/2018/07/how-to-update-rsync-on-mac-os-hig...
- perbu 8y ago
  GPL2, I doubt you'll find any GPL3 code in there. Which is why bash on MacOS is from 2007.
gpvos 8y ago
Does macOS have any security features similar to pledge/unveil or any of the Linux hardening packages?
- Fnoord 8y ago
  It has a port of PF.
  - gpvos 8y ago
    I'm more interested to know about system call and filesystem access restrictors. I think pf is only a packet filter.

joppy 8y ago

What does a "clean-room implementation" mean?

anjbe 8y ago
Not derived from the existing code. The reason it’s mentioned is to assert that openrsync is not subject to the original rsync’s GPL.
Tor3 8y ago
The first (well-known) 'clean-room' implementation was when Phoenix implemented an IBM PC-compatible BIOS by having one team studying the IBM source (which was available), then writing up a specification for how it worked, handing that specification over to somebody else (they were Phoenix' legal team, IIRC), which then handed the specs over to another team that had never seen the IBM source. They sat down in their "clean room" (b/c it wasn't tainted by actual IBM source) and implemented a BIOS from specs only. In that way Phoenix was protected from any claims of copyright infringement: Nothing was copied, and the people writing the code had never seen the original source. In that particular case the specs were reverse-engineered from actual source, but that's not a necessary part of the process. It's more common to have one team study the protocol, data going over the wire, disassembling, etc, then use the knowledge gained to write specs, and then another team implements the equivalent functionality from specifications only.