5 ms·

FreeBSD-rustdate, a reimplementation of FreeBSD-update

101 points by ggm 2y ago

toast0 2y ago
Thank you, thank you. I haven't reviewed this, and I may or may not use it, but wow was freebsd-update from 13.0 to 14.0 slow, especially since I run my freebsd systems off spinning drives, and most of them have src. I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler! But my systems running freebsd-update spend an awful lot of time on I/O, and not so much time on cpu, as I recall. I think the general flow for a replaced file is gunzip to the update dir, then install to the destination. This means all the files are written twice, and I thought there was a third write that I can't remember at the moment. Install doesn't have an option to decompress, but it'd be super handy if it did --- then you could write the file only once to the destination directory and move it into place. I'm never a fan of quite as many files in a directory as freebsd-update gets up to either. Given that everything is already named by hashes, it's not too hard to have 16 or 256 directories named 0-f or 00-ff that get a slice of the files. Large directories have gotten better over time, but they're still slow. There's some ability to exploit the embarassing parallelism available too, but that's hard in a complex shell script, so some other environment makes sense. I didn't look, but hopefully that's a knob in rustdate ... Sometimes you'd want it and other times one thing at a time is better, even when it's slow. One feature I know update can do that I want to look at more but haven't had time is the zfs boot environment stuff --- seems like with the right arguments you could do the update on an inactive environment and reboot to switch, instead of the default way which does the install on the live environment but you could reboot to switch back. I'd be much more patient if the live environment continued working and I could reboot at my convenience once the process finished. Especially if I might be able to run pkg upgrade in the boot environment before the reboot. If rustdate supports that too, that'd be neat.
- deaddodo 2y ago
  > I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler! It doesn't look too crazy: https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/freebsd-update/freebsd-update.sh https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/fr... It's fairly well broken down, and pretty well organized. I think the most difficult part is that so much of it relies on shell nice-isms that are a pain to re-implement in something like C (although, I guess you could just call fetch and the like directly via 'exec' or 'system'; rather than using their libraries to manage things directly).
  - throw0101d 2y ago
    > It's fairly well broken down, and pretty well organized. Not surprising given that it's written by Colin Percival, who was the Security Officer for FreeBSD for a while, and who runs/develops Tarsnap: * https://www.tarsnap.com https://www.tarsnap.com * https://en.wikipedia.org/wiki/Colin_Percival https://en.wikipedia.org/wiki/Colin_Percival * https://news.ycombinator.com/user?id=cperciva https://news.ycombinator.com/user?id=cperciva
  - kstrauser 2y ago
    I do have to say that's about the cleanest shell script of that size I've ever seen. cperciva, if you're reading this, that's pretty.
  - cryptonector 2y ago
    Hmmm, lots of `for C in $@;` that should be `for C in "$@";` instead.
    - ptx 2y ago
      Hmm. Yup. For example, this is broken: # freebsd-update -f "/tmp/path with spaces/freebsd-update.conf" fetch usage: freebsd-update [options] command ... ...
      - cryptonector 2y ago
        And it's not even setting IFS to the empty string.
- tux2bsd 2y ago
  [dead]
mustache_kimono 2y ago
Would you mind mirroring to a repo, with a license, so others might view?
- stackghost 2y ago
  The source tarball, on the downloads page, contains a license.
  - ThePowerOfFuet 2y ago
    Shouldn't have to download and unpack a tarball to see how it's licensed.
    - stackghost 2y ago
      This seems like a real "beggars being choosers" situation.
      - Pet_Ant 2y ago
        I think this may be a person who is concerned a out being tainted by a license. Life if the code was source-available only and they saw it and then were accused to copying some functionality into another open source project. A bit paranoid, sure, but not impossible.
    - quectophoton 2y ago
      It's only a .xz one though, why the worry? /s Now seriously, I'm not particularly interested on this, but if it helps here are additional points of reference to at least be able to verify some integrity in case you want to try it. SHA256: 90d7b2d632a3887bf26c10c42584ba899a3071725dae7dcddddcbab54768b54b freebsd-rustdate-0.6.1.tar.xz Archive link: https://web.archive.org/web/20240827090825/https://rustdate.over-yonder.net/dl/freebsd-rustdate-0.6.1.tar.xz https://web.archive.org/web/20240827090825/https://rustdate.... Doesn't prove the absence of any malicious intent, but at least it should help prove that nothing is changing between requests (i.e. that the file I got is the same as the file you got).
ode 2y ago
See https://rustdate.over-yonder.net/speed.html https://rustdate.over-yonder.net/speed.html for the numbers.
- frankjr 2y ago
  Impressive. The current state doesn't really bother me given the frequency of system updates but I do sometimes wonder what on earth it's even doing that's taking so long.
alberth 2y ago
> This is freebsd-rustdate, a reimplementation of freebsd-update. > In usage, it’s expected to be similar, but not identical to freebsd-update. Doesn't reimplementing it to not be identical, defeat the point of reimplementing it? (since it can't be used as a drop-in replacement)
- youwot0 2y ago
  The ability to rewrite incompatible subsets of basic Unix utilities is where Rust truly shines.
  - jchw 2y ago
    At the very least, the uutils project seems to be making good progress on becoming a "true" drop-in replacement for the GNU coreutils that will pass the GNU test suites and treat compatibility issues as bugs. Honestly though, as time goes on, I'm not sure if rewriting UNIX utilities to be memory-safe actually gets you that much. I mean, having multiple implementations of something as important as the coreutils is good: nobody complains about the co-existence of e.g. BSD and GNU utilities, and IMO there's really no reason to feel any different about RiiR rewrites. That said, I can't really figure out what attack model would really make memory safety a particular priority. In most cases you would be isolating things that require these sorts of utilities using namespaces/jails/etc, and I wonder if memory bugs in the coreutils is actually a serious attack vector. Of course in this case (FreeBSD-rustdate) it's literally for performance reasons rather than safety, so everything sort of makes sense (not necessarily using Rust for safety per-se, but for robust threading. That does make sense.)
    - iforgotpassword 2y ago
      I think libmagic (the file util) might be a good target for this. I think it had security issues in the past and considering how it contains a gazillion random parser for weird file formats, surely there is a good chance there are some more. But nobody would ever run file on a random file you just downloaded, would you? ;)
nmz 2y ago
Given that sh can print everything as it runs, it would be nice to know what is it that takes so long, instead of doing a full rewrite. I remember once replacing a filter written in pure bash by an awk one, the slowdown was around 2 minutes v awk which was a second.
- kelsey98765431 2y ago
  edit the .sh, add `set +x` then pipe it to ts ``` $ cat > example.sh ; bash example.sh |ts set -x echo a sleep 3 echo b sleep 1 echo c #end + echo a + sleep 3 Aug 27 09:48:02 a + echo b + sleep 1 Aug 27 09:48:05 b + echo c Aug 27 09:48:06 c ```
  - kstrauser 2y ago
    You can do all that without modifying the script or using a separate tool: PS4='+ $(date --iso-8601)\011 ' bash -x example.sh
alberth 2y ago
Suggestion: call this something else - because 'rustdate' makes me think 'rust date', which makes me believe this is date/time related.
gonzo 2y ago
Given the official stance of moving to pkg-base, updating freebsd-update with an incompatible rewrite in Rust seems like yak shaving.

101 points by ggm 2y ago

Thank you, thank you. I haven't reviewed this, and I may or may not use it, but wow was freebsd-update from 13.0 to 14.0 slow, especially since I run my freebsd systems off spinning drives, and most of them have src. I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler! But my systems running freebsd-update spend an awful lot of time on I/O, and not so much time on cpu, as I recall. I think the general flow for a replaced file is gunzip to the update dir, then install to the destination. This means all the files are written twice, and I thought there was a third write that I can't remember at the moment. Install doesn't have an option to decompress, but it'd be super handy if it did --- then you could write the file only once to the destination directory and move it into place. I'm never a fan of quite as many files in a directory as freebsd-update gets up to either. Given that everything is already named by hashes, it's not too hard to have 16 or 256 directories named 0-f or 00-ff that get a slice of the files. Large directories have gotten better over time, but they're still slow. There's some ability to exploit the embarassing parallelism available too, but that's hard in a complex shell script, so some other environment makes sense. I didn't look, but hopefully that's a knob in rustdate ... Sometimes you'd want it and other times one thing at a time is better, even when it's slow. One feature I know update can do that I want to look at more but haven't had time is the zfs boot environment stuff --- seems like with the right arguments you could do the update on an inactive environment and reboot to switch, instead of the default way which does the install on the live environment but you could reboot to switch back. I'd be much more patient if the live environment continued working and I could reboot at my convenience once the process finished. Especially if I might be able to run pkg upgrade in the boot environment before the reboot. If rustdate supports that too, that'd be neat.

deaddodo 2y ago
> I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler! It doesn't look too crazy: https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/freebsd-update/freebsd-update.sh https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/fr... It's fairly well broken down, and pretty well organized. I think the most difficult part is that so much of it relies on shell nice-isms that are a pain to re-implement in something like C (although, I guess you could just call fetch and the like directly via 'exec' or 'system'; rather than using their libraries to manage things directly).
- throw0101d 2y ago
  > It's fairly well broken down, and pretty well organized. Not surprising given that it's written by Colin Percival, who was the Security Officer for FreeBSD for a while, and who runs/develops Tarsnap: * https://www.tarsnap.com https://www.tarsnap.com * https://en.wikipedia.org/wiki/Colin_Percival https://en.wikipedia.org/wiki/Colin_Percival * https://news.ycombinator.com/user?id=cperciva https://news.ycombinator.com/user?id=cperciva
- kstrauser 2y ago
  I do have to say that's about the cleanest shell script of that size I've ever seen. cperciva, if you're reading this, that's pretty.
- cryptonector 2y ago
  Hmmm, lots of `for C in $@;` that should be `for C in "$@";` instead.
  - ptx 2y ago
    Hmm. Yup. For example, this is broken: # freebsd-update -f "/tmp/path with spaces/freebsd-update.conf" fetch usage: freebsd-update [options] command ... ...
    - cryptonector 2y ago
      And it's not even setting IFS to the empty string.
tux2bsd 2y ago
[dead]

mustache_kimono 2y ago

Would you mind mirroring to a repo, with a license, so others might view?

stackghost 2y ago
The source tarball, on the downloads page, contains a license.
- ThePowerOfFuet 2y ago
  Shouldn't have to download and unpack a tarball to see how it's licensed.
  - stackghost 2y ago
    This seems like a real "beggars being choosers" situation.
    - Pet_Ant 2y ago
      I think this may be a person who is concerned a out being tainted by a license. Life if the code was source-available only and they saw it and then were accused to copying some functionality into another open source project. A bit paranoid, sure, but not impossible.
  - quectophoton 2y ago
    It's only a .xz one though, why the worry? /s Now seriously, I'm not particularly interested on this, but if it helps here are additional points of reference to at least be able to verify some integrity in case you want to try it. SHA256: 90d7b2d632a3887bf26c10c42584ba899a3071725dae7dcddddcbab54768b54b freebsd-rustdate-0.6.1.tar.xz Archive link: https://web.archive.org/web/20240827090825/https://rustdate.over-yonder.net/dl/freebsd-rustdate-0.6.1.tar.xz https://web.archive.org/web/20240827090825/https://rustdate.... Doesn't prove the absence of any malicious intent, but at least it should help prove that nothing is changing between requests (i.e. that the file I got is the same as the file you got).

ode 2y ago

See https://rustdate.over-yonder.net/speed.html https://rustdate.over-yonder.net/speed.html for the numbers.

frankjr 2y ago
Impressive. The current state doesn't really bother me given the frequency of system updates but I do sometimes wonder what on earth it's even doing that's taking so long.

alberth 2y ago

> This is freebsd-rustdate, a reimplementation of freebsd-update. > In usage, it’s expected to be similar, but not identical to freebsd-update. Doesn't reimplementing it to not be identical, defeat the point of reimplementing it? (since it can't be used as a drop-in replacement)

youwot0 2y ago
The ability to rewrite incompatible subsets of basic Unix utilities is where Rust truly shines.
- jchw 2y ago
  At the very least, the uutils project seems to be making good progress on becoming a "true" drop-in replacement for the GNU coreutils that will pass the GNU test suites and treat compatibility issues as bugs. Honestly though, as time goes on, I'm not sure if rewriting UNIX utilities to be memory-safe actually gets you that much. I mean, having multiple implementations of something as important as the coreutils is good: nobody complains about the co-existence of e.g. BSD and GNU utilities, and IMO there's really no reason to feel any different about RiiR rewrites. That said, I can't really figure out what attack model would really make memory safety a particular priority. In most cases you would be isolating things that require these sorts of utilities using namespaces/jails/etc, and I wonder if memory bugs in the coreutils is actually a serious attack vector. Of course in this case (FreeBSD-rustdate) it's literally for performance reasons rather than safety, so everything sort of makes sense (not necessarily using Rust for safety per-se, but for robust threading. That does make sense.)
  - iforgotpassword 2y ago
    I think libmagic (the file util) might be a good target for this. I think it had security issues in the past and considering how it contains a gazillion random parser for weird file formats, surely there is a good chance there are some more. But nobody would ever run file on a random file you just downloaded, would you? ;)

nmz 2y ago

Given that sh can print everything as it runs, it would be nice to know what is it that takes so long, instead of doing a full rewrite. I remember once replacing a filter written in pure bash by an awk one, the slowdown was around 2 minutes v awk which was a second.

kelsey98765431 2y ago
edit the .sh, add `set +x` then pipe it to ts ``` $ cat > example.sh ; bash example.sh |ts set -x echo a sleep 3 echo b sleep 1 echo c #end + echo a + sleep 3 Aug 27 09:48:02 a + echo b + sleep 1 Aug 27 09:48:05 b + echo c Aug 27 09:48:06 c ```
- kstrauser 2y ago
  You can do all that without modifying the script or using a separate tool: PS4='+ $(date --iso-8601)\011 ' bash -x example.sh

alberth 2y ago

Suggestion: call this something else - because 'rustdate' makes me think 'rust date', which makes me believe this is date/time related.

gonzo 2y ago

Given the official stance of moving to pkg-base, updating freebsd-update with an incompatible rewrite in Rust seems like yak shaving.