6 ms·
Rsync.net Technical Notes – Q4 2021
- ohghiZai 5y agoNow that borg 1.2 is out, are there any plans on supporting that?
- rsync 5y agoHappy, as always, to answer any questions or discuss any comments here. Thanks again to Allan Jude of Klara Systems and Luke Young of bored.engineer for their very interesting comments and explanations. (these tech notes were published in March of 2022 - they are backward looking to the events of Q4 2021)
- tempnow987 5y agoSite loading a bit lumpily for me for a minute there. Otherwise interesting write-up and pruning unused features I do think helps with security generally.
- liberix 5y agoHi, the rsync.net website seems to be offline right now. Is there any other link where we can find the technical notes?
- rsync 5y agosite never went down but was perhaps slow for a brief moment ... try again, please ...
- anarcat 5y agoit's still sluggish at the time of writing, FWIW.
- EvanAnderson 5y agoI had trouble loading it, too, but managed to get an archive.is capture: https://archive.ph/YEzR5 https://archive.ph/YEzR5
- mattl 5y agoI can't get archive.ph to load but rsync.net loads fine...
- pokstad 5y agoAny news on adding the mkfifo command? I requested it a while back and it sounded like it was doable but I have not heard any updates.
- rsync 5y agoThanks! mkfifo was included in this new command addition/whitelisting and we're going to write up an entire section on it next quarter. Can you email ? I actually searched for our conversation for a while and could not find it ... bottom line is that 'mkfifo' is in place and I can't wait to craft up some fascinating remote ssh examples with it ...
- pokstad 5y agoBumped our email thread, thanks! For everyone else, the mkfifo command lets you transfer arbitrary streams of data between two systems while using rsync.net as an intermediary.
- dividuum 5y agoI’m slightly confused on why that would be useful within the rsync.net context. Is bouncing traffic a new intended use case?
- benlivengood 5y agoIs native encryption available on the send-capable accounts?
- rsync 5y agoYes. ZFS encryption and raw send.
- deleted 5y ago[deleted]
- cjbconnor 5y agoThe link isn't working for me.
- deleted 5y ago[deleted]
- ireflect 5y agoNot technical per se, but I'm interested to know how the "unlimited plan for life" experiment worked out. Did many folks take you up on the offer? I didn't, but I strongly considered it (if I'm being honest I just procrastinated the decision until I realized the deadline had passed).
- rsync 5y agoShhh ... :) It was, indeed, a very limited test promotion to some small subset of customers and I am still undecided on whether it makes sense to pursue or not. Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused - specifically by cloud providers. Was it Joyent that dishonored the strongspace lifetime offers ? I forget ...
- reaperducer 5y agoGenerally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused Back when Sirius Satellite Radio started, it offered "lifetime" subscriptions. People eventually learned that Sirius meant the lifetime of the radio as long as it tied to the purchaser's account, not the human being. Many people got Sirius with their new cars. When they started trading them in three or four years later, they learned that their "lifetime" subscription not only didn't go with them to their new car, but the buyer of their old car also didn't get the old subscription. Today, it wouldn't be as big a deal because you can stream Sirius ($10/month), or some other radio service over the internet. But this was back before even RealPlayer, and the mass adoption of cell phones.
- mattl 5y agoSirius was early 2000s. RealPlayer was 5+ years before that?
- reaperducer 5y agoSirius was 1990. RealPlayer was 1995.
- lechner 5y agoYour technical notes really illustrate that there is a science to running good backups. Thanks so much!
- anderspitman 5y ago> and there is your arbitrary python code execution - which is exactly what we don't want. The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process. Linux is better than most in terms of options, but you have to understand seccomp, namespaces, cgroups, containers, capabilities, etc etc. It's too confusing. Why can't I just say "spawn this process, limit it to reading files in this directory, writing files in this directory, this socket, max 10GB disk space, max 5% CPU, max 10% memory"? I'd even settle for just being able to limit filesystem and network access. Even if you do manage to make it work, since the APIs require SETUID executables you might actually be opening yourself up to a bigger risk. Someone needs to pull a WireGuard and make a small, simple API for userspace sandboxing. And please somehow miraculously make it so it is likely to be implemented by Windows.
- stephenr 5y agoI don’t know for sure about the memory and cpu rationing but doesn’t systemd expose the rest of those limits for service units?
- anderspitman 5y agoNot sure. Even if it does, it's going to be built using the APIs I already mentioned, and therefore not available in userspace. It doesn't make any sense for a process to need special privileges in order to limit a child process to strictly lesser privileges.
- dogleash 5y ago> Someone Why not you?
- anderspitman 5y agoBecause I'm not a security expert. Gaining the knowledge necessary to do it well would probably require a specialization change and years of prep, which would be an inefficient division of labor. Plus, it's an interesting problem, but not one I'm particularly drawn to. I'm currently focused on solving other problems[0] higher up the stack. [0]: https://boringproxy.io/ https://boringproxy.io/
- sporkle-feet 5y agoNot related to this technical note, but I was surprised to learn that the rsync service has scheduled downtime, like for hours at a time. Is it possible to make your service highly available? Seems that is table stakes for other cloud storage services today.
- rsync 5y agoNot sure what you mean… generally speaking we keep the storage servers up continuously - as you’d expect. I see several with 80-100 day uptimes right now …
- sporkle-feet 5y agoThen how do you explain this email sent by yourself? We are announcing an extended, off-peak maintenance window for your backup account. Beginning Saturday March 5th at 9:00a PT (UTC -8) the server will go offline for approx. 6 hours as we perform data integrity checks and rebalancing. ...
- rsync 5y agoThat's a single, one-off maintenance planned for Sat. March 5. That is not a regular, every Saturday maintenance ... perhaps the word "Beginning" made that confusing ? Let me check with engineering folks and see if they plan on repeating it - but even if they do, it is not a regular maintenance and you should expect your account to have long, uninterrupted uptimes ...
- helamonster 5y ago> However, if your workload consists of one or more zfs datasets that store, and subsequently access, many small files, you can actually store the files themselves on the device - not just the metadata. > For instance, if you have a pool named TANK and a filesystem named fs1 and you want to store all files smaller than, or equal to, 8K in the special vdev, you would run: ... This sounds like the files themselves must be of size 8K or less in order to be allocated on the metadata vdev. But doesn't that include all individual file blocks of size 8K or less, not just entire files?
- lygten 5y agoRsync.net is an awesome service. Ive been using their 100GB for 18 USD offer for the last few years to host my RESTIC backups. Have not had a single problem.