68 ms·

Interview with CEO of rsync.net: “no firewalls and no routers”

654 points by dmytton 6y ago

sideshowmel 6y ago
Don't know if running a dumb switch connected to your ISP is the best infosec policy: https://blogs.cisco.com/manufacturing/the-top-5-reasons-to-avoid-unmanaged-switches-on-your-machines https://blogs.cisco.com/manufacturing/the-top-5-reasons-to-a...
- Jonnax 6y ago
  I'm not sure those reasons really apply to their case. Especially since they're running the boxes that it's connected to. They can do resiliency, network segmentation, and monitoring on their platform. What's a Cisco box going to do for them?
  - sideshowmel 6y ago
    Dumb switches will blast packets to all interfaces that are connected. If there's a machine on the switch that's in promiscuous mode, it can see all the packets on the local network (including the backups coming in from customers). Managed switches typically have ACL support. I get the KISS principle, but this setup seems to be trading security for simplicity.
    - noir_lord 6y ago
      > including the backups coming in from customers. Which are encrypted in flight...if they aren't then anyone on the 30 machines between customer and final destination can also see the backups coming in from customers.
      - sideshowmel 6y ago
        True, but the packets in-flight can take different routes. If you have a machine on the switch, you know you've captured all the packets that were in-flight. This make it easier to break the encrypted packets. It's a choice--everything in security is a risk-management assessment, but I'm surprised rsync.net was able to get so many security certifications with this setup.
        noir_lord 6y ago
        > If you have a machine on the switch, you know you've captured all the packets that were in-flight. Same applies if someone takes over the firewall, machine on the last hop before they hit port 22. In a world where stuff like this https://www.helpnetsecurity.com/2020/09/01/zero-day-cisco-enterprise-routers/ https://www.helpnetsecurity.com/2020/09/01/zero-day-cisco-en... routinely happens there is a benefit to forgoing all of that when it makes sense.
        mcosta 6y ago
        # tcpdump -i eth0 tcpdump: eth0: You don't have permission to capture on that device (socket: Operation not permitted)
    - ptomato 6y ago
      > it can see all the packets on the local network I'm sure those packets (consisting entirely of OpenSSH) will be very useful to them
      - sideshowmel 6y ago
        Don't be so sure :) Quantum computing is improving everyday, and new methods of defeating RSA are being researched: https://eprint.iacr.org/2021/232 https://eprint.iacr.org/2021/232
        anthk 6y ago
        OpenSSH now uses eliptic curves, not RSA.
    - dividuum 6y ago
      They only support SSH (legacy FTP was sunset a year ago), so there's nothing to gain (except for maybe the volume and IP of the customer) by observing other traffic. Which happens to be the same information you can observe anywhere in the path from a customer to their machines.
    - iso1631 6y ago
      > Dumb switches will blast packets to all interfaces that are connected Multicast and broadcast sure, but dumb switches will still keep mac-address>port mapping. If the router sends to 52:54:00:ad:ra:a7, the dumb switch will remember that's on port 7 (having seen traffic from it recently - if only an arp reply) and only send the packet to port 7. Hubs (remember them!) will blast every packet to every port.
    - EvanAnderson 6y ago
      Thie first paragraph is incorrect. A hub will "blast packets to all interfaces that are connected". A switch, even a dumb one, still switches packets. Broadcasts and frames addressed to unknown destinations will flood out all ports, but not unicast frames with destinations currently in the MAC table. It is true that an attacker could flood the MAC table, spoof their MAC, etc, after compromising a layer-2 adjacent host and use that to manipulate traffic flows. That's somewhat disturbing, but no Customer backup data should be hitting their network outside of SSH anyway. I think the potential is more for DoS than compromise of confidentiality or integrity. I really admire rsync.net's simplicity, but dumb switches give me the willies. I feel blind not having per-interface counters, at the very least. If nothing else, I'd like to be able to reconcile the counters coming from my OS interface with the switch in troubleshooting scenarios.
    - deleted 6y ago
      [deleted]
    - fomine3 6y ago
      You may mean repeater hub.
    - deleted 6y ago
      [deleted]
- iso1631 6y ago
  The only "security risk" i see there is number 1, and that is all to do with physical security. > Disadvantage #1 – Open ports on unmanaged switches are a security risk Why? Is there something that would prevent an attacker with physical access from unplugging an existing cable? Does the average managed switch config have mac limits and auto shutdown if a link is lost for just a few seconds? Mac limits are easilly bypassed, even without (permanently) disconnecting the legimate device by inlining an active device, maybe some mac spoofing. I don't include 802.1x or automatically shutting down a port that loses an uplink as a "simple and effective security precaution", it would be a right pain for many situations. Is the latter even a feature? I certainly haven't come across it (unlike normal portsecurity like limiting number of mac addresses, which just adds to overhead with limited effective security). > Disadvantage #2 – No resiliency = higher downtime If my device has one ethernet cable into one switch, how does that help? If my unmanaged switch goes pop, I have a spare that I can put in and be back running in a minute. My managed cisco edge switches take 10+ minutes just to reboot. If my device has two ethernet cables, one into one unmanaged switch, one into another, losing that switch isn't a problem. > Disadvantage #3 – Unmanaged switches cannot prioritize traffic Correct they can't. Managed switches without qos set up can't prioritise traffic either. If your switch is dropping packets, you don't have enough bandwidth. I've seen packet loss when sending 500mbit down a 1G uplink on managed switches, even on QOSed traffic. Indeed I've seen higher priority traffic drop and lower priority not drop. QOS isn't trivial. Ultimately it comes down to how big your buffers are whether your packet gets through or not, so your application should cope with some loss, and if you get too much loss you need more bandwidth. If you have 48 devices connected at 1Gbit each, each firing 100mbit of traffic every second, all bang on the second, with a 10gbit uplink, on paper you only need 4.8gbit of uplink. You'll also need a 600MB packet buffer and expect a lot of delay on your packets, whether you have managed or unmanaged, QOS or no QOS. > Disadvantage #4 – Unmanaged switches cannot segment network traffic Correct, but then if I have 8 desktops in a cluster why wouldn't I pop in a desktop switch with 8 1G ports? I want them all on the same vlan anyway. > Disadvantage #5 – Unmanaged switches have limited or no tools for monitoring network activity or performance They don't, but again do I want that for a specific use case? If I want a managed switch (which I usually do), then I'll spec a managed switch. It's unlikely it will be cisco. If my requirements don't need features of a managed switch then I won't bother. I find it interesting that there's no mention of preventing broadcast storms, or IGMP snooping - both of which are far more useful for a typical edge switch than qos. Personally, I tend to use managed switches - indeed I just bought a couple of 24 port TP Link POE switches for an event I'm planning. I'm not 100% sure I'd go for an unmanaged switch in rsync's case, but from your list 1) Doesn't apply -- servers are in a secure location 2) Doesn't apply -- servers are either single connected (so need a physical visit, and replacing an unmanaged switch is far quicker and easier than a managed switch), or they're dual connected to two different switches 3) If they're doing inline management then you might want to carve out a small part of your uplink to prevent yourself from being dossed by a dodgy server (if your server is saturating your uplink bandwidth and you ssh session can't establish that could be an issue. If you've got OOB access on a separate link though, not a problem, and clearly they don't have that problem) 4) Doesn't matter -- they don't want different vlans 5) They presumably measure the bandwidth use of each of their servers. The question thus is "does the ISP give me logs I can rely on for the wan". Personally I wouldn't, but I can see the idea Spanning tree: Secure network, they aren't going to connect one port to another to cause a storm IGMP: They presumably aren't using multicast for anything major so bitrates would be very low even if they were there Reasons to use a firewall or a switch with an ACL in this specific case that I can think of: 1) 2 points of control -- a zero-day on freebsd's firewall could open a port to an unintended source which was listening but blocked by iptables (or bsd's version). If you had a non-bsd firewall it's unlikely the same zero-day would work 2) Port 22 is only open to a specific IP range, again there's a zero-day, and TTL of outbound packets is high enough to establish a session Reasons to use a managed switch even ignoring firewalling: 1) Reliable traffic stats -- you could guess at these by summing the uplinks of all the connected devices although some packets will be dropped and some may be going to other devices on the network Reasons to use QOS on a managed switch: To allow inband managment if something goes wrong. A separate ilo/ipmi/kvm connection would be better for that though. I don't think they'd need features like span ports (I personally use them all the time, and fibre taps, but I have a different use case which is UDP heavy and loss-intollerent)
rsync 6y ago
Unrelated, as an aside ... I really am enjoying the developer Q&A interviews that console.dev is putting out. They're very much like the "usesthis"[1] profiles but more in-depth and with more interesting details ... [1] https://usesthis.com/ https://usesthis.com/
- mattl 6y ago
  It was an interesting read! I did a usesthis a little while ago. https://usesthis.com/interviews/matt.lee/ https://usesthis.com/interviews/matt.lee/
aDfbrtVt 6y ago
Thanks for the interview, I was pleasantly surprised to see how simple the network architecture is at rsync.
ttsiodras 6y ago
Interesting interview - thanks John! Didn't know there was a UFS2 "phase" before ZFS... I wonder how much time those fscks took! :-)
- rsync 6y ago
  They took forever ... and then they bombed out due to lack of memory. Not lack of physical memory, but lack of ability to address it as the UFS2 tools, like fsck, were not written to handle billions of inodes ... We really can't thank Kirk M.[1] enough - he wrote custom patches to ufs and fsck just for our (dirty) filesystems and, as I mention in the article, eventually gave us the push to migrate to ZFS. [1] https://en.wikipedia.org/wiki/Marshall_Kirk_McKusick https://en.wikipedia.org/wiki/Marshall_Kirk_McKusick
  - cvwright 6y ago
    Wow, what a cool story. Having Kirk McKusick write patches for you is almost like that old Weird Al song: > I’m down with Bill Gates > I call him “money” for short > I phone him up at home > And I make him do my tech support
Aeolos 6y ago
> "I have a early-2009 “octo” Mac Pro [...]" > > OS: macOS Does this make anyone else a bit uncomfortable? I don't think MacOS is still receiving security updates on that hardware. I'm all for using old hardware for as long as it keeps working, but I would never browse the internet with a vulnerable OS on a vulnerable processor (spectre etc...) Or am I missing something?
- lunixbochs 6y ago
  They're possibly something like a dosdude patcher or modified bootloader to run an OS like Catalina on it.
  - Alupis 6y ago
    I have trouble understanding why people go through these hoops. Yeah, I get it, people love their Mac's... but the company that produces them actively undermines your ability to continue using perfectly good hardware past what they feel is "profitable". This leads to huge efforts to hack/reverse the updaters, or alter newer OS versions to trick them into installing, etc. I'd personally jump over to some system that doesn't hate it's users nearly as much. But, that's just me.
    - boardwaalk 6y ago
      Do you really need to pivot into Apple bashing on this thread? It’s not really on topic or needed.
    - gambiting 6y ago
      It's not out of some love for Macs. I have a 2008 MacBook running Catalina and it's simply because the cost of replacing it is >0. If this works and works well(and it does) then why would I get rid of it? Just to spite apple, which doesn't care either way? I also have a 2005 car that still runs - should I get rid of it because the company that made it stopped providing any kind of support for it long time ago? Or you know....keep using it because it works?
ciil 6y ago
Jealous of how well you seem to be able to keep to KISS as a principle.
- rhizome 6y ago
  The number of "simplicity? what's that?" brain-implosions in this thread is kind of hilarious, though at the same time a little concerning.
booi 6y ago
A simple layer 2 network topology only works in very narrow use cases (like this one). But a "dumb switch" means you also lose a lot of observability and it's very difficult to apply consistent network acls.
- rsync 6y ago
  Agreed - we are, in a sense, "cheating" because our product is so simple that we do have one of these "very narrow use cases". The benefits are tremendous, however, and go beyond day to day operations. A dumb switch has no credentials to protect and there is almost zero attack surface. Further, if our switch dies we can immediately replace it with any other dumb switch that just happens to be lying around. If you read failure studies - like those in the excellent Charles Perrow book _Normal Accidents_[1] - you see that in many cases there is a very special component that fails and everything goes to hell when they can't find a replacement for it. So, while I can't encourage everyone to use dumb, unmanaged switches (because not everyone can) I can encourage everyone to remove as many very special components as they can. [1] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
  - _trampeltier 6y ago
    I work in industrie automation and I can't agree more to dumb devices. There are a lot of nice special products, but if something goes broken, you have first tousend of pages manual, just maybe an identical part. The guy on the night shift has also to know this special part well .. and so on. The dumb devices, you can replace easy without any problems tomorrow or also in 10 .. 20 years.
  - chris_wot 6y ago
    Charles Perrow only died recently, very sad.
  - Bluecobra 6y ago
    How are you providing network level redundancy with dumb switches? My only guess is that the ISP is already doing HSRP/VRRP on the gateway and you can setup multiple NICs/switches with something like CARP and being careful not to make L2 loops.
robotmay 6y ago
Nice article. rsync.net is one part of my personal computing setup that I never even think twice about. It's simple and it works, and that clearly applies to the infrastructure too. I use ZFS locally and it has made managing my own data strangely pleasing, and it's nice to have the same system on my off-site storage too. On the laptop-front, I find myself drifting towards a similar setup to John. I have a hefty workstation laptop but the battery life is dire and it weighs a ton, so I pretty much just run it as a headless machine next to my server now. I'm planning on picking up a Pinebook Pro as an "outdoors" machine to just remote in. I also find myself extremely unwilling to arse about swapping multiple machines on my monitors so being able to keep my work machine separate and secure but operate it from my desktop is a nice compromise.
- nicolaslem 6y ago
  I would love to use this simple setup as well. It's too bad ZFS snapshots cannot be sent and stored encrypted. I would love to use rsync.net but the idea to have my data sitting in someone else's computer in plain text feels wrong. So instead I have to use restic, which re-implements many features of ZFS and this also feels wrong.
  - rsync 6y ago
    You can 'zfs send' to a (special kind of) rsync.net account. We support encrypted zfs[1][2][3] and raw-send, etc. The pricing is the same but there is a 1TB minimum because we need to give you your own VM (bhyve) and we have to burn an ipv4 address for you, etc. [1] https://www.rsync.net/products/zfs.html https://www.rsync.net/products/zfs.html [2] https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync... [3] https://www.servethehome.com/automating-proxmox-ve-zfs-offsite-backup-rsync-net/ https://www.servethehome.com/automating-proxmox-ve-zfs-offsi...
    - blibble 6y ago
      could you allocate the VM on demand? xinetd style (you could route the ssh traffic similarly based on login)
trollski 6y ago
i could get cloud storage from Microsft at ~1/20 of the cost. why would i use rsync.net?
frammie 6y ago
Really well done interview, some real interesting bits in there. One part concerned me though, in the interview, it mentions "we own (and have built) all of our own platform." and it fails to mention a few critically important key parts of a storage platform, first being encryption. How are personal files being handled? Is encryption being used? Are you able to access this data using a shared key? As well as contingency, what happens if critically important data is stored on your platform. On your website you mention: "We have a world class, IPV6-capable network with locations in three US cities as well as Zurich and Hong Kong" however fails to mention if replication is done across these locations. If technology (drives) is stolen from your datacenter, or mechanical failures beyond your control happen, how will you be able to recover from physical failure if you only appear to be serving from a single location? Excuse me if I'm wrong but I couldn't find anything concrete in either the interview or your website. The premise of the platform seems quite well aligned with keeping alive the the UNIX philosophy, and reminds me of Tarsnap. Either way, well made interview and interesting approach to a storage platform. As a sidenote, what keyboard are you using? It seems really interesting and you failed to mention it in the interview :) EDIT: It appears that you offer Geo-Redundant Filesystem as as separate product, maybe you would want to make this a bit more visible on your website except for only the FAQ and order pages. Either way, it seems like a sufficient move, that does still leave the topic of encryption though. As mentioned traffic is encrypted using SSH ofcourse, but is the data itself encrypted on your platform?
- dharmab 6y ago
  I've used rsync.net in the past- it's essentially "filesystem as a service." You, the customer, use it to back your own software that handles the encryption and replication. Their website has some how-to guides for some common software, or you can roll your own with the rsync protocol. Notably, their website only claims transfer encryption, not encryption at rest. You can of course encrypt your files yourself with your own keys.
  - frammie 6y ago
    Not having data encrypted by default is concerning, however I do admire the simplistic approach of handling your own dataflow and tools for sure.
antongribok 6y ago
Reading this takes me back to when I started playing with storage professionally. For me it was in 2004, also using 3Ware controllers. I was running on RedHat (before RHEL) and XFS before it was common on Linux, and similarly had memory issues when trying to repair filesystems.
sparkling 6y ago
Hetzner has a similar product at better pricing that i have been using a minimalist dropbox alternative https://www.hetzner.com/en/storage/storage-box https://www.hetzner.com/en/storage/storage-box Access via rsync/sftp/scp
- fuzzy2 6y ago
  Also, Borg backup.
  - formerly_proven 6y ago
    rsync.net was the first storage provider to support Borg out of the box and also has a special tier for Borg users (which was later expanded for restic and some others iirc). I also like that their Europe location is in Switzerland. I think it's useful for a number of reasons to store critical data in more than one jurisdiction.
- foepys 6y ago
  Your link is dead for me. Maybe you wanted to link to this? https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box Hetzner is throttling bandwidth after traffic exceeds ~5x the storage capacity while rsync.net doesn't seem to. Hetzner also only supports a very small number of snapshots in total while rsync.net supports more per day. I don't think Hetzner and rsync.net are really competing with each other. rsync.net's focus is more on business customers, while Hetzner targets private customers.
- CameronNemo 6y ago
  I tried to view the link. Got a site not found error.
- jsmith99 6y ago
  It seems a very similar product, also offering zfs snapshots, but I like the fact rsync.net snapshots are immutable: you can browse them but there is no way to delete them without contacting support (and the CEO once posted he would review every such request). It makes me feel more confident about my backups if someone got hold of the cached credentials from my backup software.
kplex 6y ago
Is rsync.net related to rsync the project?
- rsync 6y ago
  No, there is no relationship. However, in 2005 or 2006 when we spun out of JohnCompanies[1] and incorporated under the name "rsync.net" I requested, and was given, explicit permission to use the name and domain by the maintainers of rsync.
tyingq 6y ago
I do get the "no separate firewall" reasoning, but I'm paranoid enough that I'd at least want some PF rules just in case some daemon gets started by accident.
- formerly_proven 6y ago
  Oh I'm pretty sure there is a firewall configured on the nodes themselves (customers get shell access) and he just meant that there isn't a separate firewall box in front of the servers.
  - rsync 6y ago
    Correct. The storage arrays themselves have a (modest) ruleset which, among other things, locks them to TCP22 only and disallows broken/impossible things like xmas-tree packets. Simple stuff.
korethr 6y ago
I wonder if they have any sales to large enterprises or similar institutions. In my experience, the larger organizations will have a "security" questionnaire required of their vendors, and the person administering it is a droid, incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today, or to the particular product/service/vendor in question. And no firewalls or routers would be massive, disqualifying red flags on such a questionnaire. Never mind that a KISS setup tends to bring security because of its minimized attack surface. In the minds that write and administer those questionnaires, security only comes from sufficient amounts of the right kinds of complexity. I'm sure it can be done. IIRC, Cloudflare doesn't use any firewalls, and they do some big business. It just isn't easy to get past the droids programmed to ensure that all pegs shall be properly square, IME.
- rsync 6y ago
  "I wonder if they have any sales to large enterprises or similar institutions." Yes, certainly. We frequently fill out very detailed checklists and questionnaires related to our quality policy, standards, internal policies, etc. We're also very honest about how we approach these issues: https://www.rsync.net/resources/regulatory/pci.html https://www.rsync.net/resources/regulatory/pci.html ... and they generally appreciate the honesty.
  - chris_wot 6y ago
    Man, everything about your service is simple and direct! Amazing.
  - learn_more 6y ago
    FYI, your "pricing" link at the top of that pci.html page 404's. The pricing link works from other pages however.
    - rsync 6y ago
      I see that that has now been fixed - thanks for pointing it out.
canoebuilder 6y ago
With regard to the iOS import/export mentioned, does anyone have any more recommendations? (I'm not familiar with the mentioned option, nothing against it, just seeking out all options) Simple file system interface to all devices first, then any further software interfaces on top only if desired. Thanks for making the option available for remote storage John!
bflesch 6y ago
Big fan of rsync.net but the firewall comment caught me a bit off-guard. The benefit of a firewall is that it's an isolated system which - apart from port blocking - guarantees a certain level of traffic logging and known-good state. If you have everything on one host I'd say your overall setup on that host becomes much more complex because you only need to get hit by one successful exploit chain and all logs on that host cannot be trusted any more.
- klodolph 6y ago
  On a reasonable-size setup, I would expect that the logs are exported to dedicated log storage (log-only machines) as part of an effort to preserve accurate log files even in the case of a successful attack on one of the hosts. It is not especially hard to ensure that, for example, a record of an SSH login attempt gets recorded to an external server before the request is authenticated. So if you have (for example) an SSH account and a local privilege escalation exploit, there is still some evidence in the logs. In the past, the benefits of a firewall were more clear-cut, but these days I think that it’s reasonable to have “defense in depth” without using a firewall as part of your solution.
  - hertzrat 6y ago
    The firewall is still helpful in case they hire a new person who opens a port and forgets to close it one day
    - dsr_ 6y ago
      "Steve, did you open a port? We only use SSH. What's going on?"
      - hertzrat 6y ago
        Oversights happen https://thenewstack.io/junior-dev-deleted-production-database/ https://thenewstack.io/junior-dev-deleted-production-databas...
api 6y ago
I go by the rule that if something is not secure enough to plug directly into the Internet, it is not secure. That doesn't mean I'll necessarily do that, but that should be the bar. The only exception is special purpose backplane networks that are designed explicitly to be isolated. These are basically data busses for clusters, not user-facing networks.
efxhoy 6y ago
That was a nice read! Good to read about something simple after a day working with AWS and their managed magic. Scrolling through the cert pages 2015 seems to be in the future though? > We personally toured every single major datacenter in Hong Kong and Zurich to choose the facilities that best met our old-fashioned standards for datacenter and telco infrastructure. The same will be true of our upcoming Montreal location in Q4, 2015. https://www.rsync.net/resources/regulatory/sas70.html https://www.rsync.net/resources/regulatory/sas70.html
anderiv 6y ago
This was a pleasure to read. I've been an rsync.net customer for ~6 months now, and am using Borg to send de-duped, encrypted backups to rsync.net from a few on-premise linux systems. As compared to other similar backup systems I've used, it's been a pure pleasure to implement and maintain. Thank you for your great product and support, John!
jeffbee 6y ago
I always liked this set of marketing materials. But I also see where they conflict with my experience. "You may visit our datacenters any time you like for a personal tour and inspection to satis[f]y whatever due diligence requirements you may have" probably appeals to many customers, but for my dollar I would prefer a datacenter that nobody may enter.
- mcosta 6y ago
  > I would prefer a datacenter that nobody may enter. If a disk break, who changes it?
  - jeffbee 6y ago
    Not tourists, clearly.
- mypalmike 6y ago
  Presumably the racks are in cages and visitors aren't able to actually touch them.
ChrisArchitect 6y ago
I don't care for newsletters on tooling, but these Q&A interview posts are good -- immediately went in search of a twitter, couldn't find due to difficult naming, but want to follow to keep up from time to time https://twitter.com/consoledotdev https://twitter.com/consoledotdev
hertzrat 6y ago
I used to run Linux for everything but I’m having to use Windows these days. What would it take to get rsync.net playing nicely with windows? I’m imagining Windows subsystem for Linux (ubuntu) with duplicity installed to it? Are there any major hiccups to that sort of setup?
- pfortuny 6y ago
  rclone should work, afaik.
- rsync 6y ago
  From the standpoint of random access "browsing" over SSH/SFTP, you could just use filezilla or WinSCP or ... psftp.exe. However, if you want a backup process then you will, indeed, need to find some way to run 'borg' or 'restic' or 'rclone' on Windows. I've never used WSL so I can't comment, unfortunately ...
- xupybd 6y ago
  Rsync.net is amazing for Linux servers. For windows servers backups are complex and expensive. I tend to offload that to a cloud provider like Azure. Onsite I rotate hard drives. But for desktop users backblaze does everything I need. If anyone has a recommendation for backing up Windows servers I'd love to hear it.
  - jabroni_salad 6y ago
    It looks like rsync.net is indeed compatible with Windows, just perhaps not out of the box. Keeping in mind that SSH on windows is somewhat new and I haven't really tried it with a service like this yet. If you can get command line access to rsync.net with openssh and either CMD or Pwsh, then robocopy can forklift your stuff. This is without even getting into the weeds of the fact that WSL exists... I am also seeing that some documentation exists for pointing Veeam at it, which is my preference. I don't run any metal computers that aren't hypervisors and using that to back up my VMs, be they windows or linux, is my preference.
- jsmith99 6y ago
1vuio0pswjnm7 6y ago
Would be interesting to see those shell scripts for sending SMS via Twilio.
- anderiv 6y ago
  I'm not sure what John is using, but they have a very simple example in their documentation. Go here and then click on "twilio-cli" in the right code type selector: https://www.twilio.com/docs/sms/send-messages https://www.twilio.com/docs/sms/send-messages
  - secabeen 6y ago
    Note that twilio-cli is a totally over-weight, un-necessarily complicated node.js app. If you just want to send SMS from the command line, the curl code is much, much cleaner.
    - rsync 6y ago
      I have not used twilio-cli for anything ... I just write my own scripts with curl - here is my basic 'sms' command: https://0x.co/6K37UZ https://0x.co/6K37UZ
      - deleted 6y ago
        [deleted]
      - 1vuio0pswjnm7 6y ago
        Not everyone can share text using their very own pastebin. That's neat. Cheers.
tiffanyh 6y ago
@rsync If you had to do it all over again, what would you do different (if anything)? E.g. product/positioning/tech-stack/employees/business-decisions
- rsync 6y ago
  That's a really good question ... In terms of product / tech-stack I don't think I would change anything. In terms of marketing and word of mouth I think we should have given away hundreds of free accounts in the early years (2006-2010) rather than trying to chase them down as paying customers. I believe we had a lot of decent word of mouth but I don't think I appreciated the power of influencers and their ability to amplify a message. As for business decisions, I continue to wonder how much business we miss due to not having a Canadian location and we have considered deploying in Montreal for years now but have not pulled the trigger. I don't know if a Canadian location (but still a US company) solves the regulatory requirements of Canadian customers.
  - srhngpr 6y ago
    A Canadian location does solve the regulatory requirements of Canadian customers. Even federal government agencies in Canada no longer have issues using US-based hyperscalers (e.g., AWS, Azure, GCP) that have Canadian datacenters.
  - ksec 6y ago
    >but have not pulled the trigger. Is this a ( lack of ) capital issue or simply an uncertain sustainable revenue stream issue?
  - bombcar 6y ago
    I think giving away free accounts as a targeted marketing campaign (think - free accounts for the ZFS developers) can be worthwhile - but trying to have a free plan whilst offering the level of support that makes it worth paying for can end up being an exercise in futility. Even though I love free plans I think it’s better for small startups to grow organically with “cheap and easy to cancel” instead. Or offer credits for new users.
poisonborz 6y ago
This was maybe the first service I see that was somewhat complex, but the 4 line main page header text clearly explained what the tool does - the subpages are also great, low-key, great reads. Kudos to whoever copywrote the site.
- brap 6y ago
  Yes, the site is really well written, and I absolutely love its "no bullshit" approach which obviously extends to the product itself (and the CEO, from what I can tell). Signal to noise ratio is great. I wish more companies were like this. Having said that, I do think the site would really benefit from a new paint job. A good UXer can make it so much more aesthetically pleasing, while still retaining its simplicity and quick load time. It doesn't have to be fancy. Just static HTML with elegant styling and a few minor tweaks. For example, I was really surprised to see big name clients such as Disney, 3M, ARM & ESPN hidden a few clicks away (behind a button which wasn't very informative, from what I remember). Same for being in business for 20 years. A good UX/product person will tell you to put this front and center in your landing page, and rightfully so. @rsync: I love what you're doing, but please get a UX person involved :)
bacbilla 6y ago
+1 on having your laptop as an ephemeral device
- erik_seaberg 6y ago
  Yeah, assume it’s disposable not just for theft but because upgrading might be impossible and even repairs are very expensive (compared to a desktop).
richardfey 6y ago
I think they need to hire someone that is strong on the security side of the business, for two reasons: * he appears not aware of the role of hardware firewalls in mitigating DDoS by handling efficiently a lot of active TCP sessions (they have specialised hardware for this purpose) * he is describing in great detail a lot of information that a phisher or other type of hacker can treasure to target him
- tpetry 6y ago
  You cant protect from a DDoS with a hardware firewall, a DDoS consists of so much bandwidth that your network hardware is not able to simply handle the incoming traffic before any filtering happens. Your expensive hardware firewall can protect from DoS attacks, but they don‘t happen anymore as DDoS attacks are really cheap.
  - e40 6y ago
    Can you can protect yourself from certain types of things (SYNC flood) with a firewall, though.
    - richardwhiuk 6y ago
      It's easier to protect against SYN floods if you terminate the connection.
  - richardfey 6y ago
    In any situation where you need to create a filter rule that needs to run fast against a lot of TCP sessions a hardware firewall will do it faster than your general purpose server. It's not about protection but mitigation.
- rfd4sgmk8u 6y ago
  I read it this way too. Having the OS kernel have to process all internet junk traffic (including all the IoT worms) wastes CPU cycles that could be used serving traffic to legitimate users. It is better to offload the packet filtering to a dedicated device, with an OOB management. If the host was flooded, one might not be able to activate a host based firewall to fix in such a DDoS condition.
Crontab 6y ago
John's usage reminded me of something I read in Rob Rike's "Uses This" interview[1]: "I want no local storage anywhere near me other than maybe caches. No disks, no state, my world entirely in the network. Storage needs to be backed up and maintained, which should be someone else's problem, one I'm happy to pay to have them solve." [1]https://usesthis.com/interviews/rob.pike/ https://usesthis.com/interviews/rob.pike/
- chris1993 6y ago
  Essentially a Chromebook
  - wwalexander 6y ago
    It’s worth reading the rest of the interview, I find Rob Pike has a very interesting/unique take on the current landscape given his involvement with Plan 9: > Now everything isn't connected, just connected to the cloud, which isn't the same thing. And uniform? Far from it, except in mediocrity. This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary.
    - iamevn 6y ago
      If you want to see more on this theme, the Upspin docs[1] are a really interesting read. This 2017 talk[2] that Rob Pike gave on it is also really good. [1] https://upspin.io/ https://upspin.io/ [2] https://youtu.be/ENLWEfi0Tkg https://youtu.be/ENLWEfi0Tkg
      - Crontab 6y ago
        Thank you! I hadn't seen this.
      - 4ad 6y ago
        To bad development has stopped.
  - techrat 6y ago
RaitoBezarius 6y ago
You write down that you have no router, though your primary US location is connected to a "quintuple-homed network" and all global locations are at least triple-homed. What does that mean exactly? Is your IP provider quintuple-homed? Or are you running a bit more complicated setup than you explain but the gist is that you have no particular routing mechanisms? What does that say regarding your high availability? If one of your location is down, then it's definitely down until being fixed? Anyway, that was interesting, just curious about the fact of having no router at all. Thanks!
- rsync 6y ago
  The primary US location, in San Diego[1], gives us a managed, blended bandwidth product which is, in fact, quintuple homed and has been since we moved in (2001). So we have a dumb switch in our rack, but they have routers. In 2021 that's a weird bandwidth product and a weird setup but in 2001 it was "normal" and we just stay with that setup out of inertia (and the fact that we can't connect to he.net in San Diego). A similar setup exists for us in Zurich with init7. However, you are correct and we need to edit that FAQ language: our geo-redundant site in Fremont does not work that way. (I will note that it has been 11 years since we put that location in place (he.net in Fremont) and it has zero minutes of downtime) A tremendous amount of complexity and attack surface are eschewed by living with that setup and we're always looking for new ways to make that tradeoff. [1] Castle Acess datacenter on Aero drive. Is now a KIO managed datacenter.
  - bombcar 6y ago
    Good old Castle Access - and they really seem to take the “Castle” part pretty seriously. We were there at a similar time - I probably saw the rsync.net servers.
- walrus01 6y ago
  I read it not as there are no routers anywhere, but that they've abstracted the problem of running the routers to their upstream hosting/colo/datacenter provider. Obviously there are routers and their systems are connected to somebody's ASN, or you wouldn't be able to reach them over the Internet.
pjs_ 6y ago
rsync.net rules
tfsh 6y ago
Meta: I really dislike the style of console.dev, the article is shunted to the left and leaves the rest of the screen real estate to be taken up by an - albeit pretty - but unnecessary piece of digital artwork. This - https://ibb.co/nzbFxjW https://ibb.co/nzbFxjW - is what the article looks like on my ultrawide which made for very uncomfortable viewing
- chewbaxxa 6y ago
  Not sure why you couldn’t just resize the window here?
  - tfsh 6y ago
    I can, however I don't think having to resize your browser window to comfortably view an article is a very good UX, especially when it could be rectified by positioning the content in the middle of the screen.
    - Dylan16807 6y ago
      Right, but can you comfortably view anything over there? Why maximize the browser in the first place?
  - ac29 6y ago
    Thats a bit of a chore with a tiling window manager if you generally have the browser on its own workspace - you either need to spawn new windows around the browser window to push it into the geometry you want, or add borders to it.
    - bigiain 6y ago
      It's not a great suggestion for, say, iPad users either...
      - Dylan16807 6y ago
        An iPad isn't nearly big enough to have this problem. This is what I get for iPad emulation: https://i.imgur.com/FriTf6X.png https://i.imgur.com/FriTf6X.png https://i.imgur.com/gLhqvFO.png https://i.imgur.com/gLhqvFO.png It looks just fine.
lokl 6y ago
I wish I had a personal use case where the pricing of rsync.net made sense. It looks like a great service. For now, I use Backblaze Unlimited. I realize they are not the same service, but Backblaze works for my personal stuff and the price is great.
- tiernano 6y ago
  I like backblaze, dont get me wrong, but my issue with them was their software is Windows Client and Mac OS only... No Linux or Windows Server offerings... My desktop runs either Windows Server 2019 or Linux... I havent run a desktop class verison of Windows on a phsyical workstation in years... As an aside, i use RSync and their Borg Backup option[1] for backing up my Linux box, and Windows is backed up to that Linux box too... works well... Borg can be gotten to work with B2[2], but its a bit more messing... [1]:https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html [2]:https://medium.com/@mormesher/building-your-own-linux-cloud-backup-system-75750f47d550 https://medium.com/@mormesher/building-your-own-linux-cloud-...
- audience_mem 6y ago
  > Backblaze Test your backups. https://messengergeek.wordpress.com/2018/03/09/backblaze-review-data-loss/ https://messengergeek.wordpress.com/2018/03/09/backblaze-rev...
  - lokl 6y ago
    Thanks for sharing this.
    - audience_mem 6y ago
      No problem. I hate the thought of data loss. They may be better now, but who knows, it's worth being sure.
  - bombcar 6y ago
    People think of backblaze as a backup/archive but it is explicitly not an archive - it is a backup and if you delete a file on your system it is purged within 30 days from the copy. Most people don’t really think about this and expect that any backup is AlSO an archive. You can get burned by this and have to use b2 or whatever it is instead.
poorman 6y ago
"I initiate my work in the terminal by port-knocking". Guess you don't need a firewall when you have no open ports? Haha yes! Guess I'm not the only one...
mfincham 6y ago
My first experience with rsync.net was very disappointing. To this day they still advertise “append-only mode” support for restic at https://www.rsync.net/products/restic.html https://www.rsync.net/products/restic.html. Their support people confirmed it doesn’t work (though they didn’t seem to understand why it would be fine for them to support it as advertised...) yet 6 months later they still advertise that they support it, even when I have e-mailed to remind them (and it still doesn’t work either) :(
- mfincham 6y ago
  The tl;dr as to why it doesn’t work is that they blanket forbid calling “rclone serve”, which is required for “append-only” support in restic. This doesn’t make sense given that the specific invocation of “rclone serve restic --stdio” doesn’t open any network sockets, it’s no less safe than e.g. “tar”
ynx 6y ago
The replacement for Spectacle, FYI, is Rectangle, which is almost identical, but still maintained
vzaliva 6y ago
Firefox Reade mode makes this page more readable. Otherwis 50% of your screen space is taken by non-informative graphics.
mattbillenstein 6y ago
So how is this sharded? Or how do you load balance a customer to the correct server if there's no router?
AdamJacobMuller 6y ago
I really appreciate the information and would love more information on your architecture in particular. Also definitely love your information on your personal process and flow. Some of it seems interesting to adopt and some of it seems bad for me but I could see why you do it, some I do in similar ways but a bit differently. e.g. I use git and online services to make sure that any of my computers are completely replaceable and that I can pick up work at any moment from any one. I can't agree more with the "no firewalls" approach to things, though I prefer to call it "host based" firewalls as it scares people less! I'm glad you've had no compliance/audit pushback on that, I architect things similarly and have had success pushing back on the requirement as well. I'm very surprised by the l2 switches and actually choosing to run completely unmanaged switches. I assume you're running all 10G or more? Maybe i'm overthinking the complexity of your network but I would be lost without snmp counters on my switches and running switches+networking in fully l3 mode has some great isolation benefits, especially if you want full switch-level redundancy. Do you have some more details on your data architecture? I'm very curious how do you do data direction/redundancy/sharding and balancing customer data across servers. I'm not trying to pry for things you consider secret but I think you have a very similar architectural mindset and I'm curious how you solve these things.
peppermint_tea 6y ago
happy customer here. I do a simple rsync of my precious but not too sensitive data, daily. and for the more sensitive stuff, gpg before sending daily as well, the copies will add up but I prefer it that way. 10/10 great business
crazypython 6y ago
7 daily snapshots: So I could sync my hard drive over, wait for a snapshot, delete everything, and keep the space, and use the snapshot as backup?
gautamcgoel 6y ago
The pricing model doesn't make sense to me. Their prices start at $0.025/GB/month, so renting 1TB of storage for a year would cost $300 - at that price, I could just buy my own disks and run ZFS myself. I kinda hoped they could offer lower prices using economies of scale. I checked the prices for Tarsnap, expecting it to be cheaper - it's actually 10x more expensive! Maybe someone can explain what I'm missing.
- cat199 6y ago
  DIY is way cheaper, true. but for comparison, aws is ~$100/tb/month
  - generalizations 6y ago
    However, also for comparison, Backblaze is ~$5/TB/month.
  - dividedbyzero 6y ago
    AWS Glacier is ~$4/tb/month. Getting data out of there costs extra, but for backups of last resort you don't expect to ever need, that may be a workable tradeoff.
    - bombcar 6y ago
      IIRC rsync.net doesn’t charge for bandwidth in any direction which for some use cases is nice - a set and forget billing type.
  - thw0rted 6y ago
    For a backup use case, I haven't been able to beat the value of a Microsoft premium-whatever family plan. Routinely on sale for $60/yr, it gets you 1TB of backup plus an Office license for 6 users. Obviously you don't have as much control over it as "bare" cloud storage but it's hard to match the price.
- anderiv 6y ago
  Sure, you could purchase your own drives. But then it wouldn’t be offsite. And it probably wouldn’t be on a redundant internet connection. And most likely not have redundant power and cooling. And, and, and. Self-hosting at home (or in the office) is a great option for some if you’re not worried about needing an offsite backup. For those that do care about this sort of thing, though, the extra you pay to have someone else manage the thing is well worth it.
travisgriggs 6y ago
> I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit. I like that. I like that a lot. That's a very enviable practice. I think I know what I'll be experimenting with this next week. Did not expect to read that article and have the most stand out thing be a routine change I'd want to copy. You never know.
- prox 6y ago
  It’s one of the best “lifehacks” , walking, biking and the likes in a traffic free or low environment to just clear the mind. By the end of the walk you’ll be full idea’s, or just ready to start the day. It’s coffee but without stimulants :)
- driverdan 6y ago
  Get a dog and you won't have an excuse for skipping a morning walk.
zarkov99 6y ago
For people who use rsync.net, is this something that can replace Dropbox for multi-machine synching? For all its flaws, Dropbox does allows me a semi-seamless transition between my laptop and my workstation.
- eythian 6y ago
  As far as I know, no, unless you manage aspects of that yourself via git-annex or something similar. My setup to do this is that I run my own nextcloud server, which handles the computer and phone etc. syncing, then nightly that's backed up to a small computer in my house (I just use rsnapshot for that), which then backs itself up to rsync.net (using plain old rsync.)
- tecleandor 6y ago
  I think Syncthing could be better for those purposes. I use that for synching my homes (config files and essential work files, keys and so on) between my desktop and laptop. I have a copy running in my NAS to always have a copy available, one in my laptop, one in my desktop, and I was thinking about having one in my phone to run only when I'm charging (so I don't kill my battery).
  - zarkov99 6y ago
    Can you compare Synching and Dropbox? I have been a (paying) Dropbox user for many years but the product is not that good, especially on Linux. I would love a more reliable alternative.
    - tecleandor 6y ago
      Syncthing is "just" a syncing daemon with a simple status web dashboard. The pros (at least for me): - You host/own it - It's not centralized: every node can sync with the others - Seems to be fast (the more nodes, the merrier) The cons: - Setup is slightly more difficult (you need to share some keys and ips) - No iOS client (not an issue for me) - You can't share a folder or file via web link So it's great for syncing folders between systems, but it doesn't substitute the job that something like Seafile would do (managing permissions, collaboration, web file sharing, fancy web ui...) There's a bit of info in the faq: https://docs.syncthing.net/users/faq.html https://docs.syncthing.net/users/faq.html
KingOfCoders 6y ago
Used them in a startup for a long time, was very happy, excellent support, good pricing. Would always use them again. (used the Swiss location).
shydwoo 6y ago
Why would i buy 1TB for $20 per month here instead of getting 6TB for $8/month from Microsoft?
- kqr 6y ago
  Technical excellence and out-of-this-world support. I say this as a very happy rsync.net customer for over 10 years. Though if the descriptions of the service on their web page does not make you salivate, perhaps it's not for you.
- bombcar 6y ago
  The limitations on OneDrive are considerable as it’s built on top of Sharepoint somehow. But rsync.net is a backup product and OneDrive is a large file sharing drive. Attempt to use that 6TB in backup situations and you may experience issues.
hannofcart 6y ago
I wish all SaaS services were like rsync - No nonsense description of what they do - Clear and simple pricing - Simplicity as a core feature Big fan. Look forward to using your services in the future.
limaoscarjuliet 6y ago
Question for rsync: You said: This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security. You seem to suggest the big firewalls do not bring any value to the table. I always thought they had more "intelligence" - dropping sessions based on some bad patterns, guarding against DDoS (to some extent), etc. Are you saying BSD is as good as these expensive boxes? Does it apply to SSH only or HTTP(s) and some other traffic as well?
- nix23 6y ago
  OpenSSH and the OS is pretty much the best place to harden your SSH connection, no need for a Firewall.
oilbagz 6y ago
I bought an rsync.net account a few years back when John made it known on HN, and have used it solidly as a backup for my .. wristwatch! I have a LILYGO that I coded up a time-tracking app, which basically creates an event log whenever I tap it, wherever I go - and when Internet is available, it squirts the log over to some text files that live on rsync.net .. Pretty neat to be able to do this without much of a desktop or mobile phone in the way, I have to say. I wonder if there are more opportunities for this kind of IoT service out there .. it sure was fun to get this working without REST ..
nix23 6y ago
Clean, simple architecture...a sysadmin's dream.
ElectricMind 6y ago
//I try to maintain an “Hedonic Fast” Monday through Wednesday so, on those days, I am only looking for truly actionable headlines and comment threads that are relevant to my businesse// This is smart move!
yyyk 6y ago
Question to rsync: Which HDs does rsync use? Is there a preferred brand? Which brands have been found to be most reliable?
simonebrunozzi 6y ago
I like the interview a lot, but oddly this one here is the part that I liked the most: > I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit.

654 points by dmytton 6y ago

Don't know if running a dumb switch connected to your ISP is the best infosec policy: https://blogs.cisco.com/manufacturing/the-top-5-reasons-to-avoid-unmanaged-switches-on-your-machines https://blogs.cisco.com/manufacturing/the-top-5-reasons-to-a...

Jonnax 6y ago
I'm not sure those reasons really apply to their case. Especially since they're running the boxes that it's connected to. They can do resiliency, network segmentation, and monitoring on their platform. What's a Cisco box going to do for them?
- sideshowmel 6y ago
  Dumb switches will blast packets to all interfaces that are connected. If there's a machine on the switch that's in promiscuous mode, it can see all the packets on the local network (including the backups coming in from customers). Managed switches typically have ACL support. I get the KISS principle, but this setup seems to be trading security for simplicity.
  - noir_lord 6y ago
    > including the backups coming in from customers. Which are encrypted in flight...if they aren't then anyone on the 30 machines between customer and final destination can also see the backups coming in from customers.
    - sideshowmel 6y ago
      True, but the packets in-flight can take different routes. If you have a machine on the switch, you know you've captured all the packets that were in-flight. This make it easier to break the encrypted packets. It's a choice--everything in security is a risk-management assessment, but I'm surprised rsync.net was able to get so many security certifications with this setup.
      - noir_lord 6y ago
        > If you have a machine on the switch, you know you've captured all the packets that were in-flight. Same applies if someone takes over the firewall, machine on the last hop before they hit port 22. In a world where stuff like this https://www.helpnetsecurity.com/2020/09/01/zero-day-cisco-enterprise-routers/ https://www.helpnetsecurity.com/2020/09/01/zero-day-cisco-en... routinely happens there is a benefit to forgoing all of that when it makes sense.
      - mcosta 6y ago
        # tcpdump -i eth0 tcpdump: eth0: You don't have permission to capture on that device (socket: Operation not permitted)
  - ptomato 6y ago
    > it can see all the packets on the local network I'm sure those packets (consisting entirely of OpenSSH) will be very useful to them
    - sideshowmel 6y ago
      Don't be so sure :) Quantum computing is improving everyday, and new methods of defeating RSA are being researched: https://eprint.iacr.org/2021/232 https://eprint.iacr.org/2021/232
      - anthk 6y ago
        OpenSSH now uses eliptic curves, not RSA.
  - dividuum 6y ago
    They only support SSH (legacy FTP was sunset a year ago), so there's nothing to gain (except for maybe the volume and IP of the customer) by observing other traffic. Which happens to be the same information you can observe anywhere in the path from a customer to their machines.
  - iso1631 6y ago
    > Dumb switches will blast packets to all interfaces that are connected Multicast and broadcast sure, but dumb switches will still keep mac-address>port mapping. If the router sends to 52:54:00:ad:ra:a7, the dumb switch will remember that's on port 7 (having seen traffic from it recently - if only an arp reply) and only send the packet to port 7. Hubs (remember them!) will blast every packet to every port.
  - EvanAnderson 6y ago
    Thie first paragraph is incorrect. A hub will "blast packets to all interfaces that are connected". A switch, even a dumb one, still switches packets. Broadcasts and frames addressed to unknown destinations will flood out all ports, but not unicast frames with destinations currently in the MAC table. It is true that an attacker could flood the MAC table, spoof their MAC, etc, after compromising a layer-2 adjacent host and use that to manipulate traffic flows. That's somewhat disturbing, but no Customer backup data should be hitting their network outside of SSH anyway. I think the potential is more for DoS than compromise of confidentiality or integrity. I really admire rsync.net's simplicity, but dumb switches give me the willies. I feel blind not having per-interface counters, at the very least. If nothing else, I'd like to be able to reconcile the counters coming from my OS interface with the switch in troubleshooting scenarios.
  - deleted 6y ago
    [deleted]
  - fomine3 6y ago
    You may mean repeater hub.
  - deleted 6y ago
    [deleted]
iso1631 6y ago
The only "security risk" i see there is number 1, and that is all to do with physical security. > Disadvantage #1 – Open ports on unmanaged switches are a security risk Why? Is there something that would prevent an attacker with physical access from unplugging an existing cable? Does the average managed switch config have mac limits and auto shutdown if a link is lost for just a few seconds? Mac limits are easilly bypassed, even without (permanently) disconnecting the legimate device by inlining an active device, maybe some mac spoofing. I don't include 802.1x or automatically shutting down a port that loses an uplink as a "simple and effective security precaution", it would be a right pain for many situations. Is the latter even a feature? I certainly haven't come across it (unlike normal portsecurity like limiting number of mac addresses, which just adds to overhead with limited effective security). > Disadvantage #2 – No resiliency = higher downtime If my device has one ethernet cable into one switch, how does that help? If my unmanaged switch goes pop, I have a spare that I can put in and be back running in a minute. My managed cisco edge switches take 10+ minutes just to reboot. If my device has two ethernet cables, one into one unmanaged switch, one into another, losing that switch isn't a problem. > Disadvantage #3 – Unmanaged switches cannot prioritize traffic Correct they can't. Managed switches without qos set up can't prioritise traffic either. If your switch is dropping packets, you don't have enough bandwidth. I've seen packet loss when sending 500mbit down a 1G uplink on managed switches, even on QOSed traffic. Indeed I've seen higher priority traffic drop and lower priority not drop. QOS isn't trivial. Ultimately it comes down to how big your buffers are whether your packet gets through or not, so your application should cope with some loss, and if you get too much loss you need more bandwidth. If you have 48 devices connected at 1Gbit each, each firing 100mbit of traffic every second, all bang on the second, with a 10gbit uplink, on paper you only need 4.8gbit of uplink. You'll also need a 600MB packet buffer and expect a lot of delay on your packets, whether you have managed or unmanaged, QOS or no QOS. > Disadvantage #4 – Unmanaged switches cannot segment network traffic Correct, but then if I have 8 desktops in a cluster why wouldn't I pop in a desktop switch with 8 1G ports? I want them all on the same vlan anyway. > Disadvantage #5 – Unmanaged switches have limited or no tools for monitoring network activity or performance They don't, but again do I want that for a specific use case? If I want a managed switch (which I usually do), then I'll spec a managed switch. It's unlikely it will be cisco. If my requirements don't need features of a managed switch then I won't bother. I find it interesting that there's no mention of preventing broadcast storms, or IGMP snooping - both of which are far more useful for a typical edge switch than qos. Personally, I tend to use managed switches - indeed I just bought a couple of 24 port TP Link POE switches for an event I'm planning. I'm not 100% sure I'd go for an unmanaged switch in rsync's case, but from your list 1) Doesn't apply -- servers are in a secure location 2) Doesn't apply -- servers are either single connected (so need a physical visit, and replacing an unmanaged switch is far quicker and easier than a managed switch), or they're dual connected to two different switches 3) If they're doing inline management then you might want to carve out a small part of your uplink to prevent yourself from being dossed by a dodgy server (if your server is saturating your uplink bandwidth and you ssh session can't establish that could be an issue. If you've got OOB access on a separate link though, not a problem, and clearly they don't have that problem) 4) Doesn't matter -- they don't want different vlans 5) They presumably measure the bandwidth use of each of their servers. The question thus is "does the ISP give me logs I can rely on for the wan". Personally I wouldn't, but I can see the idea Spanning tree: Secure network, they aren't going to connect one port to another to cause a storm IGMP: They presumably aren't using multicast for anything major so bitrates would be very low even if they were there Reasons to use a firewall or a switch with an ACL in this specific case that I can think of: 1) 2 points of control -- a zero-day on freebsd's firewall could open a port to an unintended source which was listening but blocked by iptables (or bsd's version). If you had a non-bsd firewall it's unlikely the same zero-day would work 2) Port 22 is only open to a specific IP range, again there's a zero-day, and TTL of outbound packets is high enough to establish a session Reasons to use a managed switch even ignoring firewalling: 1) Reliable traffic stats -- you could guess at these by summing the uplinks of all the connected devices although some packets will be dropped and some may be going to other devices on the network Reasons to use QOS on a managed switch: To allow inband managment if something goes wrong. A separate ilo/ipmi/kvm connection would be better for that though. I don't think they'd need features like span ports (I personally use them all the time, and fibre taps, but I have a different use case which is UDP heavy and loss-intollerent)

rsync 6y ago

Unrelated, as an aside ... I really am enjoying the developer Q&A interviews that console.dev is putting out. They're very much like the "usesthis"[1] profiles but more in-depth and with more interesting details ... [1] https://usesthis.com/ https://usesthis.com/

mattl 6y ago
It was an interesting read! I did a usesthis a little while ago. https://usesthis.com/interviews/matt.lee/ https://usesthis.com/interviews/matt.lee/

aDfbrtVt 6y ago

Thanks for the interview, I was pleasantly surprised to see how simple the network architecture is at rsync.

ttsiodras 6y ago

Interesting interview - thanks John! Didn't know there was a UFS2 "phase" before ZFS... I wonder how much time those fscks took! :-)

rsync 6y ago
They took forever ... and then they bombed out due to lack of memory. Not lack of physical memory, but lack of ability to address it as the UFS2 tools, like fsck, were not written to handle billions of inodes ... We really can't thank Kirk M.[1] enough - he wrote custom patches to ufs and fsck just for our (dirty) filesystems and, as I mention in the article, eventually gave us the push to migrate to ZFS. [1] https://en.wikipedia.org/wiki/Marshall_Kirk_McKusick https://en.wikipedia.org/wiki/Marshall_Kirk_McKusick
- cvwright 6y ago
  Wow, what a cool story. Having Kirk McKusick write patches for you is almost like that old Weird Al song: > I’m down with Bill Gates > I call him “money” for short > I phone him up at home > And I make him do my tech support

Aeolos 6y ago

> "I have a early-2009 “octo” Mac Pro [...]" > > OS: macOS Does this make anyone else a bit uncomfortable? I don't think MacOS is still receiving security updates on that hardware. I'm all for using old hardware for as long as it keeps working, but I would never browse the internet with a vulnerable OS on a vulnerable processor (spectre etc...) Or am I missing something?

lunixbochs 6y ago
They're possibly something like a dosdude patcher or modified bootloader to run an OS like Catalina on it.
- Alupis 6y ago
  I have trouble understanding why people go through these hoops. Yeah, I get it, people love their Mac's... but the company that produces them actively undermines your ability to continue using perfectly good hardware past what they feel is "profitable". This leads to huge efforts to hack/reverse the updaters, or alter newer OS versions to trick them into installing, etc. I'd personally jump over to some system that doesn't hate it's users nearly as much. But, that's just me.
  - boardwaalk 6y ago
    Do you really need to pivot into Apple bashing on this thread? It’s not really on topic or needed.
  - gambiting 6y ago
    It's not out of some love for Macs. I have a 2008 MacBook running Catalina and it's simply because the cost of replacing it is >0. If this works and works well(and it does) then why would I get rid of it? Just to spite apple, which doesn't care either way? I also have a 2005 car that still runs - should I get rid of it because the company that made it stopped providing any kind of support for it long time ago? Or you know....keep using it because it works?

ciil 6y ago

Jealous of how well you seem to be able to keep to KISS as a principle.

rhizome 6y ago
The number of "simplicity? what's that?" brain-implosions in this thread is kind of hilarious, though at the same time a little concerning.

booi 6y ago

A simple layer 2 network topology only works in very narrow use cases (like this one). But a "dumb switch" means you also lose a lot of observability and it's very difficult to apply consistent network acls.

rsync 6y ago
Agreed - we are, in a sense, "cheating" because our product is so simple that we do have one of these "very narrow use cases". The benefits are tremendous, however, and go beyond day to day operations. A dumb switch has no credentials to protect and there is almost zero attack surface. Further, if our switch dies we can immediately replace it with any other dumb switch that just happens to be lying around. If you read failure studies - like those in the excellent Charles Perrow book _Normal Accidents_[1] - you see that in many cases there is a very special component that fails and everything goes to hell when they can't find a replacement for it. So, while I can't encourage everyone to use dumb, unmanaged switches (because not everyone can) I can encourage everyone to remove as many very special components as they can. [1] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
- _trampeltier 6y ago
  I work in industrie automation and I can't agree more to dumb devices. There are a lot of nice special products, but if something goes broken, you have first tousend of pages manual, just maybe an identical part. The guy on the night shift has also to know this special part well .. and so on. The dumb devices, you can replace easy without any problems tomorrow or also in 10 .. 20 years.
- chris_wot 6y ago
  Charles Perrow only died recently, very sad.
- Bluecobra 6y ago
  How are you providing network level redundancy with dumb switches? My only guess is that the ISP is already doing HSRP/VRRP on the gateway and you can setup multiple NICs/switches with something like CARP and being careful not to make L2 loops.

robotmay 6y ago

Nice article. rsync.net is one part of my personal computing setup that I never even think twice about. It's simple and it works, and that clearly applies to the infrastructure too. I use ZFS locally and it has made managing my own data strangely pleasing, and it's nice to have the same system on my off-site storage too. On the laptop-front, I find myself drifting towards a similar setup to John. I have a hefty workstation laptop but the battery life is dire and it weighs a ton, so I pretty much just run it as a headless machine next to my server now. I'm planning on picking up a Pinebook Pro as an "outdoors" machine to just remote in. I also find myself extremely unwilling to arse about swapping multiple machines on my monitors so being able to keep my work machine separate and secure but operate it from my desktop is a nice compromise.

nicolaslem 6y ago
I would love to use this simple setup as well. It's too bad ZFS snapshots cannot be sent and stored encrypted. I would love to use rsync.net but the idea to have my data sitting in someone else's computer in plain text feels wrong. So instead I have to use restic, which re-implements many features of ZFS and this also feels wrong.
- rsync 6y ago
  You can 'zfs send' to a (special kind of) rsync.net account. We support encrypted zfs[1][2][3] and raw-send, etc. The pricing is the same but there is a 1TB minimum because we need to give you your own VM (bhyve) and we have to burn an ipv4 address for you, etc. [1] https://www.rsync.net/products/zfs.html https://www.rsync.net/products/zfs.html [2] https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync... [3] https://www.servethehome.com/automating-proxmox-ve-zfs-offsite-backup-rsync-net/ https://www.servethehome.com/automating-proxmox-ve-zfs-offsi...
  - blibble 6y ago
    could you allocate the VM on demand? xinetd style (you could route the ssh traffic similarly based on login)

trollski 6y ago

i could get cloud storage from Microsft at ~1/20 of the cost. why would i use rsync.net?

frammie 6y ago

Really well done interview, some real interesting bits in there. One part concerned me though, in the interview, it mentions "we own (and have built) all of our own platform." and it fails to mention a few critically important key parts of a storage platform, first being encryption. How are personal files being handled? Is encryption being used? Are you able to access this data using a shared key? As well as contingency, what happens if critically important data is stored on your platform. On your website you mention: "We have a world class, IPV6-capable network with locations in three US cities as well as Zurich and Hong Kong" however fails to mention if replication is done across these locations. If technology (drives) is stolen from your datacenter, or mechanical failures beyond your control happen, how will you be able to recover from physical failure if you only appear to be serving from a single location? Excuse me if I'm wrong but I couldn't find anything concrete in either the interview or your website. The premise of the platform seems quite well aligned with keeping alive the the UNIX philosophy, and reminds me of Tarsnap. Either way, well made interview and interesting approach to a storage platform. As a sidenote, what keyboard are you using? It seems really interesting and you failed to mention it in the interview :) EDIT: It appears that you offer Geo-Redundant Filesystem as as separate product, maybe you would want to make this a bit more visible on your website except for only the FAQ and order pages. Either way, it seems like a sufficient move, that does still leave the topic of encryption though. As mentioned traffic is encrypted using SSH ofcourse, but is the data itself encrypted on your platform?

dharmab 6y ago
I've used rsync.net in the past- it's essentially "filesystem as a service." You, the customer, use it to back your own software that handles the encryption and replication. Their website has some how-to guides for some common software, or you can roll your own with the rsync protocol. Notably, their website only claims transfer encryption, not encryption at rest. You can of course encrypt your files yourself with your own keys.
- frammie 6y ago
  Not having data encrypted by default is concerning, however I do admire the simplistic approach of handling your own dataflow and tools for sure.

antongribok 6y ago

Reading this takes me back to when I started playing with storage professionally. For me it was in 2004, also using 3Ware controllers. I was running on RedHat (before RHEL) and XFS before it was common on Linux, and similarly had memory issues when trying to repair filesystems.

sparkling 6y ago

Hetzner has a similar product at better pricing that i have been using a minimalist dropbox alternative https://www.hetzner.com/en/storage/storage-box https://www.hetzner.com/en/storage/storage-box Access via rsync/sftp/scp

fuzzy2 6y ago
Also, Borg backup.
- formerly_proven 6y ago
  rsync.net was the first storage provider to support Borg out of the box and also has a special tier for Borg users (which was later expanded for restic and some others iirc). I also like that their Europe location is in Switzerland. I think it's useful for a number of reasons to store critical data in more than one jurisdiction.
foepys 6y ago
Your link is dead for me. Maybe you wanted to link to this? https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box Hetzner is throttling bandwidth after traffic exceeds ~5x the storage capacity while rsync.net doesn't seem to. Hetzner also only supports a very small number of snapshots in total while rsync.net supports more per day. I don't think Hetzner and rsync.net are really competing with each other. rsync.net's focus is more on business customers, while Hetzner targets private customers.
CameronNemo 6y ago
I tried to view the link. Got a site not found error.
jsmith99 6y ago
It seems a very similar product, also offering zfs snapshots, but I like the fact rsync.net snapshots are immutable: you can browse them but there is no way to delete them without contacting support (and the CEO once posted he would review every such request). It makes me feel more confident about my backups if someone got hold of the cached credentials from my backup software.

kplex 6y ago

Is rsync.net related to rsync the project?

rsync 6y ago
No, there is no relationship. However, in 2005 or 2006 when we spun out of JohnCompanies[1] and incorporated under the name "rsync.net" I requested, and was given, explicit permission to use the name and domain by the maintainers of rsync.

tyingq 6y ago

I do get the "no separate firewall" reasoning, but I'm paranoid enough that I'd at least want some PF rules just in case some daemon gets started by accident.

formerly_proven 6y ago
Oh I'm pretty sure there is a firewall configured on the nodes themselves (customers get shell access) and he just meant that there isn't a separate firewall box in front of the servers.
- rsync 6y ago
  Correct. The storage arrays themselves have a (modest) ruleset which, among other things, locks them to TCP22 only and disallows broken/impossible things like xmas-tree packets. Simple stuff.

korethr 6y ago

I wonder if they have any sales to large enterprises or similar institutions. In my experience, the larger organizations will have a "security" questionnaire required of their vendors, and the person administering it is a droid, incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today, or to the particular product/service/vendor in question. And no firewalls or routers would be massive, disqualifying red flags on such a questionnaire. Never mind that a KISS setup tends to bring security because of its minimized attack surface. In the minds that write and administer those questionnaires, security only comes from sufficient amounts of the right kinds of complexity. I'm sure it can be done. IIRC, Cloudflare doesn't use any firewalls, and they do some big business. It just isn't easy to get past the droids programmed to ensure that all pegs shall be properly square, IME.

rsync 6y ago
"I wonder if they have any sales to large enterprises or similar institutions." Yes, certainly. We frequently fill out very detailed checklists and questionnaires related to our quality policy, standards, internal policies, etc. We're also very honest about how we approach these issues: https://www.rsync.net/resources/regulatory/pci.html https://www.rsync.net/resources/regulatory/pci.html ... and they generally appreciate the honesty.
- chris_wot 6y ago
  Man, everything about your service is simple and direct! Amazing.
- learn_more 6y ago
  FYI, your "pricing" link at the top of that pci.html page 404's. The pricing link works from other pages however.
  - rsync 6y ago
    I see that that has now been fixed - thanks for pointing it out.

canoebuilder 6y ago

With regard to the iOS import/export mentioned, does anyone have any more recommendations? (I'm not familiar with the mentioned option, nothing against it, just seeking out all options) Simple file system interface to all devices first, then any further software interfaces on top only if desired. Thanks for making the option available for remote storage John!

bflesch 6y ago

Big fan of rsync.net but the firewall comment caught me a bit off-guard. The benefit of a firewall is that it's an isolated system which - apart from port blocking - guarantees a certain level of traffic logging and known-good state. If you have everything on one host I'd say your overall setup on that host becomes much more complex because you only need to get hit by one successful exploit chain and all logs on that host cannot be trusted any more.

klodolph 6y ago
On a reasonable-size setup, I would expect that the logs are exported to dedicated log storage (log-only machines) as part of an effort to preserve accurate log files even in the case of a successful attack on one of the hosts. It is not especially hard to ensure that, for example, a record of an SSH login attempt gets recorded to an external server before the request is authenticated. So if you have (for example) an SSH account and a local privilege escalation exploit, there is still some evidence in the logs. In the past, the benefits of a firewall were more clear-cut, but these days I think that it’s reasonable to have “defense in depth” without using a firewall as part of your solution.
- hertzrat 6y ago
  The firewall is still helpful in case they hire a new person who opens a port and forgets to close it one day
  - dsr_ 6y ago
    "Steve, did you open a port? We only use SSH. What's going on?"
    - hertzrat 6y ago
      Oversights happen https://thenewstack.io/junior-dev-deleted-production-database/ https://thenewstack.io/junior-dev-deleted-production-databas...

api 6y ago

I go by the rule that if something is not secure enough to plug directly into the Internet, it is not secure. That doesn't mean I'll necessarily do that, but that should be the bar. The only exception is special purpose backplane networks that are designed explicitly to be isolated. These are basically data busses for clusters, not user-facing networks.

efxhoy 6y ago

That was a nice read! Good to read about something simple after a day working with AWS and their managed magic. Scrolling through the cert pages 2015 seems to be in the future though? > We personally toured every single major datacenter in Hong Kong and Zurich to choose the facilities that best met our old-fashioned standards for datacenter and telco infrastructure. The same will be true of our upcoming Montreal location in Q4, 2015. https://www.rsync.net/resources/regulatory/sas70.html https://www.rsync.net/resources/regulatory/sas70.html

anderiv 6y ago

This was a pleasure to read. I've been an rsync.net customer for ~6 months now, and am using Borg to send de-duped, encrypted backups to rsync.net from a few on-premise linux systems. As compared to other similar backup systems I've used, it's been a pure pleasure to implement and maintain. Thank you for your great product and support, John!

jeffbee 6y ago

I always liked this set of marketing materials. But I also see where they conflict with my experience. "You may visit our datacenters any time you like for a personal tour and inspection to satis[f]y whatever due diligence requirements you may have" probably appeals to many customers, but for my dollar I would prefer a datacenter that nobody may enter.

mcosta 6y ago
> I would prefer a datacenter that nobody may enter. If a disk break, who changes it?
- jeffbee 6y ago
  Not tourists, clearly.
mypalmike 6y ago
Presumably the racks are in cages and visitors aren't able to actually touch them.

ChrisArchitect 6y ago

I don't care for newsletters on tooling, but these Q&A interview posts are good -- immediately went in search of a twitter, couldn't find due to difficult naming, but want to follow to keep up from time to time https://twitter.com/consoledotdev https://twitter.com/consoledotdev

hertzrat 6y ago

I used to run Linux for everything but I’m having to use Windows these days. What would it take to get rsync.net playing nicely with windows? I’m imagining Windows subsystem for Linux (ubuntu) with duplicity installed to it? Are there any major hiccups to that sort of setup?

pfortuny 6y ago
rclone should work, afaik.
rsync 6y ago
From the standpoint of random access "browsing" over SSH/SFTP, you could just use filezilla or WinSCP or ... psftp.exe. However, if you want a backup process then you will, indeed, need to find some way to run 'borg' or 'restic' or 'rclone' on Windows. I've never used WSL so I can't comment, unfortunately ...
xupybd 6y ago
Rsync.net is amazing for Linux servers. For windows servers backups are complex and expensive. I tend to offload that to a cloud provider like Azure. Onsite I rotate hard drives. But for desktop users backblaze does everything I need. If anyone has a recommendation for backing up Windows servers I'd love to hear it.
- jabroni_salad 6y ago
  It looks like rsync.net is indeed compatible with Windows, just perhaps not out of the box. Keeping in mind that SSH on windows is somewhat new and I haven't really tried it with a service like this yet. If you can get command line access to rsync.net with openssh and either CMD or Pwsh, then robocopy can forklift your stuff. This is without even getting into the weeds of the fact that WSL exists... I am also seeing that some documentation exists for pointing Veeam at it, which is my preference. I don't run any metal computers that aren't hypervisors and using that to back up my VMs, be they windows or linux, is my preference.
jsmith99 6y ago

1vuio0pswjnm7 6y ago

Would be interesting to see those shell scripts for sending SMS via Twilio.

anderiv 6y ago
I'm not sure what John is using, but they have a very simple example in their documentation. Go here and then click on "twilio-cli" in the right code type selector: https://www.twilio.com/docs/sms/send-messages https://www.twilio.com/docs/sms/send-messages
- secabeen 6y ago
  Note that twilio-cli is a totally over-weight, un-necessarily complicated node.js app. If you just want to send SMS from the command line, the curl code is much, much cleaner.
  - rsync 6y ago
    I have not used twilio-cli for anything ... I just write my own scripts with curl - here is my basic 'sms' command: https://0x.co/6K37UZ https://0x.co/6K37UZ
    - deleted 6y ago
      [deleted]
    - 1vuio0pswjnm7 6y ago
      Not everyone can share text using their very own pastebin. That's neat. Cheers.

tiffanyh 6y ago

@rsync If you had to do it all over again, what would you do different (if anything)? E.g. product/positioning/tech-stack/employees/business-decisions

rsync 6y ago
That's a really good question ... In terms of product / tech-stack I don't think I would change anything. In terms of marketing and word of mouth I think we should have given away hundreds of free accounts in the early years (2006-2010) rather than trying to chase them down as paying customers. I believe we had a lot of decent word of mouth but I don't think I appreciated the power of influencers and their ability to amplify a message. As for business decisions, I continue to wonder how much business we miss due to not having a Canadian location and we have considered deploying in Montreal for years now but have not pulled the trigger. I don't know if a Canadian location (but still a US company) solves the regulatory requirements of Canadian customers.
- srhngpr 6y ago
  A Canadian location does solve the regulatory requirements of Canadian customers. Even federal government agencies in Canada no longer have issues using US-based hyperscalers (e.g., AWS, Azure, GCP) that have Canadian datacenters.
- ksec 6y ago
  >but have not pulled the trigger. Is this a ( lack of ) capital issue or simply an uncertain sustainable revenue stream issue?
- bombcar 6y ago
  I think giving away free accounts as a targeted marketing campaign (think - free accounts for the ZFS developers) can be worthwhile - but trying to have a free plan whilst offering the level of support that makes it worth paying for can end up being an exercise in futility. Even though I love free plans I think it’s better for small startups to grow organically with “cheap and easy to cancel” instead. Or offer credits for new users.

poisonborz 6y ago

This was maybe the first service I see that was somewhat complex, but the 4 line main page header text clearly explained what the tool does - the subpages are also great, low-key, great reads. Kudos to whoever copywrote the site.

brap 6y ago
Yes, the site is really well written, and I absolutely love its "no bullshit" approach which obviously extends to the product itself (and the CEO, from what I can tell). Signal to noise ratio is great. I wish more companies were like this. Having said that, I do think the site would really benefit from a new paint job. A good UXer can make it so much more aesthetically pleasing, while still retaining its simplicity and quick load time. It doesn't have to be fancy. Just static HTML with elegant styling and a few minor tweaks. For example, I was really surprised to see big name clients such as Disney, 3M, ARM & ESPN hidden a few clicks away (behind a button which wasn't very informative, from what I remember). Same for being in business for 20 years. A good UX/product person will tell you to put this front and center in your landing page, and rightfully so. @rsync: I love what you're doing, but please get a UX person involved :)

bacbilla 6y ago

+1 on having your laptop as an ephemeral device

erik_seaberg 6y ago
Yeah, assume it’s disposable not just for theft but because upgrading might be impossible and even repairs are very expensive (compared to a desktop).

richardfey 6y ago

I think they need to hire someone that is strong on the security side of the business, for two reasons: * he appears not aware of the role of hardware firewalls in mitigating DDoS by handling efficiently a lot of active TCP sessions (they have specialised hardware for this purpose) * he is describing in great detail a lot of information that a phisher or other type of hacker can treasure to target him

tpetry 6y ago
You cant protect from a DDoS with a hardware firewall, a DDoS consists of so much bandwidth that your network hardware is not able to simply handle the incoming traffic before any filtering happens. Your expensive hardware firewall can protect from DoS attacks, but they don‘t happen anymore as DDoS attacks are really cheap.
- e40 6y ago
  Can you can protect yourself from certain types of things (SYNC flood) with a firewall, though.
  - richardwhiuk 6y ago
    It's easier to protect against SYN floods if you terminate the connection.
- richardfey 6y ago
  In any situation where you need to create a filter rule that needs to run fast against a lot of TCP sessions a hardware firewall will do it faster than your general purpose server. It's not about protection but mitigation.
rfd4sgmk8u 6y ago
I read it this way too. Having the OS kernel have to process all internet junk traffic (including all the IoT worms) wastes CPU cycles that could be used serving traffic to legitimate users. It is better to offload the packet filtering to a dedicated device, with an OOB management. If the host was flooded, one might not be able to activate a host based firewall to fix in such a DDoS condition.

Crontab 6y ago

John's usage reminded me of something I read in Rob Rike's "Uses This" interview[1]: "I want no local storage anywhere near me other than maybe caches. No disks, no state, my world entirely in the network. Storage needs to be backed up and maintained, which should be someone else's problem, one I'm happy to pay to have them solve." [1]https://usesthis.com/interviews/rob.pike/ https://usesthis.com/interviews/rob.pike/

chris1993 6y ago
Essentially a Chromebook
- wwalexander 6y ago
  It’s worth reading the rest of the interview, I find Rob Pike has a very interesting/unique take on the current landscape given his involvement with Plan 9: > Now everything isn't connected, just connected to the cloud, which isn't the same thing. And uniform? Far from it, except in mediocrity. This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary.
  - iamevn 6y ago
    If you want to see more on this theme, the Upspin docs[1] are a really interesting read. This 2017 talk[2] that Rob Pike gave on it is also really good. [1] https://upspin.io/ https://upspin.io/ [2] https://youtu.be/ENLWEfi0Tkg https://youtu.be/ENLWEfi0Tkg
    - Crontab 6y ago
      Thank you! I hadn't seen this.
    - 4ad 6y ago
      To bad development has stopped.
- techrat 6y ago

RaitoBezarius 6y ago

You write down that you have no router, though your primary US location is connected to a "quintuple-homed network" and all global locations are at least triple-homed. What does that mean exactly? Is your IP provider quintuple-homed? Or are you running a bit more complicated setup than you explain but the gist is that you have no particular routing mechanisms? What does that say regarding your high availability? If one of your location is down, then it's definitely down until being fixed? Anyway, that was interesting, just curious about the fact of having no router at all. Thanks!

rsync 6y ago
The primary US location, in San Diego[1], gives us a managed, blended bandwidth product which is, in fact, quintuple homed and has been since we moved in (2001). So we have a dumb switch in our rack, but they have routers. In 2021 that's a weird bandwidth product and a weird setup but in 2001 it was "normal" and we just stay with that setup out of inertia (and the fact that we can't connect to he.net in San Diego). A similar setup exists for us in Zurich with init7. However, you are correct and we need to edit that FAQ language: our geo-redundant site in Fremont does not work that way. (I will note that it has been 11 years since we put that location in place (he.net in Fremont) and it has zero minutes of downtime) A tremendous amount of complexity and attack surface are eschewed by living with that setup and we're always looking for new ways to make that tradeoff. [1] Castle Acess datacenter on Aero drive. Is now a KIO managed datacenter.
- bombcar 6y ago
  Good old Castle Access - and they really seem to take the “Castle” part pretty seriously. We were there at a similar time - I probably saw the rsync.net servers.
walrus01 6y ago
I read it not as there are no routers anywhere, but that they've abstracted the problem of running the routers to their upstream hosting/colo/datacenter provider. Obviously there are routers and their systems are connected to somebody's ASN, or you wouldn't be able to reach them over the Internet.

pjs_ 6y ago

rsync.net rules

tfsh 6y ago

Meta: I really dislike the style of console.dev, the article is shunted to the left and leaves the rest of the screen real estate to be taken up by an - albeit pretty - but unnecessary piece of digital artwork. This - https://ibb.co/nzbFxjW https://ibb.co/nzbFxjW - is what the article looks like on my ultrawide which made for very uncomfortable viewing

chewbaxxa 6y ago
Not sure why you couldn’t just resize the window here?
- tfsh 6y ago
  I can, however I don't think having to resize your browser window to comfortably view an article is a very good UX, especially when it could be rectified by positioning the content in the middle of the screen.
  - Dylan16807 6y ago
    Right, but can you comfortably view anything over there? Why maximize the browser in the first place?
- ac29 6y ago
  Thats a bit of a chore with a tiling window manager if you generally have the browser on its own workspace - you either need to spawn new windows around the browser window to push it into the geometry you want, or add borders to it.
  - bigiain 6y ago
    It's not a great suggestion for, say, iPad users either...
    - Dylan16807 6y ago
      An iPad isn't nearly big enough to have this problem. This is what I get for iPad emulation: https://i.imgur.com/FriTf6X.png https://i.imgur.com/FriTf6X.png https://i.imgur.com/gLhqvFO.png https://i.imgur.com/gLhqvFO.png It looks just fine.

lokl 6y ago

I wish I had a personal use case where the pricing of rsync.net made sense. It looks like a great service. For now, I use Backblaze Unlimited. I realize they are not the same service, but Backblaze works for my personal stuff and the price is great.

tiernano 6y ago
I like backblaze, dont get me wrong, but my issue with them was their software is Windows Client and Mac OS only... No Linux or Windows Server offerings... My desktop runs either Windows Server 2019 or Linux... I havent run a desktop class verison of Windows on a phsyical workstation in years... As an aside, i use RSync and their Borg Backup option[1] for backing up my Linux box, and Windows is backed up to that Linux box too... works well... Borg can be gotten to work with B2[2], but its a bit more messing... [1]:https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html [2]:https://medium.com/@mormesher/building-your-own-linux-cloud-backup-system-75750f47d550 https://medium.com/@mormesher/building-your-own-linux-cloud-...
audience_mem 6y ago
> Backblaze Test your backups. https://messengergeek.wordpress.com/2018/03/09/backblaze-review-data-loss/ https://messengergeek.wordpress.com/2018/03/09/backblaze-rev...
- lokl 6y ago
  Thanks for sharing this.
  - audience_mem 6y ago
    No problem. I hate the thought of data loss. They may be better now, but who knows, it's worth being sure.
- bombcar 6y ago
  People think of backblaze as a backup/archive but it is explicitly not an archive - it is a backup and if you delete a file on your system it is purged within 30 days from the copy. Most people don’t really think about this and expect that any backup is AlSO an archive. You can get burned by this and have to use b2 or whatever it is instead.

poorman 6y ago

"I initiate my work in the terminal by port-knocking". Guess you don't need a firewall when you have no open ports? Haha yes! Guess I'm not the only one...

mfincham 6y ago

My first experience with rsync.net was very disappointing. To this day they still advertise “append-only mode” support for restic at https://www.rsync.net/products/restic.html https://www.rsync.net/products/restic.html. Their support people confirmed it doesn’t work (though they didn’t seem to understand why it would be fine for them to support it as advertised...) yet 6 months later they still advertise that they support it, even when I have e-mailed to remind them (and it still doesn’t work either) :(

mfincham 6y ago
The tl;dr as to why it doesn’t work is that they blanket forbid calling “rclone serve”, which is required for “append-only” support in restic. This doesn’t make sense given that the specific invocation of “rclone serve restic --stdio” doesn’t open any network sockets, it’s no less safe than e.g. “tar”

ynx 6y ago

The replacement for Spectacle, FYI, is Rectangle, which is almost identical, but still maintained

vzaliva 6y ago

Firefox Reade mode makes this page more readable. Otherwis 50% of your screen space is taken by non-informative graphics.

mattbillenstein 6y ago

So how is this sharded? Or how do you load balance a customer to the correct server if there's no router?

AdamJacobMuller 6y ago

I really appreciate the information and would love more information on your architecture in particular. Also definitely love your information on your personal process and flow. Some of it seems interesting to adopt and some of it seems bad for me but I could see why you do it, some I do in similar ways but a bit differently. e.g. I use git and online services to make sure that any of my computers are completely replaceable and that I can pick up work at any moment from any one. I can't agree more with the "no firewalls" approach to things, though I prefer to call it "host based" firewalls as it scares people less! I'm glad you've had no compliance/audit pushback on that, I architect things similarly and have had success pushing back on the requirement as well. I'm very surprised by the l2 switches and actually choosing to run completely unmanaged switches. I assume you're running all 10G or more? Maybe i'm overthinking the complexity of your network but I would be lost without snmp counters on my switches and running switches+networking in fully l3 mode has some great isolation benefits, especially if you want full switch-level redundancy. Do you have some more details on your data architecture? I'm very curious how do you do data direction/redundancy/sharding and balancing customer data across servers. I'm not trying to pry for things you consider secret but I think you have a very similar architectural mindset and I'm curious how you solve these things.

peppermint_tea 6y ago

happy customer here. I do a simple rsync of my precious but not too sensitive data, daily. and for the more sensitive stuff, gpg before sending daily as well, the copies will add up but I prefer it that way. 10/10 great business

crazypython 6y ago

7 daily snapshots: So I could sync my hard drive over, wait for a snapshot, delete everything, and keep the space, and use the snapshot as backup?

gautamcgoel 6y ago

The pricing model doesn't make sense to me. Their prices start at $0.025/GB/month, so renting 1TB of storage for a year would cost $300 - at that price, I could just buy my own disks and run ZFS myself. I kinda hoped they could offer lower prices using economies of scale. I checked the prices for Tarsnap, expecting it to be cheaper - it's actually 10x more expensive! Maybe someone can explain what I'm missing.

cat199 6y ago
DIY is way cheaper, true. but for comparison, aws is ~$100/tb/month
- generalizations 6y ago
  However, also for comparison, Backblaze is ~$5/TB/month.
- dividedbyzero 6y ago
  AWS Glacier is ~$4/tb/month. Getting data out of there costs extra, but for backups of last resort you don't expect to ever need, that may be a workable tradeoff.
  - bombcar 6y ago
    IIRC rsync.net doesn’t charge for bandwidth in any direction which for some use cases is nice - a set and forget billing type.
- thw0rted 6y ago
  For a backup use case, I haven't been able to beat the value of a Microsoft premium-whatever family plan. Routinely on sale for $60/yr, it gets you 1TB of backup plus an Office license for 6 users. Obviously you don't have as much control over it as "bare" cloud storage but it's hard to match the price.
anderiv 6y ago
Sure, you could purchase your own drives. But then it wouldn’t be offsite. And it probably wouldn’t be on a redundant internet connection. And most likely not have redundant power and cooling. And, and, and. Self-hosting at home (or in the office) is a great option for some if you’re not worried about needing an offsite backup. For those that do care about this sort of thing, though, the extra you pay to have someone else manage the thing is well worth it.

travisgriggs 6y ago

> I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit. I like that. I like that a lot. That's a very enviable practice. I think I know what I'll be experimenting with this next week. Did not expect to read that article and have the most stand out thing be a routine change I'd want to copy. You never know.

prox 6y ago
It’s one of the best “lifehacks” , walking, biking and the likes in a traffic free or low environment to just clear the mind. By the end of the walk you’ll be full idea’s, or just ready to start the day. It’s coffee but without stimulants :)
driverdan 6y ago
Get a dog and you won't have an excuse for skipping a morning walk.

zarkov99 6y ago

For people who use rsync.net, is this something that can replace Dropbox for multi-machine synching? For all its flaws, Dropbox does allows me a semi-seamless transition between my laptop and my workstation.

eythian 6y ago
As far as I know, no, unless you manage aspects of that yourself via git-annex or something similar. My setup to do this is that I run my own nextcloud server, which handles the computer and phone etc. syncing, then nightly that's backed up to a small computer in my house (I just use rsnapshot for that), which then backs itself up to rsync.net (using plain old rsync.)
tecleandor 6y ago
I think Syncthing could be better for those purposes. I use that for synching my homes (config files and essential work files, keys and so on) between my desktop and laptop. I have a copy running in my NAS to always have a copy available, one in my laptop, one in my desktop, and I was thinking about having one in my phone to run only when I'm charging (so I don't kill my battery).
- zarkov99 6y ago
  Can you compare Synching and Dropbox? I have been a (paying) Dropbox user for many years but the product is not that good, especially on Linux. I would love a more reliable alternative.
  - tecleandor 6y ago
    Syncthing is "just" a syncing daemon with a simple status web dashboard. The pros (at least for me): - You host/own it - It's not centralized: every node can sync with the others - Seems to be fast (the more nodes, the merrier) The cons: - Setup is slightly more difficult (you need to share some keys and ips) - No iOS client (not an issue for me) - You can't share a folder or file via web link So it's great for syncing folders between systems, but it doesn't substitute the job that something like Seafile would do (managing permissions, collaboration, web file sharing, fancy web ui...) There's a bit of info in the faq: https://docs.syncthing.net/users/faq.html https://docs.syncthing.net/users/faq.html

KingOfCoders 6y ago

Used them in a startup for a long time, was very happy, excellent support, good pricing. Would always use them again. (used the Swiss location).

shydwoo 6y ago

Why would i buy 1TB for $20 per month here instead of getting 6TB for $8/month from Microsoft?

kqr 6y ago
Technical excellence and out-of-this-world support. I say this as a very happy rsync.net customer for over 10 years. Though if the descriptions of the service on their web page does not make you salivate, perhaps it's not for you.
bombcar 6y ago
The limitations on OneDrive are considerable as it’s built on top of Sharepoint somehow. But rsync.net is a backup product and OneDrive is a large file sharing drive. Attempt to use that 6TB in backup situations and you may experience issues.

hannofcart 6y ago

I wish all SaaS services were like rsync - No nonsense description of what they do - Clear and simple pricing - Simplicity as a core feature Big fan. Look forward to using your services in the future.

limaoscarjuliet 6y ago

Question for rsync: You said: This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security. You seem to suggest the big firewalls do not bring any value to the table. I always thought they had more "intelligence" - dropping sessions based on some bad patterns, guarding against DDoS (to some extent), etc. Are you saying BSD is as good as these expensive boxes? Does it apply to SSH only or HTTP(s) and some other traffic as well?

nix23 6y ago
OpenSSH and the OS is pretty much the best place to harden your SSH connection, no need for a Firewall.

oilbagz 6y ago

I bought an rsync.net account a few years back when John made it known on HN, and have used it solidly as a backup for my .. wristwatch! I have a LILYGO that I coded up a time-tracking app, which basically creates an event log whenever I tap it, wherever I go - and when Internet is available, it squirts the log over to some text files that live on rsync.net .. Pretty neat to be able to do this without much of a desktop or mobile phone in the way, I have to say. I wonder if there are more opportunities for this kind of IoT service out there .. it sure was fun to get this working without REST ..

nix23 6y ago

Clean, simple architecture...a sysadmin's dream.

ElectricMind 6y ago

//I try to maintain an “Hedonic Fast” Monday through Wednesday so, on those days, I am only looking for truly actionable headlines and comment threads that are relevant to my businesse// This is smart move!

yyyk 6y ago

Question to rsync: Which HDs does rsync use? Is there a preferred brand? Which brands have been found to be most reliable?

simonebrunozzi 6y ago

I like the interview a lot, but oddly this one here is the part that I liked the most: > I start the day with a short walk outdoors. I don’t want the first thing my eyes see to be print, and I don’t want the first thing my body does to be sitting. So I walk a bit.