7 ms·

I wrote my own rsync [video]

118 points by zdw 4y ago

binkHN 4y ago
While I enjoyed some of this video, I felt it was, at least, 20 minutes too long. It covered too much of his other projects, and it's title, perhaps, should be charged to note that.
- deleted 4y ago
  [deleted]
- abdusco 4y ago
  Have to agree. His motivation was interesting but also could have been left out.
  - heleninboodler 4y ago
    In fairness, the title of the video is "Why I wrote my own rsync", not what's reported here.
- hwd 4y ago
  I have to say I have enjoyed every minute of the video. Just watching what Michael Stapelberg has achieved leaves me awestruck. With his subtle humor I find it really a joy to watch.
toolslive 4y ago
why not use unison as a starting point?
kistanod 4y ago
Casually having 25 GBps internet speed... Remind me why I live in the country that has the biggest tech companies in the world + most venture capital, yet my ISP is currently offering 1 GBps max :) (I think I know the answer, yet this is painful to think about)
- m463 4y ago
  the answer is regulatory capture.
  - mschuster91 4y ago
    Usually I'd agree with you, but Switzerland has the advantage that 2/3rds of its population live in extreme density for European standards (426 ppl/km², [1]). The denser the population, the more cost-efficient the buildout of any infrastructure is. Additionally, Switzerland is all but drowning in money (in fact, they have to fight to keep the Swiss Franc affordable [2]) because everyone and their dog who can afford it parks their dark money there. That influx of wealth is reflected in the services and wages that the citizens get. [1] https://www.avenir-suisse.ch/wie-dicht-ist-die-schweiz-besiedelt/ https://www.avenir-suisse.ch/wie-dicht-ist-die-schweiz-besie... [2] https://www.handelsblatt.com/finanzen/geldpolitik/schweizer-notenbank-snb-greift-zur-franken-schwaechung-tiefer-in-die-tasche/28161194.html https://www.handelsblatt.com/finanzen/geldpolitik/schweizer-...
    - kybernetikos 4y ago
      Although the density thing is true, the entire population of the country of Switzerland is less than that of a large metropolitan area (e.g. London) so while it might not be possible to have Swiss level infrastructure in rural US areas it should be perfectly possible in major cities.
      - vitus 4y ago
        Put more concretely: the population of NYC proper (ignoring the broader metro) and Switzerland are pretty comparable (both around 8-9m people), but NYC is about 50x more dense (300 mi² vs 16000).
        Brian_K_White 4y ago
        But the Swiss poulation is not evenly spread over it's land area.
deleted 4y ago
[deleted]
jamessb 4y ago
The slides: https://docs.google.com/presentation/d/1XqWF7h8ScimDFbQgLcXP4NcnUWRGEm59F0kGvw4aYa0/edit#slide=id.g123c4a18dff_0_68 https://docs.google.com/presentation/d/1XqWF7h8ScimDFbQgLcXP... The speaker Michael Stapelberg had a few blog posts about his home 25 Gbit/s fiber connection that have been discussed here previously "My upgrade to 25 Gbit/s Fiber To The Home": https://news.ycombinator.com/item?id=31134534 https://news.ycombinator.com/item?id=31134534 "25 Gigabit Linux internet router PC build": https://news.ycombinator.com/item?id=27792738 https://news.ycombinator.com/item?id=27792738
- e40 4y ago
  I read through the slides and I still don't know why he wrote his own rsync.
  - zamadatix 4y ago
    It's a bit easier to follow in the video since he talks from the slides more than presents them but as built up earlier in the deck (11-15) the router7 system us is written in pure Go, i.e. there is no other userspace, which comes with several neat advantages over a traditional system but he wanted to merge his routing box and server box as part of the upgrade needed to support 25G. Later (20-21) he talks about how all of the other software already had popular Go alternatives that could be easily dropped in except rsync and how there was a kludge way to get rsync on the box but it broke the advantages of doing the router7 Go only approach to do so and as a result he set out to implement enough of rsync to meet his needs. The journey and results are much more interesting than the why though. Router7 has some interesting diagnostic capabilities and workarounds specific to issues he's had from the ISP. Also the implementation of rsync itself is interesting in that it outperformed canonical rsync of the time doing the same task without "better performance" being a goal while writing it. There are limitations with this implementation though, it was only made to meet the particular use case not be a general rsync replacement and it doesn't have the latest protocol/hashing changes rsync has which allowed rsync to retake the performance lead since. The short sum up would be "because he had an open router userspace written in pure Go and rsync didn't have a suitable pure Go implementation at the time". No taking over the world or anything like that just good ol' hacking for personal use.
marci 4y ago
For those who want to avoid youtube: https://media.ccc.de/v/gpn20-41-why-i-wrote-my-own-rsync https://media.ccc.de/v/gpn20-41-why-i-wrote-my-own-rsync

118 points by zdw 4y ago

While I enjoyed some of this video, I felt it was, at least, 20 minutes too long. It covered too much of his other projects, and it's title, perhaps, should be charged to note that.

deleted 4y ago
[deleted]
abdusco 4y ago
Have to agree. His motivation was interesting but also could have been left out.
- heleninboodler 4y ago
  In fairness, the title of the video is "Why I wrote my own rsync", not what's reported here.
hwd 4y ago
I have to say I have enjoyed every minute of the video. Just watching what Michael Stapelberg has achieved leaves me awestruck. With his subtle humor I find it really a joy to watch.

toolslive 4y ago

why not use unison as a starting point?

kistanod 4y ago

Casually having 25 GBps internet speed... Remind me why I live in the country that has the biggest tech companies in the world + most venture capital, yet my ISP is currently offering 1 GBps max :) (I think I know the answer, yet this is painful to think about)

m463 4y ago
the answer is regulatory capture.
- mschuster91 4y ago
  Usually I'd agree with you, but Switzerland has the advantage that 2/3rds of its population live in extreme density for European standards (426 ppl/km², [1]). The denser the population, the more cost-efficient the buildout of any infrastructure is. Additionally, Switzerland is all but drowning in money (in fact, they have to fight to keep the Swiss Franc affordable [2]) because everyone and their dog who can afford it parks their dark money there. That influx of wealth is reflected in the services and wages that the citizens get. [1] https://www.avenir-suisse.ch/wie-dicht-ist-die-schweiz-besiedelt/ https://www.avenir-suisse.ch/wie-dicht-ist-die-schweiz-besie... [2] https://www.handelsblatt.com/finanzen/geldpolitik/schweizer-notenbank-snb-greift-zur-franken-schwaechung-tiefer-in-die-tasche/28161194.html https://www.handelsblatt.com/finanzen/geldpolitik/schweizer-...
  - kybernetikos 4y ago
    Although the density thing is true, the entire population of the country of Switzerland is less than that of a large metropolitan area (e.g. London) so while it might not be possible to have Swiss level infrastructure in rural US areas it should be perfectly possible in major cities.
    - vitus 4y ago
      Put more concretely: the population of NYC proper (ignoring the broader metro) and Switzerland are pretty comparable (both around 8-9m people), but NYC is about 50x more dense (300 mi² vs 16000).
      - Brian_K_White 4y ago
        But the Swiss poulation is not evenly spread over it's land area.

deleted 4y ago

[deleted]

jamessb 4y ago

The slides: https://docs.google.com/presentation/d/1XqWF7h8ScimDFbQgLcXP4NcnUWRGEm59F0kGvw4aYa0/edit#slide=id.g123c4a18dff_0_68 https://docs.google.com/presentation/d/1XqWF7h8ScimDFbQgLcXP... The speaker Michael Stapelberg had a few blog posts about his home 25 Gbit/s fiber connection that have been discussed here previously "My upgrade to 25 Gbit/s Fiber To The Home": https://news.ycombinator.com/item?id=31134534 https://news.ycombinator.com/item?id=31134534 "25 Gigabit Linux internet router PC build": https://news.ycombinator.com/item?id=27792738 https://news.ycombinator.com/item?id=27792738

e40 4y ago
I read through the slides and I still don't know why he wrote his own rsync.
- zamadatix 4y ago
  It's a bit easier to follow in the video since he talks from the slides more than presents them but as built up earlier in the deck (11-15) the router7 system us is written in pure Go, i.e. there is no other userspace, which comes with several neat advantages over a traditional system but he wanted to merge his routing box and server box as part of the upgrade needed to support 25G. Later (20-21) he talks about how all of the other software already had popular Go alternatives that could be easily dropped in except rsync and how there was a kludge way to get rsync on the box but it broke the advantages of doing the router7 Go only approach to do so and as a result he set out to implement enough of rsync to meet his needs. The journey and results are much more interesting than the why though. Router7 has some interesting diagnostic capabilities and workarounds specific to issues he's had from the ISP. Also the implementation of rsync itself is interesting in that it outperformed canonical rsync of the time doing the same task without "better performance" being a goal while writing it. There are limitations with this implementation though, it was only made to meet the particular use case not be a general rsync replacement and it doesn't have the latest protocol/hashing changes rsync has which allowed rsync to retake the performance lead since. The short sum up would be "because he had an open router userspace written in pure Go and rsync didn't have a suitable pure Go implementation at the time". No taking over the world or anything like that just good ol' hacking for personal use.

marci 4y ago

For those who want to avoid youtube: https://media.ccc.de/v/gpn20-41-why-i-wrote-my-own-rsync https://media.ccc.de/v/gpn20-41-why-i-wrote-my-own-rsync