5 ms·

I've self-hosted the original Firefox syncserver[0] for a couple of years now. It's a now unmaintained Python 2 + gunicorn web app. It still uses the Firefox i

I've self-hosted the original Firefox syncserver[0] for a couple of years now. It's a now unmaintained Python 2 + gunicorn web app. It still uses the Firefox identities service, since I haven't bothered to try setting that one up myself. That means I need a Firefox/Mozilla account even with a self-hosted sync server, which isn't ideal. Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted. I originally went through the setup instructions manually. Now I've created a Nix package[2] and NixOS module for it[3], which was an adventure by itself. I can now easily reproduce the setup, which I did while migrating my webserver to a different provider. As for why I didn't try syncstorage-rs, it's rather petty, but the old syncserver was working well for me, and it doesn't yet support SQLite. [0] https://github.com/mozilla-services/syncserver https://github.com/mozilla-services/syncserver [1] https://github.com/mozilla-services/syncserver#removing-mozilla-hosted-data https://github.com/mozilla-services/syncserver#removing-mozi... [2] https://git.sr.ht/~williamvds/config/tree/master/item/nix/pkgs/firefox-syncserver https://git.sr.ht/~williamvds/config/tree/master/item/nix/pk... [3] https://git.sr.ht/~williamvds/config/tree/master/item/nix/modules/firefox-syncserver.nix https://git.sr.ht/~williamvds/config/tree/master/item/nix/mo...

wkat4242 4y ago
The first few lines in the repo you mention illustrate the problem: > Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered. And its replacement syncserver-rs is in this forever-unfinished state so they've effectively killed self-hosting. At least from a support standpoint.
- williamvds 4y ago
  Good point, I'll update the comment to indicate syncserver is now unmaintained. Of course, everything self-hosted is at my own risk etc. And official support is a luxury in this kind of FOSS, I generally don't count on it. But since it works for me, it may be worth sharing the details.
  - wkat4242 4y ago
    I agree that a lack of support isn't a huge issue for this kind of thing. I'm more worried about the maintenance, like a vulnerability remaining unpatched (it being python 2 doesn't help either) or a protocol change that makes the old server suddenly unviable. I really wish they'd just finish the Rust version. That would solve everything.
- MawKKe 4y ago
  Sounds like classic case of ”Second System Effect”
  - ninju 4y ago
    Better known as "if its not broken, break it" https://en.wikipedia.org/wiki/Second-system_effect https://en.wikipedia.org/wiki/Second-system_effect
- toyg 4y ago
  I don't understand. The original was barely 500 lines of code in two files, everything else being packaging machinery. Yes it's python 2, but how hard could have it been to port 500 LOCs to 3.x...? Even assuming some crypto libraries broke, surely there are working equivalents? Instead, they went for a Big Rewrite in Rust, resulting in a much bigger codebase that will be much harder to maintain... and broke it anyway.
  - acdha 4y ago
    Their issue tracker suggests that it’s mostly due to their team being focused on the main Firefox service: https://github.com/mozilla-services/syncserver/issues/264#issuecomment-754784657 https://github.com/mozilla-services/syncserver/issues/264#is... I’m sure it wouldn’t be that hard to port but if they’re committed to a different direction it’s probably best not to give the impression that you should plan to keep using it.
    - toyg 4y ago
      Yeah, but even internally it makes little sense. You've replaced a very simple system built on a widely-known language with a massive ecosystem, something you could even call "boring technology" these days (python), with a larger implementation using a niche language with a small ecosystem and a minuscule pool of skilled developers. It's now even more unlikely you'll ever get any patches from outsiders, it will be harder to hire people to maintain it, harder to monitor and expand the system... And this for a critical system. I mean, as a job-security device, I'm sure it works out great. As a strategic business decision, though, it looks debatable as a minimum - and I'm being charitable.
      - acdha 4y ago
        Not sure you could call Rust a niche language any more given how many devices it’s running on, but especially at the organization which created it that seems like an understandable choice. I have a lot of experience with Python but can totally sympathize with people running a highly-visible service prioritizing resource utilization.
kevincox 4y ago
Have you considered upstreaming the package and module to nixpkgs? It would be awesome to have those one line of config away for all nixpkgs users.
- williamvds 4y ago
  I briefly considered it, but there are some big problems: - The new replacement package, syncserver-rs, has already been added to Nixpkgs, along with a NixOS module [0] - Python 2 and the package itself are no longer supported, meaning... - support in Nixpkgs is pretty rough - dependencies have been updated such that they no longer support Python 2, or have been removed outright, meaning... - syncserver has already been removed from Nixpkgs - in order to avoid the problem of unsupported dependencies, I delegated dependency resolution to a community project, poetry2nix[1]. Even so, it was a nightmare to get working, lots of hacks and workarounds were needed Of course, my repo is public, so anyone is welcome to use the Flake within. If they do, I'd suggest dropping me a line so I don't do anything drastic to the package or module. [0] https://github.com/NixOS/nixpkgs/pull/176835 https://github.com/NixOS/nixpkgs/pull/176835 [1] https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
  - kevincox 4y ago
    Yeah, Python2 alone is enough to not bother. Hopefully the new version "ships" soon.
pmarreck 4y ago
I think Nix, as struggle-full as it was, was the right choice, here, although I don't see the flake.nix file
- williamvds 4y ago
  The flake starts at this level: https://git.sr.ht/~williamvds/config/tree/master/item/nix https://git.sr.ht/~williamvds/config/tree/master/item/nix Struggle is a good way to put it! I feel I hit one of Nix's pain points here: packaging software whose dependencies are largely missing from Nixpkgs. On top of that, it was a Python app, and the state of Python packaging is quite a mess. I also feel it Nix was the right choice. My journey was rather harsh, but I do believe in its approach.
  - pmarreck 4y ago
    The thing that guts me a little is that Nix is also perfect for software preservation, which unfortunately must include Python 2 apps. But meanwhile they've dropped support for Python 2, which means you either have to go with some clever flake solution or something like what you did. I'm dealing with an app that is breaking every time I try to upgrade a few-years-old version of Node, a version that is no longer on nixpkgs. Right now I just want something that works; I didn't sufficiently budget for ongoing maintenance as the whole site's behind a login.
    - williamvds 4y ago
      Another option is to find a Nixpkgs commit which does contain the version of a package that you want. You can pin and import it as a Flake input, and use the old version of the package provided. So in its own way, Nix and Nixpkgs do serve the purpose of software preservation, as long as you have the full commit history for Nixpkgs. Preservation doesn't imply maintenance however. It's fair for Nixpkgs to delete packages that are no longer supported, as that's a burden on the limited number of maintainers. By recreating the package and module in my own Flake, I've taken on responsibility for maintaining it myself, and not adding any work to the Nixpkgs maintainers. NixOS modules in particular aren't easy to extend, so I ended up copying it entirely. And to be honest, the thought of using an old version of Nixpkgs didn't cross my mind while setting up syncserver. Hopefully it's easier to maintain with poetry2nix now that I've got the package working.
pieter_mj 4y ago
>Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted. Note : You'll need to disable 2fa first on your account before you can delete the data.

wkat4242 4y ago

The first few lines in the repo you mention illustrate the problem: > Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered. And its replacement syncserver-rs is in this forever-unfinished state so they've effectively killed self-hosting. At least from a support standpoint.

williamvds 4y ago
Good point, I'll update the comment to indicate syncserver is now unmaintained. Of course, everything self-hosted is at my own risk etc. And official support is a luxury in this kind of FOSS, I generally don't count on it. But since it works for me, it may be worth sharing the details.
- wkat4242 4y ago
  I agree that a lack of support isn't a huge issue for this kind of thing. I'm more worried about the maintenance, like a vulnerability remaining unpatched (it being python 2 doesn't help either) or a protocol change that makes the old server suddenly unviable. I really wish they'd just finish the Rust version. That would solve everything.
MawKKe 4y ago
Sounds like classic case of ”Second System Effect”
- ninju 4y ago
  Better known as "if its not broken, break it" https://en.wikipedia.org/wiki/Second-system_effect https://en.wikipedia.org/wiki/Second-system_effect
toyg 4y ago
I don't understand. The original was barely 500 lines of code in two files, everything else being packaging machinery. Yes it's python 2, but how hard could have it been to port 500 LOCs to 3.x...? Even assuming some crypto libraries broke, surely there are working equivalents? Instead, they went for a Big Rewrite in Rust, resulting in a much bigger codebase that will be much harder to maintain... and broke it anyway.
- acdha 4y ago
  Their issue tracker suggests that it’s mostly due to their team being focused on the main Firefox service: https://github.com/mozilla-services/syncserver/issues/264#issuecomment-754784657 https://github.com/mozilla-services/syncserver/issues/264#is... I’m sure it wouldn’t be that hard to port but if they’re committed to a different direction it’s probably best not to give the impression that you should plan to keep using it.
  - toyg 4y ago
    Yeah, but even internally it makes little sense. You've replaced a very simple system built on a widely-known language with a massive ecosystem, something you could even call "boring technology" these days (python), with a larger implementation using a niche language with a small ecosystem and a minuscule pool of skilled developers. It's now even more unlikely you'll ever get any patches from outsiders, it will be harder to hire people to maintain it, harder to monitor and expand the system... And this for a critical system. I mean, as a job-security device, I'm sure it works out great. As a strategic business decision, though, it looks debatable as a minimum - and I'm being charitable.
    - acdha 4y ago
      Not sure you could call Rust a niche language any more given how many devices it’s running on, but especially at the organization which created it that seems like an understandable choice. I have a lot of experience with Python but can totally sympathize with people running a highly-visible service prioritizing resource utilization.

kevincox 4y ago

Have you considered upstreaming the package and module to nixpkgs? It would be awesome to have those one line of config away for all nixpkgs users.

williamvds 4y ago
I briefly considered it, but there are some big problems: - The new replacement package, syncserver-rs, has already been added to Nixpkgs, along with a NixOS module [0] - Python 2 and the package itself are no longer supported, meaning... - support in Nixpkgs is pretty rough - dependencies have been updated such that they no longer support Python 2, or have been removed outright, meaning... - syncserver has already been removed from Nixpkgs - in order to avoid the problem of unsupported dependencies, I delegated dependency resolution to a community project, poetry2nix[1]. Even so, it was a nightmare to get working, lots of hacks and workarounds were needed Of course, my repo is public, so anyone is welcome to use the Flake within. If they do, I'd suggest dropping me a line so I don't do anything drastic to the package or module. [0] https://github.com/NixOS/nixpkgs/pull/176835 https://github.com/NixOS/nixpkgs/pull/176835 [1] https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
- kevincox 4y ago
  Yeah, Python2 alone is enough to not bother. Hopefully the new version "ships" soon.

pmarreck 4y ago

I think Nix, as struggle-full as it was, was the right choice, here, although I don't see the flake.nix file

williamvds 4y ago
The flake starts at this level: https://git.sr.ht/~williamvds/config/tree/master/item/nix https://git.sr.ht/~williamvds/config/tree/master/item/nix Struggle is a good way to put it! I feel I hit one of Nix's pain points here: packaging software whose dependencies are largely missing from Nixpkgs. On top of that, it was a Python app, and the state of Python packaging is quite a mess. I also feel it Nix was the right choice. My journey was rather harsh, but I do believe in its approach.
- pmarreck 4y ago
  The thing that guts me a little is that Nix is also perfect for software preservation, which unfortunately must include Python 2 apps. But meanwhile they've dropped support for Python 2, which means you either have to go with some clever flake solution or something like what you did. I'm dealing with an app that is breaking every time I try to upgrade a few-years-old version of Node, a version that is no longer on nixpkgs. Right now I just want something that works; I didn't sufficiently budget for ongoing maintenance as the whole site's behind a login.
  - williamvds 4y ago
    Another option is to find a Nixpkgs commit which does contain the version of a package that you want. You can pin and import it as a Flake input, and use the old version of the package provided. So in its own way, Nix and Nixpkgs do serve the purpose of software preservation, as long as you have the full commit history for Nixpkgs. Preservation doesn't imply maintenance however. It's fair for Nixpkgs to delete packages that are no longer supported, as that's a burden on the limited number of maintainers. By recreating the package and module in my own Flake, I've taken on responsibility for maintaining it myself, and not adding any work to the Nixpkgs maintainers. NixOS modules in particular aren't easy to extend, so I ended up copying it entirely. And to be honest, the thought of using an old version of Nixpkgs didn't cross my mind while setting up syncserver. Hopefully it's easier to maintain with poetry2nix now that I've got the package working.

pieter_mj 4y ago

>Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted. Note : You'll need to disable 2fa first on your account before you can delete the data.