8 ms·

Mutagen – Cloud-based development using your local tools

174 points by vmoore 5y ago

no_wizard 5y ago
I'd love to hear more about anyone who has managed to use mutagen as a docker-compose replacement. I have so far been a bit let down by mutagen in that it doesn't really seem to be living up the promises of really being easier than using docker directly. Its file synchronization concepts also don't always seem to keep up with file watchers and hot reload I've been looking at something like Lando[0] as a result, and I realize they're not the same thing, it seems like it is more fitting with the role for the development environment, at least. [0]: https://docs.lando.dev/getting-started/ https://docs.lando.dev/getting-started/
- bennyp101 5y ago
  OT but, I switched away from Lando over to plain docker-compose files, It was much easier to configure things. For example, trying to get HMR working in Lando was just a pain. Lando is great for getting up and running quickly,outside of the recipes, it's easier to just create your own docker-compose file.
- xenoscopic 5y ago
  Hey, sorry to hear that Mutagen didn't pan out for you. I'd be curious to hear if you've tried the Mutagen Compose[0] project or some other mechanism for Compose integration. I'm still iterating on the best approach for integrating Mutagen with various Docker workflows, so I'm always looking for feedback and experience reports. [0]: https://mutagen.io/documentation/orchestration/compose https://mutagen.io/documentation/orchestration/compose
  - no_wizard 5y ago
    I'll take a second look at this to see if maybe we could optimize some bits in how we are using mutagen currently. Is there any thought - given this is a development tool - to exposing container aliases as commands? Kinda like with lando, where you can `lando composer install` or `lando yarn turbo` or something like that. One of the other issues I have with these container setups in general is that for anything that has a heavy CLI component or interactivity there's no good abstraction around interacting with the CLI.
    - xenoscopic 5y ago
      Feel free to reach out on Slack or GitHub issues or by email if there's anything I can help with. I think framework-specific stuff like that is better handled by higher-level tooling, typically something that's just embedding Mutagen. DDEV, for example, adds that type of functionality for PHP-based projects, while embedding Mutagen underneath. The goal with Mutagen itself is (ideally) to be invisible - only providing synchronization and forwarding. Even Mutagen Compose (being just tweaked version of Docker Compose) probably falls too low in the stack to know anything about the specific frameworks being used atop it. That being said, I think this is something that Compose wants to address, and that Mutagen Compose could inherit once it's added to the Compose Spec and implemented into Docker Compose. Things like profile specifications in Compose files are designed to encapsulate these types of operations (among other things).
kyriakos 5y ago
using mutagen as a workaround for the Windows WSL I/O slowness. great and rock solid tool.
- dboreham 5y ago
  Another workaround is to use WSL2.
  - pineconewarrior 5y ago
    You can still get improvements in some scenarios with mutagen on WSL2
  - kyriakos 5y ago
    I am using WSL2 - issue exists on both versions.
- bliteben 5y ago
  its not that bad if you use the native filesystem, although then you have to actually get your files into the native filesystem
  - kyriakos 5y ago
    Using IDE in Windows and execution in wsl2. So IDE has files on Windows file system and use mutagen to keep them in sync with wsl2 file system via ssh. It's actually instant, only downside is duplication but thats a small price to pay.
xenoscopic 5y ago
Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project. [0]: https://github.com/mutagen-io/mutagen-compose https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose https://mutagen.io/documentation/orchestration/compose
- slightwinder 5y ago
  How robust is the handling? Can it tolerate unreliable network? Lost connections and hiccups?
  - xenoscopic 5y ago
    It can tolerate disconnection very reliably, and it will automatically reconnect to synchronization and forwarding endpoints as soon as possible. Its synchronization algorithm in particular (which is essentially a (repeated) three-way merge with rsync-style differential file transfers) is designed for safety and robust tracking of the changes that it has propagated. It's also capable of resuming file transfers once it reconnects.
- lifty 5y ago
  Thank you for this great project! I’ve been using it for a while now and it has been rock solid and easy to use. Cheers!
- vorpalhex 5y ago
  Thank you for building this - it appears to solve a real problem that has been a pain in my butt for a couple of years. What are the plans for funding the project? It looks like everything is OSS now, will there be any closed source elements in the future?
  - xenoscopic 5y ago
    To be totally transparent: funding is a bit of an open question, although there's no risk of Mutagen disappearing in the short term. Mutagen was actually part of YC's S19 batch, and it still has a significant amount of runway from that, but also some revenue from contracting work. I really want to keep as much of Mutagen as FOSS as possible, ideally MIT licensed. I did just add a small portion of code under the SSPL, which I might experiment with dual licensing to SaaS embedders of Mutagen (since it's really only useful in cases where Mutagen is being embedded in other tooling), but even that I wanted to keep open source for other FOSS projects embedding Mutagen. In the near term, I have some ideas about plugins and tooling that I want to build on top of Mutagen that will probably be closed source, but those will be separate entities from Mutagen itself and the aim will be to avoid compromising on any functionality that belongs in the core of Mutagen.
xjia 5y ago
Does the compiler run on my workstation or in the cloud? I read through the docs but didn't find anything about this.
- xenoscopic 5y ago
  There's no compiler here - it's a tool for bidirectionally synchronizing files (specifically code, assets, build products, etc.) and forwarding network traffic to and from remote systems. The idea behind Mutagen is to use your local tools (e.g. editor, IDE, browser, etc.) to work on remote hardware (where "remote" can mean anything you like, e.g. your local system, a Docker container (local or remote), a Raspberry Pi, a cloud server, etc. - or some combination of all of those). The architecture is designed to be maximally flexible, so you can work between your local system and a remote system, between two remote systems, or in any other topology you want.
  - dangom 5y ago
    It sounds like sshfs on steroids?
    - qxxx 5y ago
      with mutagen you have a 1:1 copy on your local machine and remote machine. If the connection drops, you have still all your files on eg your laptop, with sshfs your files are "gone" locally. If your connection is back again, mutagen will start the sync and upload/download files again (only changed) so there is again a 1:1 copy.
- qbasic_forever 5y ago
  In the kind of setup you'd use with mutagen your dev tools and compilers are typically inside the container, while your code and editor are outside the container and being synced inside the container on every change. If the container is running on another machine/cloud service then yeah it could be that your compiler is running outside your machine. But much more likely it's still running as a container on your machine. The typical use case for this is making your dev environment accessible to anyone, just send them a dockerfile and they can almost perfectly reproduce it and get going with building your software (vs. having to laboriously setup the exact version of each dev tool and compiler you use).
gr8tnwz 5y ago
I’m sure this feels like an accomplishment but in practice it provides nothing over git/source control as usual, and any number of reliable open or free hosted file sharing options. What specifically is missing in the current ecosystem of electron state moving technologies that this solves? Why this instead of git and a Wireguard based mesh of devices (to offer my current setup as an example; there are others).
- pineconewarrior 5y ago
  Speed. This works so fast that it improves the efficiency of local development with containers.
  - ab00tth4t 5y ago
- xenoscopic 5y ago
  The goal with Mutagen is real-time remote development, not source control or complex networking. It's designed to facilitate making changes to code locally and then having those immediately propagated to a remote system for compilation, execution, etc. Basically, it's designed for the work you do between Git commits, saving you the need to do a full commit, push, pull, evaluate cycle to test your code on a remote system. Basically, it's like VS Code's remote development extensions, but editor-agnostic and more flexible.
karlshea 5y ago
I've been testing Mutagen's file sync through DDEV on a Drupal project and it's pretty nice! The project is way more snappy than using NFS mounts on the Mac (which was already WAY faster than bind mounts), and DDEV has file ignores set up correctly so the project's media files aren't synced and disk usage isn't doubled.
- pineconewarrior 5y ago
  Same! Mutagen + DDEV have been a godsend for me
- xenoscopic 5y ago
  Yeah, Randy's done a great job with the integration - it seems to work seamlessly for most users as far as I can tell. I'm really excited to get Mutagen v0.14 and beyond into DDEV, because support for fanotify[0] is being added and it's going to make Mutagen's Linux filesystem watching unbelievably faster for container-based development. [0]: https://man7.org/linux/man-pages/man7/fanotify.7.html https://man7.org/linux/man-pages/man7/fanotify.7.html
  - TrickardRixx 5y ago
    I'm switching from using docker-compose + docker-sync[0] to using mutagen-compose and am recommending my coworkers across Windows, Intel Mac, and Apple Silicon Mac all do the same. For me mutagen-compose is solving the same performance file mount performance problems I was using docker-sync to solve, but mutagen is doing it much more reliably. I would have to restart the docker-sync (and sometimes the docker daemon) multiple times a day. So far mutagen "just works". [0]: https://github.com/EugenMayer/docker-sync https://github.com/EugenMayer/docker-sync
    - xenoscopic 5y ago
      Glad to hear that it works for you. Don't hesitate to reach out on the Mutagen issue tracker or Slack channel if you run into any issues or questions - I'm always happy to help.
- 5y ago
incanus77 5y ago
I’ve been using Mutagen with Docker for about 3-4 months to work around an (at the time) performance problem with bound volumes. I understand the new experimental FS option in Docker might solve this, but I haven’t tested or migrated my setup yet. Basically, I’m working on a bare metal kernel in C/C++ on ARM, which only has tooling on Windows or Linux. As I work on a Mac system, I setup a Docker container that makes this trivial for me, including GDB debugging of QEMU on the host machine. Mutagen was easy to get going, well documented, and has worked perfectly for this. I’m able to make my edits in my Mac IDE, then literally by the time I’ve switched to compilation, the edits are synced to the container, and vice-versa for the build product. Prior to Mutagen, I was getting 8-10x longer build times since Docker file synchronization was slowing things down. Mutagen syncs slower than Docker, but faster than I can work, so all is well.
eysi 5y ago
We use Mutagen for Garden's hot reloading mechanism. (Garden is a dev tool for K8s and hot reloading enables users to sync changes directly to a prod like dev environment as opposed to doing a rebuild and re-deploy). It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms. If anyone's interested in how we use it, it's here: https://github.com/garden-io/garden/blob/master/core/src/plugins/kubernetes/mutagen.ts https://github.com/garden-io/garden/blob/master/core/src/plu...
- gavinray 5y ago
  This is a MASSIVE endorsement as far as I am concerned I used Garden in the rsync days and it was already pretty good. The code behind Garden is some of the best TypeScript + Node I've ever seen, so if this was better and more performant than what you already had I'm sold. Tell Jon I said hi =)
  - eysi 5y ago
    Thank you say much for the kind words Gavin :)
InsaneOstrich 5y ago
I've been using Mutagen for years to offload running docker containers during local development from my Mac to a Linux machine and it works great. I'm glad to see more people talking about it.
stephen 5y ago
Huh! This sounds a lot like my hobby project to real-time sync files between my laptop & desktop: https://github.com/stephenh/mirror https://github.com/stephenh/mirror But dressed up to work with docker + "just" :-) being a professional/polished product, which is impressive. Neat!
mutagen 5y ago
Great name! waves (not associated)
inportb 5y ago
Looks like it handles two-way sync very well, but could it keep more than two participants in sync? I'm sort of "abusing" Syncthing to sync Docker volumes (mostly just configuration and helper scripts) across several hosts. It works well, and the only things I'm missing are file ownership preservation and low-latency "instant"-feeling sync. csync2 gets it right, but it's fragile and needs a lot of babysitting. I'm wondering if Mutagen might be a suitable alternative.
- xenoscopic 5y ago
  One way you can do this with Mutagen is to use a hub-and-spoke topology, where one copy of the files (say the one on the laptop where you're coding) is the "hub," and then you have multiple two-way (or one-way) synchronization sessions going out to the various "spoke" endpoints. In the two-way case, changes on a spoke endpoint will be propagated out to the other endpoints by first propagating to the hub. Obviously, you can have contention in this case if you have ultra-high-frequency updates to the same sets of of files, though the conflicting operations would have to occur on a smaller timescale than that of a single Mutagen synchronization cycle (which is typically somewhere around 100ms for an average codebase). Also, in the case of contention, Mutagen will simply display the conflicting files to you and you can delete the "losing" copy. You can also set auto-resolution behavior with Mutagen's `two-way-resolved` mode to let the hub win in conflicts, or use `one-way-replica` mode to make all the spoke endpoints replicas of the hub, or some combination of all these things, etc.
erulabs 5y ago
Mutagen is great (I really wish Skaffold implemented it for it's sync) and I want to point out that Jacob, the author is the project, is a really awesome and intelligent person - few people in our YC batch deserve the front-page more!
galaxyLogic 5y ago
Just curious, when you develop for the Cloud where to you debug your code? In the cloud or on your work-station?
- faizshah 5y ago
  For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM. For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment. I use a file synchronization tool like mutagen at work (for ninjas) and I use mutagen on my personal laptop for the same workflow. I’m thinking of trying remote development from jetbrains soon though: https://www.jetbrains.com/remote-development/ https://www.jetbrains.com/remote-development/ I generally prefer using a remote vm for development and only using my laptop for git. Also you can use spot VMs for development so it can be cheap to get a high CPU vm for building and debugging or a high mem VM for running data science scripts and debugging them.
  - memco 5y ago
    VSCode makes it very easy to selectively install plugins on a remote that doesn’t have internet access and allows fairly seamless editing, debugging and terminal access to a remote with very little effort. But I’ve been looking for alternatives to VSCode’s remote editing experience because I’ve generally got about 6-8 projects that I keep open all the time and it can eat up a lot of RAM and CPU especially when I have several of them running and debugging stuff in tandem. The trick for me is that beyond file editing synchronization I need to be able to debug the code. There’s some promising ideas here, but I haven’t been able to make the jump yet. The jet brains link you shared is the most similar offering I’ve seen, but I’m hopeful distant.nvim or mutagen (plus whatever else I’d need) can be used to get more tools on a similar level of seamlessness.
    - galaxyLogic 5y ago
      I'm asking because I'm contemplating on doing some work on AWS. If code runs on AWS doesn't it mean it makes lots of calls to the different AWS APIs. And then in which environment do you debug the code that makes those API-calls? Is it even possible to debug it locally?
qxxx 5y ago
I have been using mutagen for few years. It has been a life saver. I use it mostly to work on remote projects, mostly for development, rarely working on prod. So there I have my projects on some cloud linux instances and I work locally on windows, mostly phpstorm. Having the webserver in the cloud and doing the heavy lifting is quite nice, especially if you work with phpstorm, which can be a resource monster for bigger projects, and working on remote projects with phpstorm built in sftp / file sync is a huge pain in the ass, really slow. With mutagen it feels almost like I work locally. Thanks to the developer for this cool tool!
zmmmmm 5y ago
Looks like a really handy tool and essentially the logical endpoint of all the various hacky manual file watcher / scripted copy type sync things we end up doing to work across environments if they were developed into a full solution. We work quite hard to maintain unified production/dev config for our docker-compose setups. Mutagen looks like it could simplify some of these since a whole layer of complexity comes from trying to optionally map ports and volumes in such a way that it works for development, while not being inappropriate in test/production. If instead, mutagen can virtually connect / synchronize these it'll be a lot simpler. I do wonder a bit how debugging works .... if something is running in a container, it's great I can hot sync my code there, but what happens when I put a breakpoint in the code in my IDE?
kamikazeturtles 5y ago
Would it be possible to integrate this with a browser editor (like codemirror) to create something like replit?
osener 5y ago
Mutagen and mutagen-compose + an EC2 instance work great as a replacement for Docker for Mac. This setup you to develop and run Intel images without performance penalty. Both on M1 and on Intel I’m never going back to Docker for Mac again if I help it.
- jeffnappi 5y ago
  An important thing to note is that the _worst_ part about Docker for Mac is the lousy disk IO performance for shared drives. This can be almost fully mitigated by using NFS mounts instead[0]. In general Docker for Mac is far from ideal - particularly considering you must allocate a set block of memory to the VM. If only it was the year of the Linux Desktop :grin: [0] https://gist.github.com/vschoener/b0b5ae08625744a1f4ad4140275d9675 https://gist.github.com/vschoener/b0b5ae08625744a1f4ad414027...
xwowsersx 5y ago
This looks interesting, but I'm not sure I correctly understand the use case or how exactly this might fit into my development workflow. Can someone enlighten me and elaborate on what this solves and what its use cases are? Thanks.

174 points by vmoore 5y ago

I'd love to hear more about anyone who has managed to use mutagen as a docker-compose replacement. I have so far been a bit let down by mutagen in that it doesn't really seem to be living up the promises of really being easier than using docker directly. Its file synchronization concepts also don't always seem to keep up with file watchers and hot reload I've been looking at something like Lando[0] as a result, and I realize they're not the same thing, it seems like it is more fitting with the role for the development environment, at least. [0]: https://docs.lando.dev/getting-started/ https://docs.lando.dev/getting-started/

bennyp101 5y ago
OT but, I switched away from Lando over to plain docker-compose files, It was much easier to configure things. For example, trying to get HMR working in Lando was just a pain. Lando is great for getting up and running quickly,outside of the recipes, it's easier to just create your own docker-compose file.
xenoscopic 5y ago
Hey, sorry to hear that Mutagen didn't pan out for you. I'd be curious to hear if you've tried the Mutagen Compose[0] project or some other mechanism for Compose integration. I'm still iterating on the best approach for integrating Mutagen with various Docker workflows, so I'm always looking for feedback and experience reports. [0]: https://mutagen.io/documentation/orchestration/compose https://mutagen.io/documentation/orchestration/compose
- no_wizard 5y ago
  I'll take a second look at this to see if maybe we could optimize some bits in how we are using mutagen currently. Is there any thought - given this is a development tool - to exposing container aliases as commands? Kinda like with lando, where you can `lando composer install` or `lando yarn turbo` or something like that. One of the other issues I have with these container setups in general is that for anything that has a heavy CLI component or interactivity there's no good abstraction around interacting with the CLI.
  - xenoscopic 5y ago
    Feel free to reach out on Slack or GitHub issues or by email if there's anything I can help with. I think framework-specific stuff like that is better handled by higher-level tooling, typically something that's just embedding Mutagen. DDEV, for example, adds that type of functionality for PHP-based projects, while embedding Mutagen underneath. The goal with Mutagen itself is (ideally) to be invisible - only providing synchronization and forwarding. Even Mutagen Compose (being just tweaked version of Docker Compose) probably falls too low in the stack to know anything about the specific frameworks being used atop it. That being said, I think this is something that Compose wants to address, and that Mutagen Compose could inherit once it's added to the Compose Spec and implemented into Docker Compose. Things like profile specifications in Compose files are designed to encapsulate these types of operations (among other things).

kyriakos 5y ago

using mutagen as a workaround for the Windows WSL I/O slowness. great and rock solid tool.

dboreham 5y ago
Another workaround is to use WSL2.
- pineconewarrior 5y ago
  You can still get improvements in some scenarios with mutagen on WSL2
- kyriakos 5y ago
  I am using WSL2 - issue exists on both versions.
bliteben 5y ago
its not that bad if you use the native filesystem, although then you have to actually get your files into the native filesystem
- kyriakos 5y ago
  Using IDE in Windows and execution in wsl2. So IDE has files on Windows file system and use mutagen to keep them in sync with wsl2 file system via ssh. It's actually instant, only downside is duplication but thats a small price to pay.

xenoscopic 5y ago

Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project. [0]: https://github.com/mutagen-io/mutagen-compose https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose https://mutagen.io/documentation/orchestration/compose

slightwinder 5y ago
How robust is the handling? Can it tolerate unreliable network? Lost connections and hiccups?
- xenoscopic 5y ago
  It can tolerate disconnection very reliably, and it will automatically reconnect to synchronization and forwarding endpoints as soon as possible. Its synchronization algorithm in particular (which is essentially a (repeated) three-way merge with rsync-style differential file transfers) is designed for safety and robust tracking of the changes that it has propagated. It's also capable of resuming file transfers once it reconnects.
lifty 5y ago
Thank you for this great project! I’ve been using it for a while now and it has been rock solid and easy to use. Cheers!
vorpalhex 5y ago
Thank you for building this - it appears to solve a real problem that has been a pain in my butt for a couple of years. What are the plans for funding the project? It looks like everything is OSS now, will there be any closed source elements in the future?
- xenoscopic 5y ago
  To be totally transparent: funding is a bit of an open question, although there's no risk of Mutagen disappearing in the short term. Mutagen was actually part of YC's S19 batch, and it still has a significant amount of runway from that, but also some revenue from contracting work. I really want to keep as much of Mutagen as FOSS as possible, ideally MIT licensed. I did just add a small portion of code under the SSPL, which I might experiment with dual licensing to SaaS embedders of Mutagen (since it's really only useful in cases where Mutagen is being embedded in other tooling), but even that I wanted to keep open source for other FOSS projects embedding Mutagen. In the near term, I have some ideas about plugins and tooling that I want to build on top of Mutagen that will probably be closed source, but those will be separate entities from Mutagen itself and the aim will be to avoid compromising on any functionality that belongs in the core of Mutagen.

xjia 5y ago

Does the compiler run on my workstation or in the cloud? I read through the docs but didn't find anything about this.

xenoscopic 5y ago
There's no compiler here - it's a tool for bidirectionally synchronizing files (specifically code, assets, build products, etc.) and forwarding network traffic to and from remote systems. The idea behind Mutagen is to use your local tools (e.g. editor, IDE, browser, etc.) to work on remote hardware (where "remote" can mean anything you like, e.g. your local system, a Docker container (local or remote), a Raspberry Pi, a cloud server, etc. - or some combination of all of those). The architecture is designed to be maximally flexible, so you can work between your local system and a remote system, between two remote systems, or in any other topology you want.
- dangom 5y ago
  It sounds like sshfs on steroids?
  - qxxx 5y ago
    with mutagen you have a 1:1 copy on your local machine and remote machine. If the connection drops, you have still all your files on eg your laptop, with sshfs your files are "gone" locally. If your connection is back again, mutagen will start the sync and upload/download files again (only changed) so there is again a 1:1 copy.
qbasic_forever 5y ago
In the kind of setup you'd use with mutagen your dev tools and compilers are typically inside the container, while your code and editor are outside the container and being synced inside the container on every change. If the container is running on another machine/cloud service then yeah it could be that your compiler is running outside your machine. But much more likely it's still running as a container on your machine. The typical use case for this is making your dev environment accessible to anyone, just send them a dockerfile and they can almost perfectly reproduce it and get going with building your software (vs. having to laboriously setup the exact version of each dev tool and compiler you use).

gr8tnwz 5y ago

I’m sure this feels like an accomplishment but in practice it provides nothing over git/source control as usual, and any number of reliable open or free hosted file sharing options. What specifically is missing in the current ecosystem of electron state moving technologies that this solves? Why this instead of git and a Wireguard based mesh of devices (to offer my current setup as an example; there are others).

pineconewarrior 5y ago
Speed. This works so fast that it improves the efficiency of local development with containers.
- ab00tth4t 5y ago
xenoscopic 5y ago
The goal with Mutagen is real-time remote development, not source control or complex networking. It's designed to facilitate making changes to code locally and then having those immediately propagated to a remote system for compilation, execution, etc. Basically, it's designed for the work you do between Git commits, saving you the need to do a full commit, push, pull, evaluate cycle to test your code on a remote system. Basically, it's like VS Code's remote development extensions, but editor-agnostic and more flexible.

karlshea 5y ago

I've been testing Mutagen's file sync through DDEV on a Drupal project and it's pretty nice! The project is way more snappy than using NFS mounts on the Mac (which was already WAY faster than bind mounts), and DDEV has file ignores set up correctly so the project's media files aren't synced and disk usage isn't doubled.

pineconewarrior 5y ago
Same! Mutagen + DDEV have been a godsend for me
xenoscopic 5y ago
Yeah, Randy's done a great job with the integration - it seems to work seamlessly for most users as far as I can tell. I'm really excited to get Mutagen v0.14 and beyond into DDEV, because support for fanotify[0] is being added and it's going to make Mutagen's Linux filesystem watching unbelievably faster for container-based development. [0]: https://man7.org/linux/man-pages/man7/fanotify.7.html https://man7.org/linux/man-pages/man7/fanotify.7.html
- TrickardRixx 5y ago
  I'm switching from using docker-compose + docker-sync[0] to using mutagen-compose and am recommending my coworkers across Windows, Intel Mac, and Apple Silicon Mac all do the same. For me mutagen-compose is solving the same performance file mount performance problems I was using docker-sync to solve, but mutagen is doing it much more reliably. I would have to restart the docker-sync (and sometimes the docker daemon) multiple times a day. So far mutagen "just works". [0]: https://github.com/EugenMayer/docker-sync https://github.com/EugenMayer/docker-sync
  - xenoscopic 5y ago
    Glad to hear that it works for you. Don't hesitate to reach out on the Mutagen issue tracker or Slack channel if you run into any issues or questions - I'm always happy to help.
5y ago

incanus77 5y ago

I’ve been using Mutagen with Docker for about 3-4 months to work around an (at the time) performance problem with bound volumes. I understand the new experimental FS option in Docker might solve this, but I haven’t tested or migrated my setup yet. Basically, I’m working on a bare metal kernel in C/C++ on ARM, which only has tooling on Windows or Linux. As I work on a Mac system, I setup a Docker container that makes this trivial for me, including GDB debugging of QEMU on the host machine. Mutagen was easy to get going, well documented, and has worked perfectly for this. I’m able to make my edits in my Mac IDE, then literally by the time I’ve switched to compilation, the edits are synced to the container, and vice-versa for the build product. Prior to Mutagen, I was getting 8-10x longer build times since Docker file synchronization was slowing things down. Mutagen syncs slower than Docker, but faster than I can work, so all is well.

eysi 5y ago

We use Mutagen for Garden's hot reloading mechanism. (Garden is a dev tool for K8s and hot reloading enables users to sync changes directly to a prod like dev environment as opposed to doing a rebuild and re-deploy). It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms. If anyone's interested in how we use it, it's here: https://github.com/garden-io/garden/blob/master/core/src/plugins/kubernetes/mutagen.ts https://github.com/garden-io/garden/blob/master/core/src/plu...

gavinray 5y ago
This is a MASSIVE endorsement as far as I am concerned I used Garden in the rsync days and it was already pretty good. The code behind Garden is some of the best TypeScript + Node I've ever seen, so if this was better and more performant than what you already had I'm sold. Tell Jon I said hi =)
- eysi 5y ago
  Thank you say much for the kind words Gavin :)

InsaneOstrich 5y ago

I've been using Mutagen for years to offload running docker containers during local development from my Mac to a Linux machine and it works great. I'm glad to see more people talking about it.

stephen 5y ago

Huh! This sounds a lot like my hobby project to real-time sync files between my laptop & desktop: https://github.com/stephenh/mirror https://github.com/stephenh/mirror But dressed up to work with docker + "just" :-) being a professional/polished product, which is impressive. Neat!

mutagen 5y ago

Great name! waves (not associated)

inportb 5y ago

Looks like it handles two-way sync very well, but could it keep more than two participants in sync? I'm sort of "abusing" Syncthing to sync Docker volumes (mostly just configuration and helper scripts) across several hosts. It works well, and the only things I'm missing are file ownership preservation and low-latency "instant"-feeling sync. csync2 gets it right, but it's fragile and needs a lot of babysitting. I'm wondering if Mutagen might be a suitable alternative.

xenoscopic 5y ago
One way you can do this with Mutagen is to use a hub-and-spoke topology, where one copy of the files (say the one on the laptop where you're coding) is the "hub," and then you have multiple two-way (or one-way) synchronization sessions going out to the various "spoke" endpoints. In the two-way case, changes on a spoke endpoint will be propagated out to the other endpoints by first propagating to the hub. Obviously, you can have contention in this case if you have ultra-high-frequency updates to the same sets of of files, though the conflicting operations would have to occur on a smaller timescale than that of a single Mutagen synchronization cycle (which is typically somewhere around 100ms for an average codebase). Also, in the case of contention, Mutagen will simply display the conflicting files to you and you can delete the "losing" copy. You can also set auto-resolution behavior with Mutagen's `two-way-resolved` mode to let the hub win in conflicts, or use `one-way-replica` mode to make all the spoke endpoints replicas of the hub, or some combination of all these things, etc.

erulabs 5y ago

Mutagen is great (I really wish Skaffold implemented it for it's sync) and I want to point out that Jacob, the author is the project, is a really awesome and intelligent person - few people in our YC batch deserve the front-page more!

galaxyLogic 5y ago

Just curious, when you develop for the Cloud where to you debug your code? In the cloud or on your work-station?

faizshah 5y ago
For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM. For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment. I use a file synchronization tool like mutagen at work (for ninjas) and I use mutagen on my personal laptop for the same workflow. I’m thinking of trying remote development from jetbrains soon though: https://www.jetbrains.com/remote-development/ https://www.jetbrains.com/remote-development/ I generally prefer using a remote vm for development and only using my laptop for git. Also you can use spot VMs for development so it can be cheap to get a high CPU vm for building and debugging or a high mem VM for running data science scripts and debugging them.
- memco 5y ago
  VSCode makes it very easy to selectively install plugins on a remote that doesn’t have internet access and allows fairly seamless editing, debugging and terminal access to a remote with very little effort. But I’ve been looking for alternatives to VSCode’s remote editing experience because I’ve generally got about 6-8 projects that I keep open all the time and it can eat up a lot of RAM and CPU especially when I have several of them running and debugging stuff in tandem. The trick for me is that beyond file editing synchronization I need to be able to debug the code. There’s some promising ideas here, but I haven’t been able to make the jump yet. The jet brains link you shared is the most similar offering I’ve seen, but I’m hopeful distant.nvim or mutagen (plus whatever else I’d need) can be used to get more tools on a similar level of seamlessness.
  - galaxyLogic 5y ago
    I'm asking because I'm contemplating on doing some work on AWS. If code runs on AWS doesn't it mean it makes lots of calls to the different AWS APIs. And then in which environment do you debug the code that makes those API-calls? Is it even possible to debug it locally?

qxxx 5y ago

I have been using mutagen for few years. It has been a life saver. I use it mostly to work on remote projects, mostly for development, rarely working on prod. So there I have my projects on some cloud linux instances and I work locally on windows, mostly phpstorm. Having the webserver in the cloud and doing the heavy lifting is quite nice, especially if you work with phpstorm, which can be a resource monster for bigger projects, and working on remote projects with phpstorm built in sftp / file sync is a huge pain in the ass, really slow. With mutagen it feels almost like I work locally. Thanks to the developer for this cool tool!

zmmmmm 5y ago

Looks like a really handy tool and essentially the logical endpoint of all the various hacky manual file watcher / scripted copy type sync things we end up doing to work across environments if they were developed into a full solution. We work quite hard to maintain unified production/dev config for our docker-compose setups. Mutagen looks like it could simplify some of these since a whole layer of complexity comes from trying to optionally map ports and volumes in such a way that it works for development, while not being inappropriate in test/production. If instead, mutagen can virtually connect / synchronize these it'll be a lot simpler. I do wonder a bit how debugging works .... if something is running in a container, it's great I can hot sync my code there, but what happens when I put a breakpoint in the code in my IDE?

kamikazeturtles 5y ago

Would it be possible to integrate this with a browser editor (like codemirror) to create something like replit?

osener 5y ago

Mutagen and mutagen-compose + an EC2 instance work great as a replacement for Docker for Mac. This setup you to develop and run Intel images without performance penalty. Both on M1 and on Intel I’m never going back to Docker for Mac again if I help it.

jeffnappi 5y ago
An important thing to note is that the _worst_ part about Docker for Mac is the lousy disk IO performance for shared drives. This can be almost fully mitigated by using NFS mounts instead[0]. In general Docker for Mac is far from ideal - particularly considering you must allocate a set block of memory to the VM. If only it was the year of the Linux Desktop :grin: [0] https://gist.github.com/vschoener/b0b5ae08625744a1f4ad4140275d9675 https://gist.github.com/vschoener/b0b5ae08625744a1f4ad414027...

xwowsersx 5y ago

This looks interesting, but I'm not sure I correctly understand the use case or how exactly this might fit into my development workflow. Can someone enlighten me and elaborate on what this solves and what its use cases are? Thanks.