5 ms·

The email mostly talks about performance, but could this technique finally give us VM shared folders with proper inotify support (at least on Linux hosts)? That

The email mostly talks about performance, but could this technique finally give us VM shared folders with proper inotify support (at least on Linux hosts)? That has always been my number one gripe with using Vagrant for local development.

noir_lord 8y ago
Mine as well, I have this horrible NFS setup that kinda/sorta works but this (if what you query is the case) would be awesome. I even looked at writing my own 'vagrant' (to fit exactly my use case of Linux on Linux via KVM and deployed via ansible) because the NFS thing and VirtualBox issues mar what is otherwise a great idea.
- jitl 8y ago
  Vagrant is quite extensible; you can plug in KVM via QEMU and Libvirt as a VM "provider", and use Ansible to provision the guest out of the box. vagrant-libvirt (3rd party): https://github.com/vagrant-libvirt/vagrant-libvirt https://github.com/vagrant-libvirt/vagrant-libvirt ansible provisioning (built in): https://www.vagrantup.com/docs/provisioning/ansible.html https://www.vagrantup.com/docs/provisioning/ansible.html This isn't to say these will make the guest filesystem more performant than NFS; you'll probably hit the same bottlenecks with mounting host filesystems into the guest. The solution many users arrived at with Vagrant is to use a user-space synchronization program like Unison that watches for filesystem events (eg, inotify) on both the guest and the host. This is its own can of worms: O(size of files monitored) resource use on both hosts, file ignore lists, async races, etc. There's even a Unison plugin for vagrant, but I found it quite finicky and thus unsuitable for large-scale deployment unison: https://www.cis.upenn.edu/~bcpierce/unison/ https://www.cis.upenn.edu/~bcpierce/unison/ unison-vagrant2: https://github.com/dcosson/vagrant-unison2 https://github.com/dcosson/vagrant-unison2 At Airbnb, we eventually wrote our own wrapper around Unison to make it easier to use, and saw 90x better perf for filesystem-heavy loads like Webpack once we switched off of NFS in our local VMs.
  - noir_lord 8y ago
    Thanks for the info, I was aware of vagrant-libvirt and ansible (already use that to deploy vagrant guests) but unison I hadn't seen. Funnily enough webpack is where I had the most problems (it certainly surfaces "what happens when we smack the FS in the face repeatedly" issues better than most) which is what got me thinking down the KVM route in the first place. In terms of hacking around on vagrant, it's ruby which isn't on the list of languages I'm fluent in and its flexibility makes it more complex than what I need for my use case. There is something to be said sometimes for rolling your own for your own itch but I'll certainly have a look at unison as part of vagrant first.
  - RussianCow 8y ago
    > At Airbnb, we eventually wrote our own wrapper around Unison to make it easier to use Is there any chance you guys open sourced this wrapper, or have any plans to? :)
    - jitl 8y ago
      Unlikely. 1. I no longer work at Airbnb, so I can't help you there. 2. The wrapper is somewhat deeply embedded in the dev tools swiss army knife, and is written in Ruby, so (a) it's difficult to extract since its tied with other Airbnb specific concerns and support libraries, and (b) distributing perf-sensitive Ruby code to end-users is challenging so you might not want it anyways. You might want to take a look at Mutagen, a unison replacement written in Go which seems to avoid most of what makes Unison annoying to work with. I haven't tested it, though, so I can't vouch for it. https://github.com/havoc-io/mutagen https://github.com/havoc-io/mutagen
      - RussianCow 8y ago
        That makes sense. I hadn't heard of Mutagen before, but it seems like it would solve a lot of the issues that drove me away from Unison. Thanks for mentioning it!
    - crgwbr 8y ago
      Here’s a similar project we made for syncing stuff to remote docker hosts: https://github.com/crgwbr/accordance https://github.com/crgwbr/accordance
- bryanlarsen 8y ago
  Vagrant-lxc works for us. File sharing is a bind mount so no problems with inotifu.
  - RussianCow 8y ago
    Did it work well for you out of the box, or did you have do some configuration early on? I haven't gotten this to work for me since I upgraded LXC to 2.0 (something about the `lxc-create` command failing IIRC), but I never spent much time digging into it, so it's entirely possible it's something to do with my setup and not vagrant-lxc.
mayli 8y ago
Same here, near-native performance is quite critical for certain use cases, including kernel compiling. If you ever tried to compile from source over NFS, that's almost 10~30 times slower than compile from a native/local filesystem.
bbeausej 8y ago
I sure hope so! I'm encouraged to see a project potentially tackling this need. The current "best practice" to use a unison synched directory between the host and your VM (and even then maybe to a docker volume bind mount) is very error prone and comes with many pitfalls.
fulafel 8y ago
Does Vagrant have its own file sharing system, or does it use whatever the backends (mount for lxd, 9p for kvm, proprietary kernel module for virtualbox etc) happen to provide?
- ofrzeta 8y ago
  It uses different technologies depending on the used provider. For instance Virtualbox shared folders, NFS, SMB or rsync'd folders.

noir_lord 8y ago

Mine as well, I have this horrible NFS setup that kinda/sorta works but this (if what you query is the case) would be awesome. I even looked at writing my own 'vagrant' (to fit exactly my use case of Linux on Linux via KVM and deployed via ansible) because the NFS thing and VirtualBox issues mar what is otherwise a great idea.

jitl 8y ago
Vagrant is quite extensible; you can plug in KVM via QEMU and Libvirt as a VM "provider", and use Ansible to provision the guest out of the box. vagrant-libvirt (3rd party): https://github.com/vagrant-libvirt/vagrant-libvirt https://github.com/vagrant-libvirt/vagrant-libvirt ansible provisioning (built in): https://www.vagrantup.com/docs/provisioning/ansible.html https://www.vagrantup.com/docs/provisioning/ansible.html This isn't to say these will make the guest filesystem more performant than NFS; you'll probably hit the same bottlenecks with mounting host filesystems into the guest. The solution many users arrived at with Vagrant is to use a user-space synchronization program like Unison that watches for filesystem events (eg, inotify) on both the guest and the host. This is its own can of worms: O(size of files monitored) resource use on both hosts, file ignore lists, async races, etc. There's even a Unison plugin for vagrant, but I found it quite finicky and thus unsuitable for large-scale deployment unison: https://www.cis.upenn.edu/~bcpierce/unison/ https://www.cis.upenn.edu/~bcpierce/unison/ unison-vagrant2: https://github.com/dcosson/vagrant-unison2 https://github.com/dcosson/vagrant-unison2 At Airbnb, we eventually wrote our own wrapper around Unison to make it easier to use, and saw 90x better perf for filesystem-heavy loads like Webpack once we switched off of NFS in our local VMs.
- noir_lord 8y ago
  Thanks for the info, I was aware of vagrant-libvirt and ansible (already use that to deploy vagrant guests) but unison I hadn't seen. Funnily enough webpack is where I had the most problems (it certainly surfaces "what happens when we smack the FS in the face repeatedly" issues better than most) which is what got me thinking down the KVM route in the first place. In terms of hacking around on vagrant, it's ruby which isn't on the list of languages I'm fluent in and its flexibility makes it more complex than what I need for my use case. There is something to be said sometimes for rolling your own for your own itch but I'll certainly have a look at unison as part of vagrant first.
- RussianCow 8y ago
  > At Airbnb, we eventually wrote our own wrapper around Unison to make it easier to use Is there any chance you guys open sourced this wrapper, or have any plans to? :)
  - jitl 8y ago
    Unlikely. 1. I no longer work at Airbnb, so I can't help you there. 2. The wrapper is somewhat deeply embedded in the dev tools swiss army knife, and is written in Ruby, so (a) it's difficult to extract since its tied with other Airbnb specific concerns and support libraries, and (b) distributing perf-sensitive Ruby code to end-users is challenging so you might not want it anyways. You might want to take a look at Mutagen, a unison replacement written in Go which seems to avoid most of what makes Unison annoying to work with. I haven't tested it, though, so I can't vouch for it. https://github.com/havoc-io/mutagen https://github.com/havoc-io/mutagen
    - RussianCow 8y ago
      That makes sense. I hadn't heard of Mutagen before, but it seems like it would solve a lot of the issues that drove me away from Unison. Thanks for mentioning it!
  - crgwbr 8y ago
    Here’s a similar project we made for syncing stuff to remote docker hosts: https://github.com/crgwbr/accordance https://github.com/crgwbr/accordance
bryanlarsen 8y ago
Vagrant-lxc works for us. File sharing is a bind mount so no problems with inotifu.
- RussianCow 8y ago
  Did it work well for you out of the box, or did you have do some configuration early on? I haven't gotten this to work for me since I upgraded LXC to 2.0 (something about the `lxc-create` command failing IIRC), but I never spent much time digging into it, so it's entirely possible it's something to do with my setup and not vagrant-lxc.

mayli 8y ago

Same here, near-native performance is quite critical for certain use cases, including kernel compiling. If you ever tried to compile from source over NFS, that's almost 10~30 times slower than compile from a native/local filesystem.

bbeausej 8y ago

I sure hope so! I'm encouraged to see a project potentially tackling this need. The current "best practice" to use a unison synched directory between the host and your VM (and even then maybe to a docker volume bind mount) is very error prone and comes with many pitfalls.

fulafel 8y ago

Does Vagrant have its own file sharing system, or does it use whatever the backends (mount for lxd, 9p for kvm, proprietary kernel module for virtualbox etc) happen to provide?

ofrzeta 8y ago
It uses different technologies depending on the used provider. For instance Virtualbox shared folders, NFS, SMB or rsync'd folders.