4 ms·

I fucking love LXC! It allowed me to “virtualize” several physical servers on a single newer server, keeping all of the original setup (except some networking c

by fuzzy2 4y ago

I fucking love LXC! It allowed me to “virtualize” several physical servers on a single newer server, keeping all of the original setup (except some networking changes). Combined with ipvlan, I could even work around the MAC address restriction of my server provider, with both IPv4 and IPv6. I also use it to host a Jitsi instance. Jitsi is rather picky about the underlying distribution. All this without the overhead of an actual virtual machine of course. The only pain I have with LXC is its behavior when errors occur. It’s not easy to tell what’s wrong, the error messages are worthless. Sometimes the debug log can help, sometimes not so much.

fariszr 4y ago
You are using hetzner right? I'm interested to see how you bypassed the Mac address problem, any blogs, guides ?
- dspillett 4y ago
  I'm not sure what the problem is, but if it is stopping you bridging the physical interface effectively, could you not define a virtual bridge with local only addresses and have the host NAT to/from that and the physical interface? Adds a little extra latency to be sure, more so if you have two or more boxes on the same network with containers that would otherwise talk directly to each other on a private vlan, but if your physical box is being a firewall/router for the tasks in the containers anyway it is already practically doing the job.
  - dizhn 4y ago
    They also provide a virtual bridge for dedicated servers. Me I could not get it to work reliably so I added another bridge on my servers which link via wireguard. It works like a lan for both the physical servers and VMs.
- fuzzy2 4y ago
  Yes, it’s Hetzner. The magic ingredient is ipvlan, a special interface type loosely related to macvlan. An ipvlan interface is tied to a physical interface. Whether traffic goes to the virtual interface is decided based on the IP address alone. The ipvlan virtual interface can be moved to a different network namespace and will still work. Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this. There’s good info in the Linux kernel docs: https://www.kernel.org/doc/html/latest/networking/ipvlan.html https://www.kernel.org/doc/html/latest/networking/ipvlan.htm... I have a Debian host. In my interfaces file, I have the following code for the Jitsi ipvlan link: auto ipvl-meet iface ipvl-meet inet manual pre-up ip link add link eth0 name ipvl-meet type ipvlan mode l2 post-down ip link delete ipvl-meet Then, in my LXC guest config, I have this: lxc.net.0.type = phys lxc.net.0.link = ipvl-meet lxc.net.0.ipv4.address = 192.0.2.46/26 lxc.net.0.ipv4.gateway = 192.0.2.1 lxc.net.0.ipv6.address = 2001:0DB8::4/128 lxc.net.0.ipv6.gateway = fe80::1 A word of advice though: LXC likes to eat network interfaces when it fails to start a container. If you experiment, keep in mind that you may have to recreate the ipvlan interface after errors.
  - fariszr 4y ago
    > Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this. Does this apply to the IP supplied with the server? Because I can't find any Mac address option for it.
    - fuzzy2 4y ago
      No, the primary IP address cannot have its MAC address changed.
- dizhn 4y ago
  What mac address restriction is there? One sets them up on their robot interface so things will actually route. You get the assigned mac and use that in your virt ip or bridge settings. Are we talking about something else?
  - fuzzy2 4y ago
    The Robot interface thing is actually relatively new (some years). Before they introduced that, all your IP addresses (and maybe subnets, too?) were delivered to your physical MAC address, period. Wanted to use a VM? Maybe use proxy_arp or some 1:1 NAT. Bridging? Not possible.
    - dizhn 4y ago
      I checked my emails. My initial server with them was provisioned at the end of 2009 and it's accompanied by my Robot details. I did have IP aliasing on this bare metal server which also needed the assigned mac from Robot.

by fuzzy2 4y ago

fariszr 4y ago

You are using hetzner right? I'm interested to see how you bypassed the Mac address problem, any blogs, guides ?

dspillett 4y ago
I'm not sure what the problem is, but if it is stopping you bridging the physical interface effectively, could you not define a virtual bridge with local only addresses and have the host NAT to/from that and the physical interface? Adds a little extra latency to be sure, more so if you have two or more boxes on the same network with containers that would otherwise talk directly to each other on a private vlan, but if your physical box is being a firewall/router for the tasks in the containers anyway it is already practically doing the job.
- dizhn 4y ago
  They also provide a virtual bridge for dedicated servers. Me I could not get it to work reliably so I added another bridge on my servers which link via wireguard. It works like a lan for both the physical servers and VMs.
fuzzy2 4y ago
Yes, it’s Hetzner. The magic ingredient is ipvlan, a special interface type loosely related to macvlan. An ipvlan interface is tied to a physical interface. Whether traffic goes to the virtual interface is decided based on the IP address alone. The ipvlan virtual interface can be moved to a different network namespace and will still work. Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this. There’s good info in the Linux kernel docs: https://www.kernel.org/doc/html/latest/networking/ipvlan.html https://www.kernel.org/doc/html/latest/networking/ipvlan.htm... I have a Debian host. In my interfaces file, I have the following code for the Jitsi ipvlan link: auto ipvl-meet iface ipvl-meet inet manual pre-up ip link add link eth0 name ipvl-meet type ipvlan mode l2 post-down ip link delete ipvl-meet Then, in my LXC guest config, I have this: lxc.net.0.type = phys lxc.net.0.link = ipvl-meet lxc.net.0.ipv4.address = 192.0.2.46/26 lxc.net.0.ipv4.gateway = 192.0.2.1 lxc.net.0.ipv6.address = 2001:0DB8::4/128 lxc.net.0.ipv6.gateway = fe80::1 A word of advice though: LXC likes to eat network interfaces when it fails to start a container. If you experiment, keep in mind that you may have to recreate the ipvlan interface after errors.
- fariszr 4y ago
  > Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this. Does this apply to the IP supplied with the server? Because I can't find any Mac address option for it.
  - fuzzy2 4y ago
    No, the primary IP address cannot have its MAC address changed.
dizhn 4y ago
What mac address restriction is there? One sets them up on their robot interface so things will actually route. You get the assigned mac and use that in your virt ip or bridge settings. Are we talking about something else?
- fuzzy2 4y ago
  The Robot interface thing is actually relatively new (some years). Before they introduced that, all your IP addresses (and maybe subnets, too?) were delivered to your physical MAC address, period. Wanted to use a VM? Maybe use proxy_arp or some 1:1 NAT. Bridging? Not possible.
  - dizhn 4y ago
    I checked my emails. My initial server with them was provisioned at the end of 2009 and it's accompanied by my Robot details. I did have IP aliasing on this bare metal server which also needed the assigned mac from Robot.