8 ms·

Ahh, /etc/machine-id, we meet again... Last time I met you, you were causing my cloned VM to get the same DHCP address as its original and blow up my VM network

by nyx 4y ago

Ahh, /etc/machine-id, we meet again... Last time I met you, you were causing my cloned VM to get the same DHCP address as its original and blow up my VM network, despite libvirt's virt-clone utility properly randomizing vNIC MAC addresses, because netplan defaults to using you as the DHCP client identifier. If only we could meet under more pleasant circumstances.

cosmin800 4y ago
this is not an ubuntu only issue, happens on debian too
dark-star 4y ago
Can't you just get rid of that file entirely? I would assume that there is a (sane) fallback for DHCP in case this file is missing? What other uses besides the phased updates and DHCP does that file solve?
- tremon 4y ago
  Google Chrome reads it, probably for tracking purposes.
usr1106 4y ago
As another comment pointed out elsewhere you are not supposed to touch /etc/machine-id to control apt. It has its own Apt::Machine-ID you can change without breaking anything else.
- yjftsjthsd-h 4y ago
  That's almost worse in my opinion. If you're going to have unique machine ids, is it too much to ask that there's only one to deal with?
  - Denvercoder9 4y ago
    In the default configuration apt uses /etc/machine-id, it just has its own override (which is reasonable, imo).
    - yjftsjthsd-h 4y ago
      Oh, perfect; yes, that's the best way to do it. (Best of both worlds, perhaps I should say.)
lima 4y ago
virt-clone only takes care of the libvirt config but won't touch unique identifiers in your disk image. /etc/machine-id isn't the only thing to worry about when duplicating a VM - think SSH host key, DHCP leases, various filesystem UUIDs, log files, MAC addresses in ifcfg-* files and udev-persistent-net rules... You can use virt-sysprep[1] to clean up a disk image. [1]: https://www.libguestfs.org/virt-sysprep.1.html https://www.libguestfs.org/virt-sysprep.1.html
- Godel_unicode 4y ago
  Cloning VMs is kind of an anti-pattern anyway, it’s absolutely full of this type of foot gun. Just use ansible or similar and build a new one.
  - lathiat 4y ago
    or cloud-init boot the clean base image and configure it on boot with packages, ssh keys, custom commands, etc.
  - selfhoster11 4y ago
    Ansible has a non-trivial transition cost (speaking from experience here). Cloning VMs is a legitimate stopgap measure, because that’s sometimes the best you can do if available engineer hours are tight.
    - Godel_unicode 4y ago
      Ansible (as a proxy for infrastructure as code in general) being hard to implement is a warning that your setup is too convoluted. Cloning VMs is the high-interest unsecured loan of tech debt, and when that bill comes due it’s going to be much worse than spending a few days on some scripting.
      - deleted 4y ago
        [deleted]
      - selfhoster11 4y ago
kstenerud 4y ago
NixOS has a related problem in that you can't set the machine's MAC address without a hack: # Hack: Change the default MAC address after network but before dhcpcd runs systemd.services.setmacaddr = { script = '' /run/current-system/sw/bin/ip link set dev eth0 address ${macaddr} /run/current-system/sw/bin/systemctl stop dhcpcd.service /run/current-system/sw/bin/ip addr flush eth0 /run/current-system/sw/bin/systemctl start dhcpcd.service ''; wantedBy = [ "basic.target" ]; after = [ "dhcpcd.service" ]; }; If you don't do this, it uses some black magic to decide the MAC address based on various hardware, making system migration and maintenance a nightmare.
- mananaysiempre 4y ago
  Is it NixOS doing it or is it e.g. systemd-networkd? (Sounds like something systemd would do.) While I’m confused as to why the boot would need to decide on a MAC address (VM or cheap SBC without a burned-in address? the first case might be easier to correct from the outside), the general state of NixOS is that some things are very flexible while others only cover some common cases (ones that the original author needed to solve). Unlike a traditional distro where the package manager will complain if you replace distro-provided stuff, in NixOS it’s entirely possible to override parts that don’t work for you rather than paper over them with programmatic overrides like these. It’s not even hard to upstream your changes if you make them backwards-compatible, although the benefit can be limited because the testing is not particularly thorough so other changes may still inadvertently break them.
  - kstenerud 4y ago
    I gave up trying to find a root cause for this after a couple of days of rabbit holes and yak shaving. NixOS is simply too impenetrable once you fall off the happy path (which is unnervingly often). The next time I rebuild this server, I'll go back to Ubuntu or Debian and use build scripts for "good enough" determinism. For the time being, I just run everything important in LXC and Docker containers on top of this delicately balanced NixOS hypervisor for as long as it'll last.
tjoff 4y ago
What is the reasonable for not use the mac as the identifier? Trying to think of a use case I thought of wireless + wired, would be kind of neat if they had the same IP. But that falls apart completely if you have them connected at the same time (which I often do).
- linsomniac 4y ago
  At one point I set up my wired and wireless interfaces as a bond with wired as the primary, and I could do a file transfer and watch the speed go up or down as I plugged and unplugged the wired interface. That was pretty slick.
  - josteink 4y ago
    That sounds cool. Did you document it anywhere for others to setup themselves? We’re there any obvious issues in such a setup?
    - linsomniac 4y ago
      I did, but that blog was taken offline the beginning of this year when the company I was a part of when I wrote it went out off business. It was horribly out of pace with modern setups though, I did that ~20 years ago, it used a network manager that no longer exists. I'm not sure how it'd fit into a Network Manager or systemd world. There really wasn't any particular trick to it, IIRC I used link monitoring to detect link failure, though I might have used ARP, and set the ethernet as the primary interface, just using the standard Linux bond driver.
- justinsaccount 4y ago
  With things like docking stations/usb-c docks/other adapters, the actual mac address doesn't necessarily identify a machine.
  - tjoff 4y ago
    As my example illustrates, that is a feature.
magicalhippo 4y ago
Back in the NT4/Windows 2k days I recall having to do some extra steps to modify an identifier in the registry (or similar) when cloning Windows images. Otherwise the clone would not properly register on the network. Perhaps it was only when speaking to the Domain Controller though. IIRC the later versions of Norton Ghost, which was what we used, did this process for us automatically.
- luma 4y ago
  It's the SID you were changing, and it turns out it was never actually required: https://techcommunity.microsoft.com/t5/windows-blog-archive/the-machine-sid-duplication-myth-and-why-sysprep-matters/ba-p/723859 https://techcommunity.microsoft.com/t5/windows-blog-archive/... Russinovich eventually pulled the tool from circulation.
  - magicalhippo 4y ago
    Ah yes, that was it. I do recall we got error messages preventing the machine from working, and the guys spent some time researching before a solution was found. The error messages went away after changing the SID.
  - toyg 4y ago
    There are some conspiracy theories, fuelled by the fact that Russinovich only pulled the tool after being acquihired by Microsoft. It's undeniable, in my experience, that the tool did help, despite all the swearing to the contrary. Making a leap from there to believing that it probably made it too easy to clone Windows machines in a way that Microsoft had no control on, and hence asked him to pull it, doesn't seem so crazy though.
    - Godel_unicode 4y ago
      You just needed to actually read the documentation and use sysprep (ideally with an unattend file). Just remember to image it before sysprepping since you can only run it a few times. There are a ton of things sysprep does that are really helpful and not handled at all by tools like that. Microsoft doesn’t care if you clone systems, (why would they?) they care about the volume of help desk tickets created by doing it wrong.
ilyt 4y ago
When I read what netplan is supposed to be I thought "FINALLY", when I saw who is making it (Canonical) I said "fuck, they will fuck it up"

by nyx 4y ago

cosmin800 4y ago

this is not an ubuntu only issue, happens on debian too

dark-star 4y ago

Can't you just get rid of that file entirely? I would assume that there is a (sane) fallback for DHCP in case this file is missing? What other uses besides the phased updates and DHCP does that file solve?

tremon 4y ago
Google Chrome reads it, probably for tracking purposes.

usr1106 4y ago

As another comment pointed out elsewhere you are not supposed to touch /etc/machine-id to control apt. It has its own Apt::Machine-ID you can change without breaking anything else.

yjftsjthsd-h 4y ago
That's almost worse in my opinion. If you're going to have unique machine ids, is it too much to ask that there's only one to deal with?
- Denvercoder9 4y ago
  In the default configuration apt uses /etc/machine-id, it just has its own override (which is reasonable, imo).
  - yjftsjthsd-h 4y ago
    Oh, perfect; yes, that's the best way to do it. (Best of both worlds, perhaps I should say.)

lima 4y ago

virt-clone only takes care of the libvirt config but won't touch unique identifiers in your disk image. /etc/machine-id isn't the only thing to worry about when duplicating a VM - think SSH host key, DHCP leases, various filesystem UUIDs, log files, MAC addresses in ifcfg-* files and udev-persistent-net rules... You can use virt-sysprep[1] to clean up a disk image. [1]: https://www.libguestfs.org/virt-sysprep.1.html https://www.libguestfs.org/virt-sysprep.1.html

Godel_unicode 4y ago
Cloning VMs is kind of an anti-pattern anyway, it’s absolutely full of this type of foot gun. Just use ansible or similar and build a new one.
- lathiat 4y ago
  or cloud-init boot the clean base image and configure it on boot with packages, ssh keys, custom commands, etc.
- selfhoster11 4y ago
  Ansible has a non-trivial transition cost (speaking from experience here). Cloning VMs is a legitimate stopgap measure, because that’s sometimes the best you can do if available engineer hours are tight.
  - Godel_unicode 4y ago
    Ansible (as a proxy for infrastructure as code in general) being hard to implement is a warning that your setup is too convoluted. Cloning VMs is the high-interest unsecured loan of tech debt, and when that bill comes due it’s going to be much worse than spending a few days on some scripting.
    - deleted 4y ago
      [deleted]
    - selfhoster11 4y ago

kstenerud 4y ago

NixOS has a related problem in that you can't set the machine's MAC address without a hack: # Hack: Change the default MAC address after network but before dhcpcd runs systemd.services.setmacaddr = { script = '' /run/current-system/sw/bin/ip link set dev eth0 address ${macaddr} /run/current-system/sw/bin/systemctl stop dhcpcd.service /run/current-system/sw/bin/ip addr flush eth0 /run/current-system/sw/bin/systemctl start dhcpcd.service ''; wantedBy = [ "basic.target" ]; after = [ "dhcpcd.service" ]; }; If you don't do this, it uses some black magic to decide the MAC address based on various hardware, making system migration and maintenance a nightmare.

mananaysiempre 4y ago
Is it NixOS doing it or is it e.g. systemd-networkd? (Sounds like something systemd would do.) While I’m confused as to why the boot would need to decide on a MAC address (VM or cheap SBC without a burned-in address? the first case might be easier to correct from the outside), the general state of NixOS is that some things are very flexible while others only cover some common cases (ones that the original author needed to solve). Unlike a traditional distro where the package manager will complain if you replace distro-provided stuff, in NixOS it’s entirely possible to override parts that don’t work for you rather than paper over them with programmatic overrides like these. It’s not even hard to upstream your changes if you make them backwards-compatible, although the benefit can be limited because the testing is not particularly thorough so other changes may still inadvertently break them.
- kstenerud 4y ago
  I gave up trying to find a root cause for this after a couple of days of rabbit holes and yak shaving. NixOS is simply too impenetrable once you fall off the happy path (which is unnervingly often). The next time I rebuild this server, I'll go back to Ubuntu or Debian and use build scripts for "good enough" determinism. For the time being, I just run everything important in LXC and Docker containers on top of this delicately balanced NixOS hypervisor for as long as it'll last.

tjoff 4y ago

What is the reasonable for not use the mac as the identifier? Trying to think of a use case I thought of wireless + wired, would be kind of neat if they had the same IP. But that falls apart completely if you have them connected at the same time (which I often do).

linsomniac 4y ago
At one point I set up my wired and wireless interfaces as a bond with wired as the primary, and I could do a file transfer and watch the speed go up or down as I plugged and unplugged the wired interface. That was pretty slick.
- josteink 4y ago
  That sounds cool. Did you document it anywhere for others to setup themselves? We’re there any obvious issues in such a setup?
  - linsomniac 4y ago
    I did, but that blog was taken offline the beginning of this year when the company I was a part of when I wrote it went out off business. It was horribly out of pace with modern setups though, I did that ~20 years ago, it used a network manager that no longer exists. I'm not sure how it'd fit into a Network Manager or systemd world. There really wasn't any particular trick to it, IIRC I used link monitoring to detect link failure, though I might have used ARP, and set the ethernet as the primary interface, just using the standard Linux bond driver.
justinsaccount 4y ago
With things like docking stations/usb-c docks/other adapters, the actual mac address doesn't necessarily identify a machine.
- tjoff 4y ago
  As my example illustrates, that is a feature.

magicalhippo 4y ago

Back in the NT4/Windows 2k days I recall having to do some extra steps to modify an identifier in the registry (or similar) when cloning Windows images. Otherwise the clone would not properly register on the network. Perhaps it was only when speaking to the Domain Controller though. IIRC the later versions of Norton Ghost, which was what we used, did this process for us automatically.

luma 4y ago
It's the SID you were changing, and it turns out it was never actually required: https://techcommunity.microsoft.com/t5/windows-blog-archive/the-machine-sid-duplication-myth-and-why-sysprep-matters/ba-p/723859 https://techcommunity.microsoft.com/t5/windows-blog-archive/... Russinovich eventually pulled the tool from circulation.
- magicalhippo 4y ago
  Ah yes, that was it. I do recall we got error messages preventing the machine from working, and the guys spent some time researching before a solution was found. The error messages went away after changing the SID.
- toyg 4y ago
  There are some conspiracy theories, fuelled by the fact that Russinovich only pulled the tool after being acquihired by Microsoft. It's undeniable, in my experience, that the tool did help, despite all the swearing to the contrary. Making a leap from there to believing that it probably made it too easy to clone Windows machines in a way that Microsoft had no control on, and hence asked him to pull it, doesn't seem so crazy though.
  - Godel_unicode 4y ago
    You just needed to actually read the documentation and use sysprep (ideally with an unattend file). Just remember to image it before sysprepping since you can only run it a few times. There are a ton of things sysprep does that are really helpful and not handled at all by tools like that. Microsoft doesn’t care if you clone systems, (why would they?) they care about the volume of help desk tickets created by doing it wrong.

ilyt 4y ago

When I read what netplan is supposed to be I thought "FINALLY", when I saw who is making it (Canonical) I said "fuck, they will fuck it up"