Comments
You can use your Mastodon account to reply to this post.
Tuesday, September 19, 2023
Reading time 10 minutes
Once I managed to install and learn to use Libvirt, as I’ve already mentioned in a couple of posts, there was one important part for some of my projects: making a server available via IPv6.
The reason is very simple. I have an ISP that puts my IPv4 address behind CG-NAT, so it’s not possible for my servers or virtual machines to be accessed through this protocol, since my IPv4 address is actually shared with many other clients, which removes the possibility of opening or forwarding ports. However, what my provider does is give us IPv6 addresses and from what I know, each device that requests it is assigned a fully functional address that can be connected to through the internet, as I’ve also explained on another occasion. Now, what I wanted to do was essentially allow my VMs to have the ability to request their own IPv6 address from my router, so it would assign them one and thus be able to connect to SSH or any other service, for example.
In libvirt, by default, when creating a new domain, it becomes part of a network at 192.168.122.1/24. This network works and provides internet since it uses NAT (network address translation) and packet forwarding to perform its function. Internally, Libvirt adds several iptables rules to ensure the host machine knows how to send packets from libvirt’s default network to the entire internet. For practical purposes, this default network is not connected to any physical device and for many virtual machines this configuration is more than sufficient since it allows “outbound” access to the entire internet, but without more configurations, it doesn’t allow “inbound” access because the virtual machines don’t have an IP that can be resolved by any network external to libvirt itself.
For those cases where it is important to have a globally accessible IP address on the internet, however, there’s a problem. The network that libvirt provides doesn’t offer that possibility, since in reality, not being connected to any “physical” device, the IP addresses that are assigned are done internally. In my particular case, what I wanted was to allow my devices and virtual machines to be part of the same network, and be served by my router. This way, my router wouldn’t see the difference between a VM and another device connected to my home network. To do this, I basically created a virtual network bridge that’s connected to the only network device my machine has. All kinds of things can be connected to the network bridge: from other networks, as is the case with the network that libvirt creates (although for this example I’ve defined a completely new network), to docker or LXD itself (which I’ll talk about soon). A virtual network bridge is essentially a device to which other devices can be connected at its endpoints. On one end we’ll connect the real device, which provides internet and allows us to communicate with the ISP’s router, while on the other side of the bridge will be all the virtual devices that Libvirt, LXD, docker and other applications will generate. The bridge will take the packets that the virtual machines send and pass them to the router, likewise it will pass everything the router sends back through it and communicate it to the virtual machine that originated the communication. Finally, we can also assign IP addresses via DHCP or SLAAC to any virtual devices that require it.
Note for screen reader users: Creating a virtual network bridge involves having to touch the machine’s network configuration. This can cause, if there’s an error, the machine to stop responding to the network connection, and therefore to SSH, when restarting the network configuration. The way to solve these kinds of things is usually by physically accessing the machine through the keyboard, editing the interfaces file and trying to restart the network configuration. For those using Debian, for example, it’s easy if espeakup was installed during the operating system installation, because in the worst case it’s just a matter of logging into the console using espeakup and correcting the network file. But if you don’t have any way to access the machine physically and accessibly, it might not be very advisable to play with network interface configuration.
Note: These notes are based only on my configuration. In a home network, which is normally served by DHCP, this configuration should work. However, if another ISP assigns different configurations it might not work without modifications.
The first thing is to figure out which physical network device we have. This device will be used to “attach” it to the network bridge we’ll create later. It’s important to note that you can’t make a network bridge on wireless devices, like any WiFi adapter. In my case, I used the network card that came with the machine, and I was able to know its name with this command:
$ cat /etc/network/interfaces | grep iface
iface lo inet loopback
iface enp34s0 inet dhcp
In this file, on a Debian 12 without additional network managers, network devices are configured during installation. In my case I’m interested in the second line, which says “iface enp34s0 inet dhcp”, since “enp34s0” is the name of my device. To verify that this device is the one that has internet access, we can run this command:
$ ip a
This command will show many things in the output, among them the most important is the definition of the device in question. If here we can see that the physical device we want to use already has a local IPv4 address assigned, which belongs to the LAN, and one or more IPv6 addresses (one local and one global), then we can safely assume that this device is the one we need to connect to our future virtual bridge.
Before continuing, the Debian wiki recommends the bridge-utils package, which provides the brctl command and allows us to add or view different network bridges, as well as see the interfaces connected to both ends of them. I’m not entirely sure if this package is necessary when performing a manual configuration, but since it will be useful to us anyway, it’s recommended to verify if the package in question is already installed:
$ sudo apt-get install bridge-utils
To add a permanent virtual network bridge, it’s necessary to edit the network interfaces configuration file, which in Debian is /etc/network/interfaces.
$ sudo nano /etc/network/interfaces
On my machine, this configuration looks as follows, taking into account that I’ve changed the default configuration and placed the name of my device, which is “enp34s0”:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto br0
iface br0 inet dhcp
bridge_ports enp34s0
bridge_fd 0
bridge_stp off
bridge_maxwait 0
Note: It’s important to note that you must remove the enp34s0 interface definition. In this file you can only use the interface once. If in the example the definition wasn’t removed, and you try to use the same physical device to connect it to the virtual network bridge and to the internet directly, the network configuration will fail.
Once the file is saved, it’s necessary to restart either the system or the network service. In my tests, sometimes restarting the network service didn’t end up working, so restarting the system was the easiest way to ensure the network bridge always came up correctly.
$ sudo reboot
After some time, just enough to configure the new interface, we should notice that our machine is alive. Now, if we receive an IP configuration via DHCP based on the device’s MAC address, we’ll have to pay attention, since the virtual bridge brings its own MAC, and, in my case at least, that meant changing my local IP to a different one. Anyway, that’s nothing that can’t be checked from the Router.
Now, we can run the command again to check our network connections:
$ ip a
In my case, it shows a bunch of connections, but since our virtual bridge device is called “br0”, that’s the one we can look for. This is an example of how it could look:
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 11:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.9/24 brd 192.168.1.255 scope global dynamic br0
valid_lft 55767sec preferred_lft 55767sec
inet6 2001:db8:3333:4444:5555:6666:7777:8888/64 scope global dynamic mngtmpaddr
valid_lft 428647sec preferred_lft 428647sec
inet6 fe80::44c8:d1ff:fe62:9587/64 scope link
valid_lft forever preferred_lft forever
Finally, in my specific case and regarding IPv6 deployment, I had to set the accept_ra value to 2 for the kernel, which always allows having the ability to accept announcements from the Router regarding IPv6. If this is not activated for the device where libvirt will be connected (in this case br0), the network cannot start and might not provide internet connectivity to virtual machines. To do this, you must include the option “net.ipv6.conf.br0.accept_ra = 2” in the /etc/sysctl.conf file and restart the machine before continuing.
Once this part works, and if everything we need to work with the network (watch out for docker here) works, we can proceed to the second part, where we need to define a new network within libvirt to be able to add it as “bridged”.
The last step to be able to have global IPv6 addresses in our virtual machines is to define a new type of network in libvirt, since by default, as we’ve seen, it uses a network in NAT mode. For this, we need to create an xml file with this content:
$ sudo nano bridged-network.xml
<network>
<name>bridged-network</name>
<forward mode="bridge" />
<bridge name="br0" />
</network>
The only thing to pay attention to in this file is the network name, which we’ll use to reference it to virtual machines during their creation or when editing them, and the device we’re going to connect to, which must be the virtual bridge we created, which in this case is called br0. Once the file is saved, we’ll use virsh to define this network like this:
$ sudo virsh net-define bridged-network.xml
We also need to start the network, and configure automatic startup of it when the libvirtd service starts:
$ sudo virsh net-start bridged-network
$ sudo virsh net-autostart bridged-network
Finally, we can see if the network is currently operational with the net-list command:
$ sudo virsh net-list
Name State Autostart Persistent
----------------------------------------------------
bridged-network active yes yes
default active yes yes
When creating a new virtual machine, you can specify the network you want to use with the –network parameter. Our previous command, for a Debian machine, would look as follows:
$ sudo virt-install --name test --cdrom /var/lib/libvirt/iso/debian-12.1.0-amd64-netinst.iso --os-variant=debian11 --network network=bridged-network --disk size=20,cache=none,bus=virtio --memory 2048 --sound default --graphics spice,port=5901,listen=::,password=test1 --vcpu 2 --noautoconsole
The change is the same when dealing with a virtual machine with Windows, you just have to replace the “default” parameter in the network with our new network, in this case called bridged-network.
In the case of a virtual machine that has already been defined, we can update the file with its definition to change the network interface it connects to. We do this with the virsh edit “vm” command, which opens, generally with the default command-line editor, the XML file with the virtual machine configuration. Normally you can look for the “interface type=‘network’” node. Within this definition, there’s another node called “source” with a “network” parameter, which is where we can change and place the name of our network interface created within libvirt.
It’s always advisable to stop and restart the domain once network parameters have been changed, with the virsh shutdown/destroy and virsh start commands, respectively.
Once the virtual network bridge is created, we can make each machine we create, that needs for some reason to be accessible to the local network, or auto-configure with the main router, connect using the bridged-network network. This virtual bridge can also serve us to connect other networks, for example, when configuring LXD, which we’ll talk about on another occasion, as it offers more than interesting features.
Linux Administration Tutorials