Connecting VMs Using Tunnels (Userspace)¶
This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over VXLAN tunnels. Unlike Connecting VMs Using Tunnels, this configuration works entirely in userspace.
Note
This guide covers the steps required to configure VXLAN tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch.
+--------------+
| vm0 | 192.168.1.1/24
+--------------+
(vm_port0)
|
|
|
+--------------+
| br-int | 192.168.1.2/24
+--------------+ +--------------+
| vxlan0 | | vxlan0 |
+--------------+ +--------------+
| |
| |
| |
172.168.1.1/24 |
+--------------+ |
| br-phy | 172.168.1.2/24
+--------------+ +---------------+
| dpdk0/eth1 |----------------------------------| eth1 |
+--------------+ +---------------+
Host A with OVS. Remote host.
Setup¶
This guide assumes the environment is configured as described below.
Two Physical Hosts¶
The environment assumes the use of two hosts, named host1 and host2. We only detail the configuration of host1 but a similar configuration can be used for host2. Both hosts should be configured with Open vSwitch (with or without DPDK), QEMU/KVM and suitable VM images. Open vSwitch should be running before proceeding.
Configuration Steps¶
Perform the following configuration on host1:
Create a
br-int
bridge:$ ovs-vsctl --may-exist add-br br-int \ -- set Bridge br-int datapath_type=netdev \ -- br-set-external-id br-int bridge-id br-int \ -- set bridge br-int fail-mode=standalone
Add a port to this bridge. If using tap ports, first boot a VM and then add the port to the bridge:
$ ovs-vsctl add-port br-int tap0
If using DPDK vhost-user ports, add the port and then boot the VM accordingly, using
vm_port0
as the interface name:$ ovs-vsctl add-port br-int vm_port0 \ -- set Interface vm_port0 type=dpdkvhostuserclient \ options:vhost-server-path=/tmp/vm_port0
Configure the IP address of the VM interface in the VM itself:
$ ip addr add 192.168.1.1/24 dev eth0 $ ip link set eth0 up
On host1, add a port for the VXLAN tunnel:
$ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.2
Note
172.168.1.2
is the remote tunnel end point address. On the remote host this will be172.168.1.1
Create a
br-phy
bridge:$ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr=<mac address of eth1 interface>
Note
This additional bridge is required when running Open vSwitch in userspace rather than kernel-based Open vSwitch. The purpose of this bridge is to allow use of the kernel network stack for routing and ARP resolution. The datapath needs to look-up the routing table and ARP table to prepare the tunnel header and transmit data to the output port.
Note
eth1
is used rather thaneth0
. This is to ensure network connectivity is retained.Attach
eth1
/dpdk0
to thebr-phy
bridge.If the physical port
eth1
is operating as a kernel network interface, run:$ ovs-vsctl --timeout 10 add-port br-phy eth1 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ ip addr flush dev eth1 2>/dev/null $ ip link set eth1 up $ iptables -F
If instead the interface is a DPDK interface and bound to the
igb_uio
orvfio
driver, run:$ ovs-vsctl --timeout 10 add-port br-phy dpdk0 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:06:00.0 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ iptables -F
The commands are different as DPDK interfaces are not managed by the kernel, thus, the port details are not visible to any
ip
commands.Important
Attempting to use the kernel network commands for a DPDK interface will result in a loss of connectivity through
eth1
. Refer to Basic Configuration for more details.
Once complete, check the cached routes using ovs-appctl command:
$ ovs-appctl ovs/route/show
If the tunnel route is missing, adding it now:
$ ovs-appctl ovs/route/add 172.168.1.1/24 br-phy
Repeat these steps if necessary for host2, but using the below commands for the VM interface IP address:
$ ip addr add 192.168.1.2/24 dev eth0
$ ip link set eth0 up
And the below command for the the host2 VXLAN tunnel:
$ ovs-vsctl add-port br-int vxlan0 \
-- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.1
Testing¶
With this setup, ping to VXLAN target device (192.168.1.2
) should work.
Traffic will be VXLAN encapsulated and sent over the eth1
/dpdk0
interface.