Connecting VMs Using Tunnels (Userspace)

This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over VXLAN tunnels. Unlike Connecting VMs Using Tunnels, this configuration works entirely in userspace.

Note

This guide covers the steps required to configure VXLAN tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch.

+--------------+
|     vm0      | 192.168.1.1/24
+--------------+
   (vm_port0)
       |
       |
       |
+--------------+
|    br-int    |                                    192.168.1.2/24
+--------------+                                   +--------------+
|    vxlan0    |                                   |    vxlan0    |
+--------------+                                   +--------------+
       |                                                  |
       |                                                  |
       |                                                  |
 172.168.1.1/24                                           |
+--------------+                                          |
|    br-phy    |                                   172.168.1.2/24
+--------------+                                  +---------------+
|  dpdk0/eth1  |----------------------------------|      eth1     |
+--------------+                                  +---------------+
Host A with OVS.                                     Remote host.

Setup

This guide assumes the environment is configured as described below.

Two Physical Hosts

The environment assumes the use of two hosts, named host1 and host2. We only detail the configuration of host1 but a similar configuration can be used for host2. Both hosts should be configured with Open vSwitch (with or without DPDK), QEMU/KVM and suitable VM images. Open vSwitch should be running before proceeding.

Configuration Steps

Perform the following configuration on host1:

  1. Create a br-int bridge:

    $ ovs-vsctl --may-exist add-br br-int \
      -- set Bridge br-int datapath_type=netdev \
      -- br-set-external-id br-int bridge-id br-int \
      -- set bridge br-int fail-mode=standalone
    
  2. Add a port to this bridge. If using tap ports, first boot a VM and then add the port to the bridge:

    $ ovs-vsctl add-port br-int tap0
    

    If using DPDK vhost-user ports, add the port and then boot the VM accordingly, using vm_port0 as the interface name:

    $ ovs-vsctl add-port br-int vm_port0 \
      -- set Interface vm_port0 type=dpdkvhostuserclient \
         options:vhost-server-path=/tmp/vm_port0
    
  3. Configure the IP address of the VM interface in the VM itself:

    $ ip addr add 192.168.1.1/24 dev eth0
    $ ip link set eth0 up
    
  4. On host1, add a port for the VXLAN tunnel:

    $ ovs-vsctl add-port br-int vxlan0 \
      -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.2
    

    Note

    172.168.1.2 is the remote tunnel end point address. On the remote host this will be 172.168.1.1

  5. Create a br-phy bridge:

    $ ovs-vsctl --may-exist add-br br-phy \
        -- set Bridge br-phy datapath_type=netdev \
        -- br-set-external-id br-phy bridge-id br-phy \
        -- set bridge br-phy fail-mode=standalone \
             other_config:hwaddr=<mac address of eth1 interface>
    

    Note

    This additional bridge is required when running Open vSwitch in userspace rather than kernel-based Open vSwitch. The purpose of this bridge is to allow use of the kernel network stack for routing and ARP resolution. The datapath needs to look-up the routing table and ARP table to prepare the tunnel header and transmit data to the output port.

    Note

    eth1 is used rather than eth0. This is to ensure network connectivity is retained.

  6. Attach eth1/dpdk0 to the br-phy bridge.

    If the physical port eth1 is operating as a kernel network interface, run:

    $ ovs-vsctl --timeout 10 add-port br-phy eth1
    $ ip addr add 172.168.1.1/24 dev br-phy
    $ ip link set br-phy up
    $ ip addr flush dev eth1 2>/dev/null
    $ ip link set eth1 up
    $ iptables -F
    

    If instead the interface is a DPDK interface and bound to the igb_uio or vfio driver, run:

    $ ovs-vsctl --timeout 10 add-port br-phy dpdk0 \
      -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:06:00.0
    $ ip addr add 172.168.1.1/24 dev br-phy
    $ ip link set br-phy up
    $ iptables -F
    

    The commands are different as DPDK interfaces are not managed by the kernel, thus, the port details are not visible to any ip commands.

    Important

    Attempting to use the kernel network commands for a DPDK interface will result in a loss of connectivity through eth1. Refer to Basic Configuration for more details.

Once complete, check the cached routes using ovs-appctl command:

$ ovs-appctl ovs/route/show

If the tunnel route is missing, adding it now:

$ ovs-appctl ovs/route/add 172.168.1.1/24 br-phy

Repeat these steps if necessary for host2, but using the below commands for the VM interface IP address:

$ ip addr add 192.168.1.2/24 dev eth0
$ ip link set eth0 up

And the below command for the the host2 VXLAN tunnel:

$ ovs-vsctl add-port br-int vxlan0 \
  -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.1

Testing

With this setup, ping to VXLAN target device (192.168.1.2) should work. Traffic will be VXLAN encapsulated and sent over the eth1/dpdk0 interface.