Visualizing flows with ovs-flowviz¶
When troubleshooting networking issues with OVS, it’s common to end up looking at OpenFlow or datapath flow dumps. These dumps tend to be quite dense and difficult to reason about.
ovs-flowviz
is a utility script that helps visualizing OpenFlow and
datapath flows to make it easier to understand what is going on.
The ovs-flowviz(8) manpage describes its basic usage. In this document a few of its advanced visualization formats will be expanded.
Installing ovs-flowviz¶
ovs-flowviz
is part of the openvswitch python package but its
extra dependencies have to be installed explicitly by running:
$ pip install openvswitch[flowviz]
Or, if you are working with the OVS tree:
$ cd python && pip install .[flowviz]
Visualizing OpenFlow logical block¶
When controllers such as OVN write OpenFlow flows, they typically organize flows in functional blocks. These blocks can expand to multiple flows that “look similar”, in the sense that they match on the same fields and have similar actions.
However, looking at a flow dump the number of flows can make it difficult to perceive this logical functionality that the controller is trying to implement using OpenFlow.
ovs-flowviz openflow logic
visualization can be used to understand an OVN
flow dump a bit better.
On a particular flow dump table 0 contains 23 flows:
$ grep -c "table=0" flows.txt
23
Looking at the first few lines, the amount of information can be overwhelming and difficult our analysis:
$ head flows.txt
cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,vlan_tci=0x0000/0x1000 actions=conjunction(100,2/2)
cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,conj_id=100,in_port="patch-br-int-to",vlan_tci=0x0000/0x1000 actions=load:0xa->NXM_NX_REG13[],load:0xc->NXM_NX_REG11[],load:0xb->NXM_NX_REG12[],load:0xb->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:02:42:ac:12:00:03,resubmit(,8)
cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-6bb3b3-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-a6ff98-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
cookie=0xf2ca6195, duration=765.107s, table=0, n_packets=6, n_bytes=636, priority=100,in_port="ovn-k8s-mp0" actions=load:0x1->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
cookie=0x236e941d, duration=408.874s, table=0, n_packets=11, n_bytes=846, priority=100,in_port=aceac9829941d11 actions=load:0x11->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x3->NXM_NX_REG14[],resubmit(,8)
cookie=0x3facf689, duration=405.581s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="363ba22029cd92b" actions=load:0x12->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x4->NXM_NX_REG14[],resubmit(,8)
cookie=0xe7c8c4bb, duration=405.570s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="6a62cde0d50ef44" actions=load:0x13->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x5->NXM_NX_REG14[],resubmit(,8)
cookie=0x99a0ffc1, duration=59.391s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="5ff3bfaaa4eb622" actions=load:0x14->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x6->NXM_NX_REG14[],resubmit(,8)
cookie=0xe1b5c263, duration=59.365s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="8d9e0bc76347e59" actions=load:0x15->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x7->NXM_NX_REG14[],resubmit(,8)
However, table 0 can be better understood by looking at its logical representation:
$ ovs-flowviz -i flows.txt -f "table=0" openflow logic
Ofproto Flows (logical)
└── ** TABLE 0 **
├── priority=180 priority,vlan_tci ---> conjunction ( x 1 )
├── priority=180 priority,conj_id,in_port,vlan_tci ---> load,load,load,load,load,mod_dl_src resubmit(,8), ( x 1 )
├── priority=100 priority,in_port ---> move,move,move resubmit(,40), ( x 2 )
├── priority=100 priority,in_port ---> load,load,load,load,load resubmit(,8), ( x 16 )
├── priority=100 priority,in_port,vlan_tci ---> load,load,load,load,load resubmit(,8), ( x 1 )
├── priority=100 priority,in_port,dl_vlan ---> strip_vlan,load,load,load,load,load resubmit(,8), ( x 1 )
└── priority=0 priority ---> drop, ( x 1 )
In only a few logical blocks, there is a good overview of what this table is doing. It looks like it’s adding metadata based on input ports and vlan IDs and mainly sending traffic to table 8.
A possible next step might be to look at table 8, and in this case, filter out the flows that have not been hit by actual traffic. This is quite easy to do with the arithmetic filtering expressions:
$ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic
Ofproto Flows (logical)
└── ** TABLE 8 **
├── priority=50 priority,reg14,metadata,dl_dst ---> load resubmit(,9), ( x 3 )
└── priority=50 priority,metadata ---> load,move resubmit(,73),resubmit(,9), ( x 2 )
At this point, understanding the output might be difficult without relating it
to the matadata OVN stored in the previous table. This is where
ovs-flowviz
’s OVN integration is useful:
$ export OVN_NB_DB=tcp:172.18.0.4:6641
$ export OVN_SB_DB=tcp:172.18.0.4:6642
$ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic --ovn-detrace
Ofproto Flows (logical)
└── ** TABLE 8 **
├── cookie=0xe10c34ee priority=50 priority,reg14,metadata,dl_dst ---> load resubmit(,9), ( x 1 )
│ └── OVN Info
│ ├── * Logical datapaths:
│ ├── * "ovn_cluster_router" (366e1c41-0f3d-4420-b796-10692b64e3e4)
│ ├── * Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtos-ovn-worker2), actions=(xreg0[0..47] = 0a:58:0a:f4:01:01; next;)
│ └── * Logical Router Port: rtos-ovn-worker2 mac 0a:58:0a:f4:01:01 networks ['10.244.1.1/24'] ipv6_ra_configs {}
├── cookie=0x11e1adbc priority=50 priority,reg14,metadata,dl_dst ---> load resubmit(,9), ( x 1 )
│ └── OVN Info
│ ├── * Logical datapaths:
│ ├── * "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
│ ├── * Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
│ └── * Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
├── cookie=0xf42133f priority=50 priority,reg14,metadata,dl_dst ---> load resubmit(,9), ( x 1 )
│ └── OVN Info
│ ├── * Logical datapaths:
│ ├── * "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
│ ├── * Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.dst == 02:42:ac:12:00:03 && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
│ └── * Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
└── cookie=0x43a0327 priority=50 priority,metadata ---> load,move resubmit(,73),resubmit(,9), ( x 2 )
└── OVN Info
├── * Logical datapaths:
├── * "ovn-worker" (24280d0b-fee0-4f8e-ba4f-036a9b9af921)
├── * "ovn-control-plane" (3262a782-8961-416b-805e-08233e8fda72)
├── * "ext_ovn-worker2" (3f88dcd2-c56d-478f-a3b1-c7aee2efe967)
├── * "ext_ovn-worker" (5facbaf0-485d-4cf5-8940-eff9678ef7bb)
├── * "ext_ovn-control-plane" (8b0aecb6-b05a-48a7-ad09-72524bb91d40)
├── * "join" (e2dc230e-2f2a-4b93-93fa-0fe495163514)
├── * "ovn-worker2" (f7709fbf-d728-4cff-9b9b-150461cc75d2)
└── * Logical flow: table=0 (ls_in_check_port_sec), priority=50, match=(1), actions=(reg0[15] = check_in_port_sec(); next;)
ovs-flowviz
has automatically added the cookie to the logical block key
so more blocks have been printed. In exchange, it has looked up each cookie on
the running OVN databases and inserted the known information on each
block.
The logical flow that generated each OpenFlow flow and the logical datapath it belongs to are now printed, making OVN’s pipeline clearer.
Visualizing datapath flow trees¶
Another typical usecase that can lead to eyestrain is understanding datapath conntrack recirculations.
OVS makes heavy use of connection tracking and the recirc()
action
to build complex datapaths. Typically, OVS will insert a flow that,
when matched, will send the packet through conntrack (using the ct
action)
and recirculate it with a particular recirculation id (recirc_id
). Then,
flows matching on that recirc_id
will be matched and further process the
packet. This can happen more than once for a given packet.
This sequential set of events is, however, difficult to visualize when you look at a datapath flow dump. Flows are unordered so recirculations need to be followed manually (typically, with heavy use of “grep”).
For this use-case, ovs-flowviz datapath tree
format can be extremely
useful. It builds a hierarchical tree based on the recirc_id
, in_port
and recirc()
actions.
Furthermore, it is common to end up with multiple flows that have the same list of actions. An example of this is a number flows that perform mac/vlan checks for a given port and send the traffic though the same conntrack zone. In order to better visualize this and reduce the amount of duplicated flows that are printed in this view, these flows are combined into a block, and the match keys that are equal for all flows are removed.
For example:
Datapath Flows (logical)
└── ╭────────────────────────────────╮
│ [recirc_id(0x0) in_port(eth0)] │
╰────────────────────────────────╯
└── ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=0a:58:0a:84:00:07,dst=22:a1:5d:dc:95:50),eth_type(0x0800),ipv4(src=10.132.0.7,dst=1 │
│ 0.128.0.0/255.128.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:4924, bytes:468961, │
│ recirc_id(0),dp_hash(...),skb_priority(...),in_port(eth0),skb_mark(...),ct_state(...),ct_zone(...),ct_mark(...),ct_label(...),eth(src=0a:58:0a:84:00:07,dst=0a:58:0a:84:00:01),eth_type(......),ipv4(src=10.132.0.7,dst=1 │
│ 0.0.0.0/255.255.128.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=32768/0x8000,dst=0/0), packets:711, bytes:114236, │
│ recirc_id(0),dp_hash(...),skb_priority(...),in_port(eth0),skb_mark(...),ct_state(...),ct_zone(...),ct_mark(...),ct_label(...),eth(src=0a:58:0a:84:00:07,dst=0a:58:0a:84:00:14),eth_type(......),ipv4(src=10.132.0.7,dst=1 │
│ 0.128.0.0/255.128.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=4096/0xf000,dst=0/0), packets:140, bytes:114660, │
│ recirc_id(0),dp_hash(...),skb_priority(...),in_port(eth0),skb_mark(...),ct_state(...),ct_zone(...),ct_mark(...),ct_label(...),eth(src=0a:58:0a:84:00:07,dst=0a:58:0a:84:00:22),eth_type(......),ipv4(src=10.132.0.7,dst=1 │
│ 0.128.0.0/255.128.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:1, bytes:66, │
│ recirc_id(0),dp_hash(...),skb_priority(...),in_port(eth0),skb_mark(...),ct_state(...),ct_zone(...),ct_mark(...),ct_label(...),eth(src=0a:58:0a:84:00:07,dst=0a:58:0a:84:00:09),eth_type(......),ipv4(src=10.132.0.7,dst=1 │
│ 0.128.0.0/255.128.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=4096/0xf000,dst=0/0), packets:0, bytes:0, │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
└── ╭───────────────────────────────────────╮
│ actions: ct(zone=32,nat),recirc(0xc1) │
╰───────────────────────────────────────╯
└── ╭─────────────────────────────────╮
│ [recirc_id(0xc1) in_port(eth0)] │
╰─────────────────────────────────╯
├── ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │ recirc_id(0xc1),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x2a/0x3f),ct_zone(0/0),ct_mark(0/0xf),ct_label(0/0),eth(src=0a:58:0a:84:00:07,dst=22:a1:5d:dc:95:50),eth_type(0x0800),ip │
│ │ v4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:4924, bytes:468961, │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
│ └── ╭───────────────────────────────────────╮
│ │ actions: ct(zone=14,nat),recirc(0xc2) │
│ ╰───────────────────────────────────────╯
│ └── ╭─────────────────────────────────╮
│ │ [recirc_id(0xc2) in_port(eth0)] │
│ ╰─────────────────────────────────╯
│ └── ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │ recirc_id(0xc2),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x2a/0x3f),ct_zone(0/0),ct_mark(0/0x1),ct_label(0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00 │
│ │ :00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:4924, bytes:468961, │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
│ └── ╭──────────────────────╮
│ │ actions: ovn-k8s-mp0 │
│ ╰──────────────────────╯
├── ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │ recirc_id(0xc1),dp_hash(0/0),skb_priority(0/0),in_port(eth0),skb_mark(0/0),ct_state(0x2a/0x3f),ct_zone(0/0),ct_mark(0/0xf),ct_label(0/0),eth(src=0a:58:0a:84:00:07,dst=0a:58:0a:84:00:14),eth_type(0x0800),ip │
│ │ v4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=4096/0xf000,dst=0/0), packets:140, bytes:114660 │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The above shows a part of a bigger tree with an initial block of flows
at recirc_id(0)
which match on different destination Ethernet
addresses and protocols, and send traffic through conntrack (zone 32).
Then some additional flows at recirc_id(0xc1)
process each
connection independently. One of them, shown in the example, sends packets
through conntrack zone 14, and after another recirculation the packet is
ultimately sent through a port.
This is a truly complex multi-zone conntrack pipeline that is now significantly clearer thanks to this visualization.
Also note, the flows in the block are conveniently sorted by sent packets.
This example shows only a single “subtree”. Even though the combination of flows with the same action helps, if we use this command to display a large dump, the output can be verbose. There are two, combinable, mechanisms that can help.
Plotting datapath trees¶
By using the ovs-flowviz datapath html
format, long datapath trees can
be displayed in an interactive HTML table. The resulting web page allows
subtrees to be expanded and collapsed, allowing focus on the desired
information.
The ovs-flowviz datapath graph
format generates a graphviz
graph definition where blocks of flows with the same recirc_id
match
are arranged together, and edges are created to represent recirculations.
This format comes with further features such as displaying the conntrack
zones, which are key to understanding what the datapath is really doing with a
packet.
The html
and graph
can also be combined.
ovs-flowviz datapath graph --html
command will output an interactive
HTML table alongside a SVG graphical representation of the flows. Flows in the
SVG representation link to the corresponding entry in the HTML table.
Filtering¶
As well as allowing expanding and collapsing subtrees, filtering can be used.
However, filtering works in a slightly different way than it does with OpenFlow flows. Instead of just removing non-matching flows, the output of a filtered datapath flow tree will show full sub-trees containing at least one flow that satisfies the filter.
For example, the following command allows understanding the flows in the above
example in the context of traffic going out on port ovn-k8s-mp0
:
$ ovs-appctl dpctl/dump-flows | ovs-flowviz -f "output.port=ovn-k8s-mp0" datapath tree
The resulting flow tree will contain all of the flows above, including those
with recirc_id(0)
and recirc_id(0xc1)
that don’t actually output
traffic to port ovn-k8s-mp0
. This is because they are part of a subtree
that contains flows that output packets on port ovn-k8s-mp0
This provides a “full picture” of how traffic, ending up in a particular port, is being processed.