Debugging with Record/Replay¶
ovs-replay library provides a set of internal functions for recording
certain events for later replay. This library is integrated into the
stream and some other modules to record all incoming data across all
streams (ssl, tcp, unixctl) of applications based on Open vSwitch libraries
and play these streams later for debugging or performance testing purposes.
Support for this feature is currently integrated into the
ovsdb-client applications. As a result, this allows to record lifecycle
ovsdb-server process in large OVN deployments. Later, by using only
the recorded data, the user can replay transactions and connections that
occurred in a large deployment on their local PC. At the same time it is
possible to tweak various log levels, run a process under debugger or tracer,
measure performance with
perf, and so on.
The current version of record/replay engine does not work correctly with internal time-based events that leats to communications with other processes. For this reason it can not be used with clustered databases (RAFT implementation is heavily time dependent). In addition, recording automatically disables inactivity probes on JSONRPC connections and updates for the Manager status in a _Server database.
High-level feature overview was presented on Open vSwitch and OVN 2020 Fall Conference: Debugging OVSDB with stream record/replay
Recording ovsdb-server events¶
To start recording events for the
ovsdb-server process, there is a special
command line argument
--record. Before starting the database server, make
sure that you have a copy of a database file, so you can use it for replay
later. Here are the general steps to take:
Create a directory where the replay files will be stored:
$ mkdir replay-dir $ REPLAY_DIR=$(pwd)/replay-dir
Copy the current database file for later use:
$ cp my_database $REPLAY_DIR/
ovsdb-serverwith recording enabled:
$ ovsdb-server --record=$REPLAY_DIR <other arguments> my_database
Work with the database as usual.
ovsdb-serverprocess at the end (it is important to send an
exitcommand so that during replay the process will exit in the end too):
$ ovs-appctl -t ovsdb-server exit
$REPLAY_DIR should contain replay files with recorded data.
Replay of recorded session¶
During replay, the
ovsdb-server will receive all the same connections,
transactions and commands as it had at the time of recording, but it will not
create any actual network/socket connections and will not communicate with
any other process. Everything will be read from the replay files.
Since there is no need to wait for IPC, all events will be received one by one
without any delays, so the application will process them as quickly as
possible. This can be used as a performance test where the user can measure how
ovsdb-server can handle some workload recorded in a real
The command line argument to start a replay session is
--replay. The steps
will look like this:
Restore the database file from a previous copy:
$ cp $REPLAY_DIR/my_database my_database
ovsdb-serverwith the same set of arguments as in the recording stage, except for
$ ovsdb-server --replay=$REPLAY_DIR <other arguments> my_database
The process should exit in the end when the
exitcommand is replayed.
On step 2 it is possible to add extra logging arguments to debug some recorded
issue, or run the process under debugger. It’s also possible to replay with
a different version of
ovsdb-server binary as long as this does not affect
the data that goes in and out of the process, e.g. pure performance
The record/replay engine has the following limitations:
Record/Replay of clustered databases is not supported.
Inactivity probes on JSONRPC connections are suppressed.
Manager status updates suppressed in
To remove above limitations, it is necessary to implement correct handling of internally generated time-based events. (possibly by recording of time and subsequent time warping).