Debugging with Record/Replay¶
The ovs-replay
library provides a set of internal functions for recording
certain events for later replay. This library is integrated into the
stream
and some other modules to record all incoming data across all
streams (ssl, tcp, unixctl) of applications based on Open vSwitch libraries
and play these streams later for debugging or performance testing purposes.
Support for this feature is currently integrated into the ovsdb-server
and
ovsdb-client
applications. As a result, this allows to record lifecycle
of the ovsdb-server
process in large OVN deployments. Later, by using only
the recorded data, the user can replay transactions and connections that
occurred in a large deployment on their local PC. At the same time it is
possible to tweak various log levels, run a process under debugger or tracer,
measure performance with perf
, and so on.
Note
The current version of record/replay engine does not work correctly with internal time-based events that lead to communications with other processes. For this reason it can not be used with clustered databases (RAFT implementation is heavily time dependent). In addition, recording automatically disables inactivity probes on JSONRPC connections and updates for the Manager status in a _Server database.
High-level feature overview was presented on Open vSwitch and OVN 2020 Fall Conference: Debugging OVSDB with stream record/replay
Recording ovsdb-server events¶
To start recording events for the ovsdb-server
process, there is a special
command line argument --record
. Before starting the database server, make
sure that you have a copy of a database file, so you can use it for replay
later. Here are the general steps to take:
Create a directory where the replay files will be stored:
$ mkdir replay-dir $ REPLAY_DIR=$(pwd)/replay-dir
Copy the current database file for later use:
$ cp my_database $REPLAY_DIR/
Run
ovsdb-server
with recording enabled:$ ovsdb-server --record=$REPLAY_DIR <other arguments> my_database
Work with the database as usual.
Stop the
ovsdb-server
process at the end (it is important to send anexit
command so that during replay the process will exit in the end too):$ ovs-appctl -t ovsdb-server exit
After that $REPLAY_DIR
should contain replay files with recorded data.
Replay of recorded session¶
During replay, the ovsdb-server
will receive all the same connections,
transactions and commands as it had at the time of recording, but it will not
create any actual network/socket connections and will not communicate with
any other process. Everything will be read from the replay files.
Since there is no need to wait for IPC, all events will be received one by one
without any delays, so the application will process them as quickly as
possible. This can be used as a performance test where the user can measure how
quickly the ovsdb-server
can handle some workload recorded in a real
deployment.
The command line argument to start a replay session is --replay
. The steps
will look like this:
Restore the database file from a previous copy:
$ cp $REPLAY_DIR/my_database my_database
Start
ovsdb-server
with the same set of arguments as in the recording stage, except for--record
:$ ovsdb-server --replay=$REPLAY_DIR <other arguments> my_database
The process should exit in the end when the
exit
command is replayed.
On step 2 it is possible to add extra logging arguments to debug some recorded
issue, or run the process under debugger. It’s also possible to replay with
a different version of ovsdb-server
binary as long as this does not affect
the data that goes in and out of the process, e.g. pure performance
optimizations.
Limitations¶
The record/replay engine has the following limitations:
Record/Replay of clustered databases is not supported.
Inactivity probes on JSONRPC connections are suppressed.
Manager status updates suppressed in
ovsdb-server
.
To remove above limitations, it is necessary to implement correct handling of internally generated time-based events. (possibly by recording of time and subsequent time warping).