DES-TBMS

Motivation

Experimentation in testbed environments is more complicated for researchers than using for example a network simulator. Many challenges, such as node access, correct timing, external influences like interference have to be considered throughout the experiment design, execution, and analysis. Additionally, researchers usually execute experiments manually, a labor-intensive and error-prone task, since there is no reliable automation of the overall experimentation process.

The Distributed Embedded Systems Testbed Management System (DES-TBMS) for the automation of experimentation in  wireless  mesh networks and wireless sensor networks is meant to ease the process of experimentation for the researcher. DES-TBMS consists of several components, each being responsible for specific tasks in the experimentation process. The different tasks include the following:

• Definition of experiment descriptions in a standardized format
• Scheduling and management of experiments
• Automatic execution of experiments
• Distributed monitoring of the network nodes during an experiment
• A visualization tool to display the state of the testbed
• An evaluation tool to compute performance metrics from collected experiment data

Architecture

DES-TBMS consists of five components and four data bases, which have a tight relationship as depicted in the grahpic below. DES-Cript is the domain specific language to define experiments, DES-Exp is the experiment manager which maintains all experiments stored in the Experiment Description data base and is responsible for scheduling, executing, and collecting the log files of an experiment. The raw results of experiments are stored in the Result Data data base. The testbed monitored by DES-Mon during experiments. The Monitoring Data is stored in the Monitoring Data data base. Based on this data the status of the testbed is visualized by DES-Vis. The Result Data and Monitoring Data are used by DES-Eval in order to compute Performance Metrics for a particular experiment as defined in the experiment description.

Components

DES-Exp

DES-Exp provides an experiment manager which handles the tasks to create, schedule, and execute experiments. The actual execution of an experiment is carried out automatically: The researcher specifies the preferred starting time and is notified via email after the execution when the results are available. Therefore, the execution of the experiment can not be influenced by the researchers interactions. This ensures a high degree of reproducibility in case the same experiment is repeated multiple times.

The functionality of DES-Exp is provided by two main applications: A web application realized as a Java Servlet allows the user to create new experiments from the scratch or modify previously defined experiments. These experiments are scheduled considering the users preferred time. This enables to automatically execute experiments at night time, when fewer interferences caused by possible in-parallel operated networks are to be expected.
 

Additionally, an execution unit queries the database at regular intervals to check for any pending experiments. In case such an experiment exists, DES-Exp gathers the associated experiment description file and starts the execution. The execution unit uses an SSH-API for node communication. At the beginning of an experiment, SSH connections are established to all participating nodes, which are used throughout the experiment to configure the nodes and send commands. After the experiment run, all output files generated by action commands are gathered for further processing by DES-Eval. After the post-configuration of the testbed, it is released and further experiments can be carried out.

DES-Mon

The task of DES-Mon is to monitor the state of the testbed during the execution of an experiment. The state of the participating nodes in an experiment is collected by scanning these nodes in a fixed interval as specified in the experiment description file using SNMP. The gathered data of each scan consists of the mesh router state, the state of its WLAN interfaces and the Linux kernel routing table. The latter is used to visualize the connectivity during the experiment with DES-Vis. Using the SNMP-Set functionality, custom settings can be applied to the mesh routers in the configuration phase. DES-Mon is implemented as a multi-threaded Java application and uses the SNMP4J framework.

DES-Eval

The evaluation component DES-Eval parses the output generated by the actions of an experiment and inserts the formatted results into the Performance Metrics data base. Therefore, DES-Eval automates the tedious task of manually gathering all log files and extracting the important information for further analysis. It provides a suite of Python scripts for automatic evaluation of commonly used tools in networking experiments such as ping, ttcp, and iperf. In the experiment description, a particular evaluation script may be assigned to each action to process the generated output. DES-Eval runs the corresponding evaluation script, which processes the output file and provides the formatted results. Based on the total number of replications of an experiment statistical data like average, standard error, variance, and confidence intervals are calculated for the specified performance metrics. In a second step, further scripts can be used to generate standard graphs of the data.

DES-Vis

DES-Vis is a 3D-visualization software based on the JavaView framework. Its main purpose is to display the connectivity of the network during experiment runs. The graphic below shows a screenshot of the connectivity of the wireless mesh network at a particular time. It relies on the kernel routing table information gathered during the network scans by DES-Mon. The changes in connectivity can be animated throughout an entire experiment run.  For this, the corresponding network state snapshots are sorted according to their timestamp and progressed through them. This feature enables researchers to quickly discover explanations for obtained results and simplifies insight into observed phenomena.