Experimentation

The significance of experimentation in wireless network research has gained more attention in the last years. This trend can be traced back to the observation, that results obtained from simulations and testbeds can deviate, since environment parameters of a network scenario, which have a big impact on real network deployments, are usually simplified or not accounted.

In contrast to simulation environments, the set up of a testbed often only comprises the provision of the hardware infrastructure and its maintenance. The users usually access the testbed by directly connecting to the nodes and then have to take care of all steps of the execution of an experiment themselves. Researchers ease this situation by automating small steps of the experimentation process, for example by using simple shell scripts. In consequence, these customized solutions render experiments in a testbed environment harder to repeat and obtained results are more difficult to compare. This gets more complicated when experiments are performed on different testbeds. Few testbeds have addressed this issue by either introducing a custom experiment design and execution framework or by providing support for experiment descriptions as used in simulation environments such as ns-2. Regardless of a particular experiment environment and independent from the research subject, some requirements exist which have to be fulfilled by all researchers and their experiments to ensure meaningful results. We consider the following requirements as the most important each experiment should meet:
 

• Unbiased: An experiment should be as general as possible and should not be trimmed to fit to a particular experiment environment or setting.
• Rigorous: An experiment should only focus on the crucial matter it wants to investigate to reduce side effects.
• Repeatable: An experiment can be executed as often as desired. This is often needed to obtain statistically sound results.
• Reproducible: Results obtained from different executions of an experiment in the same environment should deviate only slightly.
• Statistically sound: The experiment should be evaluated based on mathematical constructs which are appropriate to the experiment.

While some of these requirements depend fully on the user such as the invention and design of meaningful experiments, experiment environments are capable of supporting others. For example, it is rather easy to ensure repeatability by defining a standardized experiment description format which can be executed an arbitrary number of times. A more difficult task is to ensure the reproducibility of results. It becomes almost impossible, if, for example, a testbed is set up in an unshielded environment where uncontrollable other wireless devices and Access Points are operating on the same frequency band. These interferences likely affect the outcome of the experiment.

As next, we describe the typical workflow of networking experiements and then present our two approaches to meet the aforementioned requirements for sound experimentation. We describe the DES-TestBed Management System (TBMS) and a experiment description language (DES-Cript) in detail.