Getting started with Coverage on RTEMS

Code coverage is a form of white box testing, where a piece of code is said to be 100% covered if the whole of the code is executed at least once. Of the several criteria [0], RTEMS uses the object coverage => every instruction (in the assembly code) must be executed.

As [1] indicates, coverage does not prove correctness, but, in my opinion, being vigilant while writing the coverage test cases could help uncovering a lot of bugs.

Profiles:

There are several profiles used in RTEMS. These profiles indicate what is being tested. Is it some part of RTEMS or the whole of it? Do we include POSIX library or not? And so on...

Profiles are formed based the following:
Part of the code considered for coverage: Core / Developmental (includes core)
Optimization level: Optimized (Os), Not optimized (O2)
Posix Enabled / Posix Disabled

Getting the tools in place:

You first need to get a emulator which provides coverage information.

Qemu-Couverture:

I am using Couverture [2] for coverage purpose of pc386 platform.
Couverture can be obtained using

git clone http://forge.open-do.org/anonscm/git/couverture-qemu/couverture-qemu.git

The usual configure, make, make install would complete the installation. Make sure (by changing the $PATH) that 'type qemu' leads to the newly obtained qemu.

Right now, I am using the qemu from [5].

Skyeye:

Skyeye is another one of such coverage enabled emulator. I will be updating this section in a while, after I get a hands on experience on it.

How to "do coverage" on RTEMS?

Get the rtems-testing module from the cvs repository. Down the tree, rtems-coverage/, we see the do_coverage, which is the script used to perform a coverage run.

Before one starts off with the do_coverage, we need to localize the environment, by modifying the VERSIONS-COVERAGE file. Once this is done, you need to choose from one of the profiles mentioned above. The flow performed by do_coverage is as follows:

- Clean up the build and test (executables) directory
- Updating and building the covmerge itself
- Updating RTEMS
- Configuring RTEMS
- Build RTEMS
- Copy tests
- Running the tests (the longest stride) (with coverage details enabled from the simulator)
- Analysing the coverage output from the simulator and generating a unified coverage map for each symbol
- Putting the output in a more readable form

All this in just one step: ./do_coverage -B

Output of coverage and what it means:

- annotated.txt / annotated.html: Annotated source code, with details on each line whether the line was executed or not
- branch.txt / branch.html: Branch always taken / never taken data
- configuration.txt: To indicate the configuration with which the run was performed
- index.html: the index for all the reports/results from the coverage run
- NotReferenced: List of symbols that were never referenced even once (Eyes widen)
- row.html: the results of the run, in a single row, that need to go into the site
- sizes.txt / sizes.html: file to indicate how much amount of a symbol is still uncovered
- summary.txt
- symbolSummary.txt: Details about each symbol, and the uncovered code in that routine
- testsWithNoEndOfTest.txt: Indicates the tests that did not end as expected
- uncovered.txt: details about uncovered code, and corresponding routines / symbols

Approach

Start with a coverage result, either from the rtems site::coverage results [3] or by running the coverage tests for a specific profile

Choose the next set of routines / next routine one needs to work on either by -
- Discussion with mentor / Dr. Joel [4]
Check the sizes and the possibility to write coverage code for the routines
Find a place to put your code
- Either a new test under one of the directories @ testsuites/ (where should you put it? Ask your mentor / make a best guess!)
- Patch an existing test to improve the coverage
- Run coverage
- Check the target code. Is it covered? Any improvement? Other possible tests?
- Patch / introduce new test
and so on...

With this, we now can perform any coverage enabled runs and check coverage status / improvement.

Next up: How did the _rename_r coverage go? Its here.

[0] - http://www.rtems.com/wiki/index.php/Coverage_Analysis_Theory#Object_Coverage
[1] - http://www.rtems.com/wiki/index.php/RTEMS_Coverage_Analysis#Applying_Coverage_Analysis_to_RTEMS
[2] - http://www.open-do.org/projects/couverture/
[3] - http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
[4] - joel.sherrill AT OARcorp.com
[5] - http://www.rtems.org/ftp/private/coverage/