RTEMS - GSoC-2010 and years to come: May 2010

Monday, May 10, 2010

Getting started with Coverage on RTEMS

Code coverage is a form of white box testing, where a piece of code is said to be 100% covered if the whole of the code is executed at least once. Of the several criteria [0], RTEMS uses the object coverage => every instruction (in the assembly code) must be executed.

As [1] indicates, coverage does not prove correctness, but, in my opinion, being vigilant while writing the coverage test cases could help uncovering a lot of bugs.

Profiles:

There are several profiles used in RTEMS. These profiles indicate what is being tested. Is it some part of RTEMS or the whole of it? Do we include POSIX library or not? And so on...

Profiles are formed based the following:
Part of the code considered for coverage: Core / Developmental (includes core)
Optimization level: Optimized (Os), Not optimized (O2)
Posix Enabled / Posix Disabled

Getting the tools in place:

You first need to get a emulator which provides coverage information.

Qemu-Couverture:

I am using Couverture [2] for coverage purpose of pc386 platform.
Couverture can be obtained usinggit clone http://forge.open-do.org/anonscm/git/couverture-qemu/couverture-qemu.git

The usual configure, make, make install would complete the installation. Make sure (by changing the $PATH) that 'type qemu' leads to the newly obtained qemu.

Right now, I am using the qemu from [5].

Skyeye:

Skyeye is another one of such coverage enabled emulator. I will be updating this section in a while, after I get a hands on experience on it.

How to "do coverage" on RTEMS?

Get the rtems-testing module from the cvs repository. Down the tree, rtems-coverage/, we see the do_coverage, which is the script used to perform a coverage run.

Before one starts off with the do_coverage, we need to localize the environment, by modifying the VERSIONS-COVERAGE file. Once this is done, you need to choose from one of the profiles mentioned above. The flow performed by do_coverage is as follows:

- Clean up the build and test (executables) directory
- Updating and building the covmerge itself
- Updating RTEMS
- Configuring RTEMS
- Build RTEMS
- Copy tests
- Running the tests (the longest stride) (with coverage details enabled from the simulator)
- Analysing the coverage output from the simulator and generating a unified coverage map for each symbol
- Putting the output in a more readable form

All this in just one step: ./do_coverage -B

Output of coverage and what it means:

- annotated.txt / annotated.html: Annotated source code, with details on each line whether the line was executed or not
- branch.txt / branch.html: Branch always taken / never taken data
- configuration.txt: To indicate the configuration with which the run was performed
- index.html: the index for all the reports/results from the coverage run
- NotReferenced: List of symbols that were never referenced even once (Eyes widen)
- row.html: the results of the run, in a single row, that need to go into the site
- sizes.txt / sizes.html: file to indicate how much amount of a symbol is still uncovered
- summary.txt
- symbolSummary.txt: Details about each symbol, and the uncovered code in that routine
- testsWithNoEndOfTest.txt: Indicates the tests that did not end as expected
- uncovered.txt: details about uncovered code, and corresponding routines / symbols

Approach

Start with a coverage result, either from the rtems site::coverage results [3] or by running the coverage tests for a specific profile

Choose the next set of routines / next routine one needs to work on either by -
- Discussion with mentor / Dr. Joel [4]
Check the sizes and the possibility to write coverage code for the routines
Find a place to put your code
- Either a new test under one of the directories @ testsuites/ (where should you put it? Ask your mentor / make a best guess!)
- Patch an existing test to improve the coverage
- Run coverage
- Check the target code. Is it covered? Any improvement? Other possible tests?
- Patch / introduce new test
and so on...

With this, we now can perform any coverage enabled runs and check coverage status / improvement.

Next up: How did the _rename_r coverage go? Its here.

[0] - http://www.rtems.com/wiki/index.php/Coverage_Analysis_Theory#Object_Coverage
[1] - http://www.rtems.com/wiki/index.php/RTEMS_Coverage_Analysis#Applying_Coverage_Analysis_to_RTEMS
[2] - http://www.open-do.org/projects/couverture/
[3] - http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
[4] - joel.sherrill AT OARcorp.com
[5] - http://www.rtems.org/ftp/private/coverage/

Sunday, May 9, 2010

Getting started with RTEMS

Here are some notes to help (anybody and me) getting started with RTEMS. In case you get stuck, the best place to ask for help would be the IRC - #rtems on freenode

Setting up the development environment.

The best start would be to see something working. This requires setting the development environment and running the "Hello World" application consequently.

To have a development environment up and running you would need

A development PC, running on Linux (preferably an RPM based system. Eg: Fedora)
Or you could use vmware [0] or virtualbox [1] with fedora from [2] or [3] on it
If you are comfortable with building tool chains from source, you could manage with Ubuntu (debian based) too.
Internet connectivity

I would best guide you to relevant links on the web and fill in the missing gaps, if any instead of putting down the whole thing here. (Documentation reuse, like code reuse :) )

1. Building the toolchain

If you are keen on working on Debian based systems, see [4]
If you are using rpm based systems, [5] (Damn easy, I tell you)

Get the file: rtems-X.XX-yum-conf-.YY.noarch.rpm (XX - RTEMS version, YY - OS version). Eg: I got the rpm at [6]
rpm -ivh above_file
Get the file : rtems-X.XX-release-.YY.noarch.rpm (XX - RTEMS version, YY - OS version). Eg: I got the rpm at [7]
rpm -ivh above_file
yum install rtems-4.9-auto*
yum install rtems-4.9-* ; Eg: yum install rtems-4.9-sparc* would install the required toolchain for i386 platform

With this, you are done with installing the toolchain required for a particular platform. With the above steps, the binaries were in /opt/rtems-4.10/bin/

extend the $PATH variable to include the path where all the rtems-specific compilers, gdbs and other binaries are present, which in this case would be /opt/rtems-4.10/bin

2. Getting RTEMS:

Have a look at [8]
Use cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS login ; which uses the required CVSROOT and tries to log you in. If prompted for a password, press return (no password)
Get rtems by issuing cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P rtems
To get a particular branch, use cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -r rtems-x-y-branch rtems

Once the above steps are done and successful, you will have a directory in the location where you issued the check out command called rtems.

3. Building rtems:

Follow [9]
Heads up: One MUST build RTEMS for the same platform you got / built the toolchain for. Eg: Both RTEMS (below) and the toolchain have to be i386 specific or sparc specific or arm specific or

Some notes about the build steps:

I started with rtems-4.10 and it requires >autoconf-2.64. When I was building, Fedora had autoconf-2.63 and I had to get 2.64. So make sure you have 2.64
Firstly, while issuing the configure, you have to the target using --target=XXX, enable some corresponding BSP using --enable-rtemsbsp=YYY and the location where RTEMS needs to be installed using --prefix=. The XXX and YYY are best obtained from the README.configure in the rtems root directory
make
make install (might require su rights, depending on install path)

Eg: My configure looked like:
../configure --target=sparc-rtems4.10 --enable-rtemsbsp=sis --prefix=/opt/rtems-4.10/rtems/

Getting example applications:

You can get the example applications, by checking out the examples directory from CVS by cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P examples

Compiling applications:

Reference [10]
move to the directory hello_world_c
before you make, you need to set the RTEMS_MAKEFILE_PATH as per [10]
issue make there

Eg: With the above configuration, the RTEMS_MAKEFILE_PATH needs to be set to /opt/rtems-4.10/rtems/sparc-rtems4.10/sis/

Getting ready for the 1st run:

Of the several possible bsps/platforms, I am putting down two of them below which I have had tried: sparc and pc386

Lets start of with the easier one, which uses a simulator with gdb

1. sparc

make your that the $PATH variable now can reach the rtems-specific compilers, gdbs and other binaries
Run the sparc gdb: sparc-rtems4.10-gdb o-optimize/hello.exe
set target as simulator, load the program and run; tar sim; start; load o-optimize/hello.exe; start
You should now be seeing the output of hello.exe

So, what you did till now was running the test.c, compiled for sparc platform on a sparc simulator using the sparc specific gdb, that came along with the toolchain you installed

2. pc386 using qemu

Make sure you have build the rtems for i386 / pc386. Eg:
You now need qemu, a simulator, which simulates many platforms [11]. We will be using it to simulate the pc386 for us; yum install qemu should do the trick
Download the images [12] that qemu provides and test if the qemu installation works well
To quickly use and test qemu, [13]
Get the gcc-testing folder from the rtems site (CVS access): cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P gcc-testing
Lets consider a folder (call it root)
Get the rtems-boot.img from [14] and rtems-grub.cfg from [15] and place it in this directory
Edit rtems-grub.cfg to have valid boot entries
Also place the executable build for pc386 in this directory
Invoke qemu while in this directory as qemu -boot a -fda rtems-boot.img -hda fat :. (boot from floppy(a), use current directory as hda, with these make sure your rtems-grub.cfg has the valid root as (hd0,0) and the valid path for the executables (hd0,0)/

Eg:
Have hello.exe you built for i386, rtems-boot.img and rtems-grub.cfg in a directory
rtems-grub.cfg would look like
set default=0

menuentry "RTEMS - Hello" {
set root=(hd0,0)
multiboot (hd0,0)/hello.exe
}
Use qemu -boot a -fda rtems-boot.img -hda fat:. and you should see the hello.exe output

3. An extension: Using gdb with qemu

Here is a small, but powerful extension to the above method (with qemu) where one can debug programs for pc386 using the i386-specific gdb

Reference: [17]
All the steps are the same as for the above method, except for the following:

Invoke qemu with the additional switch -s (meaning wait for a gdb connection on localhost:1234)
Invoke i386-rtems4.10-gdb
On gdb prompt, connect to remote target, as tar remote localhost:1234
On this one can put breakpoints, watch registers and all the other debugging possibilities

4. Using the ~~gcc-testing~~ rtems-testing module:

Another way to get things working is what I call "the easy Dr.Joel" way where all is ready, put things in place and bamm-you are good to go.

Reference: [16] - works like magic...!

Notes:

In the sim-scripts/pc386, you might want to change the shell you want to use. I had to change it to /bin/bash

Also, to see some "output" in front of your eyes, just change the way qemu is invoked to something similar to what was done in the previous method

Another method to see things running is using the qemu-nic in the qemu-support directory.

You need to have the ~/qemu and the pc386_fda in that directory. Also the ~/qemu/hd should be present
Copy the executable built for i386 into ~/qemu/hd as test.exe
In qemu-nic, change the QEMUDIR to $HOME
Invoke qemu-nic
You should now be seeing the output of hello.exe

Kick-off

Business, straight away!

Let us try to answer a few obvious questions, if this is the first time you are hearing the words RTEMS or GSoC.

RTEMS: In short, RTEMS is an real-time operating system supporting several open API and interface standards. Multi-tasking, the POSIX 1003.1b API standard, several networking standards are a few among several other supported features. To know more about RTEMS and to have a look at its exhaustive feature set, I would direct you to [0] and [1].

GSoC: Google Summer of Code, in short GSoC is what I would call an opportunity for students worldwide (almost) to get engaged with the world of open source software during the summer and also get paid (bonus!). You could have a look at [2] and for people who do not prefer reading text, [3] or [4].

This summer (2010), I have got the opportunity to work with RTEMS in the gsoc '10 program. A brief about the project and planned schedule is available at [5].

Till now,

- Met a lot of interesting people @ #rtems
- Some hands on RTEMS, thanks to the guided execution of "Hello World"
- Many iterations of change-review of the proposal
- Submission of proposal
- Some more work with RTEMS - Coverage analysis and attempts to improve coverage figures. These were partly successful, but no patches submitted yet.

Next,

- Submission of patch (hopefully the first one into RTEMS repo)
- Continuous interaction with the community and mentors via IRC / mailing list
- Submission of documents to google [6]
- Deciding the next chunk of code to be covered

Next up: Getting started with RTEMS

Links and References: Accessed on 8th May, 2010

[0] - www.rtems.org
[1] - http://www.rtems.org/wiki/index.php/RTEMSFeatures
[2] - http://code.google.com/soc/
[3] - http://www.youtube.com/watch?v=vBRRR0BQyz0&feature=player_embedded#!
[4] - http://www.youtube.com/watch?v=f-hYJAfovm8&feature=channel
[5] - http://sites.google.com/site/bharathswiki/home/actualstuff/from-the-world-of-foss/google-summer-of-code/RTEMS-GSoC2010-Proposal-Bharath-TestCoverage.pdf?attredirects=0&d=1
[6] - http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2010/faqs#student_docs

RTEMS - GSoC-2010 and years to come

Monday, May 10, 2010

Getting started with Coverage on RTEMS

Sunday, May 9, 2010

Getting started with RTEMS

Kick-off

About Me

Followers

Blog Archive