fifo_open - The next big thing

With the _rename_r mostly done and all the things falling in place, the next target was fifo_open routine under the libfs/. At first look, the routine looked too small to be so large in terms of code size and then realized that all the static routines invoked were inlined. Now, fifo_open was mostly fifo.c. This made it a big chunk.

Pipes / fifos are connectors used in a -ix system, which can be used by programs to convey data to other programs. A pipe has two (virtual) ends, write end - where a writer can fill in the data and a read end, from where a reader extracts data. From the implementation point of view, a pipe is a buffer, to which a reader / writer has exclusive access and modify the pipe by writing / reading.

About the routine:

Signature:

int fifo_open(
pipe_control_t **pipep,
rtems_libio_t *iop
);

This routine is called by open -> IMFS_fifo_open -> fifo_open

Open takes in a file name and the mode in which the pipe is to be opened. open creates an fd and provides the IMFS_fifo_open with the rtems_libio_t corresponding to the fd. IMFS_fifo_open extracts info from the rtems_libio_t and provides the (pipe_control_t **) and the (rtems_libio_t *) to fifo_open.

pipep - pipe control structure that is either already present or NULL.
iop - rtems_libio_t which is used to extract useful info like modes of the open

Purpose:

* open a fifo / pipe in the required mode

Misc:

* It is also called during a unnamed pipe creation - a posix pipe creation routine

Flow:

* Create a new pipe, if necessary
- If the pipe control structure does not exist, allocate a new pipe control stucture.
- A new pipe control structure requires
# pipe buffer
# Two barriers, one each for readers and writers
# A semaphore for the producer-consumer solution

* Check the mode in which the pipe is to be opened
* If read mode, wake up the possibly waiting writers
* If a non-blocking read mode is requested, return. Else wait at a barrier
* If write mode, wake up the possibly waiting readers
* If a non-blocking write mode is requested, its not possible and return with an error. Else, wait at a barrier until a reader wakes us up
* If the mode is a read-write mode, register the increase in number of readers/writers and return

Test cases:

With Pipes/fifos disabled in the configuration
* attempt opening a fifo

With Pipes/fifos enabled in the configuration, with a single task,

* Consume all barriers and semaphores, try opening fifo.
* Release a barrier, try opening fifo
* Release another barrier, try opening fifo
* Delete a semaphore, try opening fifo
* Consume all of the memory, try opening fifo
* open fifo in RD-WR mode
* open fifo in RDONLY, nonblock
* open fifo in WRONLY, nonblock
* open fifo several times, to get the wrap around a->z->a

With Pipes/fifos enabled in the configuration, with two tasks,

* open pipe in a read-only mode first, block the reader and then open the pipe in a write-only mode, which unblocks the reader
* open pipe in a write-only mode first, block the writer and then open the pipe in a read-only mode, which unblocks the writer
* write and read from the pipe to be sure about the fifo_open.

Changes to RTEMS, issues with routine:

* Without fifos enabled, fifo_opening was possible. New config item
CONFIGURE_FIFOS(PIPES)_ENABLED and
CONFIGURE_MAXIMUM_FIFOS (PIPES)
created as per which number of semaphore / barriers required are calculated. Fixed.

* when the fifo was opened in WRONLY | NONBLOCK mode, the pipe_release did not destroy the pipe. PR 1542 - Fixed.

* fifo_open returned EINTR where it was supposed to return ENOMEM. PR 1577 - Fixed.

* pipe_release did not delete the semaphore, since it was still in use. PR 1585 - Fixed.

Coverage result:

* fifo_open remains mostly covered, except for the following cases:
- Errors on barrier wait
- Errors on semaphore obtain

Challenges:

* Making my way through the code tree
* Understanding the working of a pipe / fifo
* Understanding task creation, deletion, blocking

Submitted patch:

PR on rtems-bugzilla - PR 1546

https://code.google.com/p/rtems-coverage-improvements-gsoc-2010/source/browse/#svn/trunk/fifo-open-patch

Learnings:

* How to better submit a test
* How to better submit a patch to the RTEMS
* Pipe / fifo related concepts
* Tasks, their creation, deletion, when do they block...
* Semaphore / barrier related concepts
* Get a ENOMEM by allocating most of heap
* How to hit the code, more hard :)

Next:

After this long stretch, thanks to the examinations at the university, next were the lot easier error reporting routines.

The 1st of the rest - _rename_r

This was the first task at hand, before the coding phase of GSoC 2010 begun. This was just to make sure I get hold to things I need to do in the summer. I had to move the numbers up, which would ensure I have all the infrastructure in place and know how to handle things. Fortunately, with support from the mentors and the folks on #rtems, I could do it. And better... I could hit a big chunk in the first shot (Psssst, with a single call).

About the routine:

Signature:

int _rename_r(
struct _reent *ptr __attribute__((unused)),
const char *old,
const char *new
);

old - The path, absolute or relative of the node (file / directory) which needs to be changed

new - The new path, absolute or relative to which the old path is to be renamed. This path needs to be complete. new cannot be an existing directory on to which we are trying to move the file. _rename_r is not mv.

Purpose:

* Rename an existing node as a new node

Misc:

* It is the reentrant version of the rename routine (which I don't see)

Flow:

* Get the old parent location info
* Resolve the old path
* Evaluate the parent of the new path
* Make sure that the rename is within the same filesystem
* Invoke the filesystem's rename routine
* Free the used location info variables
* Return

Test cases:

With IMFS,
* Valid old path and valid new path, both absolute
* Invalid old path, absolute
* Invalid new path, absolute
* Valid old and new paths, both relative
* Mount new filesystem, and attempt to rename across filesystems

Changes to RTEMS, issues with routine:

* The routine returned the errno, instead of the status. PR 1522 - Fixed.
* Few macros were converted to functions under libcsupport/. PR 1541 - Fixed.
* mount system call was changed. PR 1517 - Fixed.

Coverage result:

* _rename_r is now completely covered. Changes to mount call required changes to the test cases and were dealt with (not me)

Challenges:

* The initial setup was hard to get.
* Moving through the code tree to find the routines invoked
* Understanding the data structures used
* Understanding the path evaluation routine
* Understanding the filesystem and mount related concepts

Submitted patch:

PR on rtems-bugzilla - NA.

https://code.google.com/p/rtems-coverage-improvements-gsoc-2010/source/browse/#svn/trunk/_rename_r

Learnings:

* Learnt how to better use emacs. :)
* How path evaluation works, well... most of it.
* Filesystem and mount related concepts.
* _rename_r is not mv of linux
* How to submit a patch (v1).
* How not to submit a patch (v1)
* Destroying the coverage infrastructure and getting it back. :D
* Creating a "PR" and submitting the patch at bugzilla

Next:

fifo/pipe routines

Getting started with Coverage on RTEMS

Code coverage is a form of white box testing, where a piece of code is said to be 100% covered if the whole of the code is executed at least once. Of the several criteria [0], RTEMS uses the object coverage => every instruction (in the assembly code) must be executed.

As [1] indicates, coverage does not prove correctness, but, in my opinion, being vigilant while writing the coverage test cases could help uncovering a lot of bugs.

Profiles:

There are several profiles used in RTEMS. These profiles indicate what is being tested. Is it some part of RTEMS or the whole of it? Do we include POSIX library or not? And so on...

Profiles are formed based the following:
Part of the code considered for coverage: Core / Developmental (includes core)
Optimization level: Optimized (Os), Not optimized (O2)
Posix Enabled / Posix Disabled

Getting the tools in place:

You first need to get a emulator which provides coverage information.

Qemu-Couverture:

I am using Couverture [2] for coverage purpose of pc386 platform.
Couverture can be obtained using

git clone http://forge.open-do.org/anonscm/git/couverture-qemu/couverture-qemu.git

The usual configure, make, make install would complete the installation. Make sure (by changing the $PATH) that 'type qemu' leads to the newly obtained qemu.

Right now, I am using the qemu from [5].

Skyeye:

Skyeye is another one of such coverage enabled emulator. I will be updating this section in a while, after I get a hands on experience on it.

How to "do coverage" on RTEMS?

Get the rtems-testing module from the cvs repository. Down the tree, rtems-coverage/, we see the do_coverage, which is the script used to perform a coverage run.

Before one starts off with the do_coverage, we need to localize the environment, by modifying the VERSIONS-COVERAGE file. Once this is done, you need to choose from one of the profiles mentioned above. The flow performed by do_coverage is as follows:

- Clean up the build and test (executables) directory
- Updating and building the covmerge itself
- Updating RTEMS
- Configuring RTEMS
- Build RTEMS
- Copy tests
- Running the tests (the longest stride) (with coverage details enabled from the simulator)
- Analysing the coverage output from the simulator and generating a unified coverage map for each symbol
- Putting the output in a more readable form

All this in just one step: ./do_coverage -B

Output of coverage and what it means:

- annotated.txt / annotated.html: Annotated source code, with details on each line whether the line was executed or not
- branch.txt / branch.html: Branch always taken / never taken data
- configuration.txt: To indicate the configuration with which the run was performed
- index.html: the index for all the reports/results from the coverage run
- NotReferenced: List of symbols that were never referenced even once (Eyes widen)
- row.html: the results of the run, in a single row, that need to go into the site
- sizes.txt / sizes.html: file to indicate how much amount of a symbol is still uncovered
- summary.txt
- symbolSummary.txt: Details about each symbol, and the uncovered code in that routine
- testsWithNoEndOfTest.txt: Indicates the tests that did not end as expected
- uncovered.txt: details about uncovered code, and corresponding routines / symbols

Approach

Start with a coverage result, either from the rtems site::coverage results [3] or by running the coverage tests for a specific profile

Choose the next set of routines / next routine one needs to work on either by -
- Discussion with mentor / Dr. Joel [4]
Check the sizes and the possibility to write coverage code for the routines
Find a place to put your code
- Either a new test under one of the directories @ testsuites/ (where should you put it? Ask your mentor / make a best guess!)
- Patch an existing test to improve the coverage
- Run coverage
- Check the target code. Is it covered? Any improvement? Other possible tests?
- Patch / introduce new test
and so on...

With this, we now can perform any coverage enabled runs and check coverage status / improvement.

Next up: How did the _rename_r coverage go? Its here.

[0] - http://www.rtems.com/wiki/index.php/Coverage_Analysis_Theory#Object_Coverage
[1] - http://www.rtems.com/wiki/index.php/RTEMS_Coverage_Analysis#Applying_Coverage_Analysis_to_RTEMS
[2] - http://www.open-do.org/projects/couverture/
[3] - http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
[4] - joel.sherrill AT OARcorp.com
[5] - http://www.rtems.org/ftp/private/coverage/