fifo_open - The next big thing

With the _rename_r mostly done and all the things falling in place, the next target was fifo_open routine under the libfs/. At first look, the routine looked too small to be so large in terms of code size and then realized that all the static routines invoked were inlined. Now, fifo_open was mostly fifo.c. This made it a big chunk.

Pipes / fifos are connectors used in a -ix system, which can be used by programs to convey data to other programs. A pipe has two (virtual) ends, write end - where a writer can fill in the data and a read end, from where a reader extracts data. From the implementation point of view, a pipe is a buffer, to which a reader / writer has exclusive access and modify the pipe by writing / reading.

About the routine:

Signature:

int fifo_open(
pipe_control_t **pipep,
rtems_libio_t *iop
);

This routine is called by open -> IMFS_fifo_open -> fifo_open

Open takes in a file name and the mode in which the pipe is to be opened. open creates an fd and provides the IMFS_fifo_open with the rtems_libio_t corresponding to the fd. IMFS_fifo_open extracts info from the rtems_libio_t and provides the (pipe_control_t **) and the (rtems_libio_t *) to fifo_open.

pipep - pipe control structure that is either already present or NULL.
iop - rtems_libio_t which is used to extract useful info like modes of the open

Purpose:

* open a fifo / pipe in the required mode

Misc:

* It is also called during a unnamed pipe creation - a posix pipe creation routine

Flow:

* Create a new pipe, if necessary
- If the pipe control structure does not exist, allocate a new pipe control stucture.
- A new pipe control structure requires
# pipe buffer
# Two barriers, one each for readers and writers
# A semaphore for the producer-consumer solution

* Check the mode in which the pipe is to be opened
* If read mode, wake up the possibly waiting writers
* If a non-blocking read mode is requested, return. Else wait at a barrier
* If write mode, wake up the possibly waiting readers
* If a non-blocking write mode is requested, its not possible and return with an error. Else, wait at a barrier until a reader wakes us up
* If the mode is a read-write mode, register the increase in number of readers/writers and return

Test cases:

With Pipes/fifos disabled in the configuration
* attempt opening a fifo

With Pipes/fifos enabled in the configuration, with a single task,

* Consume all barriers and semaphores, try opening fifo.
* Release a barrier, try opening fifo
* Release another barrier, try opening fifo
* Delete a semaphore, try opening fifo
* Consume all of the memory, try opening fifo
* open fifo in RD-WR mode
* open fifo in RDONLY, nonblock
* open fifo in WRONLY, nonblock
* open fifo several times, to get the wrap around a->z->a

With Pipes/fifos enabled in the configuration, with two tasks,

* open pipe in a read-only mode first, block the reader and then open the pipe in a write-only mode, which unblocks the reader
* open pipe in a write-only mode first, block the writer and then open the pipe in a read-only mode, which unblocks the writer
* write and read from the pipe to be sure about the fifo_open.

Changes to RTEMS, issues with routine:

* Without fifos enabled, fifo_opening was possible. New config item
CONFIGURE_FIFOS(PIPES)_ENABLED and
CONFIGURE_MAXIMUM_FIFOS (PIPES)
created as per which number of semaphore / barriers required are calculated. Fixed.

* when the fifo was opened in WRONLY | NONBLOCK mode, the pipe_release did not destroy the pipe. PR 1542 - Fixed.

* fifo_open returned EINTR where it was supposed to return ENOMEM. PR 1577 - Fixed.

* pipe_release did not delete the semaphore, since it was still in use. PR 1585 - Fixed.

Coverage result:

* fifo_open remains mostly covered, except for the following cases:
- Errors on barrier wait
- Errors on semaphore obtain

Challenges:

* Making my way through the code tree
* Understanding the working of a pipe / fifo
* Understanding task creation, deletion, blocking

Submitted patch:

PR on rtems-bugzilla - PR 1546

https://code.google.com/p/rtems-coverage-improvements-gsoc-2010/source/browse/#svn/trunk/fifo-open-patch

Learnings:

* How to better submit a test
* How to better submit a patch to the RTEMS
* Pipe / fifo related concepts
* Tasks, their creation, deletion, when do they block...
* Semaphore / barrier related concepts
* Get a ENOMEM by allocating most of heap
* How to hit the code, more hard :)

Next:

After this long stretch, thanks to the examinations at the university, next were the lot easier error reporting routines.

The 1st of the rest - _rename_r

This was the first task at hand, before the coding phase of GSoC 2010 begun. This was just to make sure I get hold to things I need to do in the summer. I had to move the numbers up, which would ensure I have all the infrastructure in place and know how to handle things. Fortunately, with support from the mentors and the folks on #rtems, I could do it. And better... I could hit a big chunk in the first shot (Psssst, with a single call).

About the routine:

Signature:

int _rename_r(
struct _reent *ptr __attribute__((unused)),
const char *old,
const char *new
);

old - The path, absolute or relative of the node (file / directory) which needs to be changed

new - The new path, absolute or relative to which the old path is to be renamed. This path needs to be complete. new cannot be an existing directory on to which we are trying to move the file. _rename_r is not mv.

Purpose:

* Rename an existing node as a new node

Misc:

* It is the reentrant version of the rename routine (which I don't see)

Flow:

* Get the old parent location info
* Resolve the old path
* Evaluate the parent of the new path
* Make sure that the rename is within the same filesystem
* Invoke the filesystem's rename routine
* Free the used location info variables
* Return

Test cases:

With IMFS,
* Valid old path and valid new path, both absolute
* Invalid old path, absolute
* Invalid new path, absolute
* Valid old and new paths, both relative
* Mount new filesystem, and attempt to rename across filesystems

Changes to RTEMS, issues with routine:

* The routine returned the errno, instead of the status. PR 1522 - Fixed.
* Few macros were converted to functions under libcsupport/. PR 1541 - Fixed.
* mount system call was changed. PR 1517 - Fixed.

Coverage result:

* _rename_r is now completely covered. Changes to mount call required changes to the test cases and were dealt with (not me)

Challenges:

* The initial setup was hard to get.
* Moving through the code tree to find the routines invoked
* Understanding the data structures used
* Understanding the path evaluation routine
* Understanding the filesystem and mount related concepts

Submitted patch:

PR on rtems-bugzilla - NA.

https://code.google.com/p/rtems-coverage-improvements-gsoc-2010/source/browse/#svn/trunk/_rename_r

Learnings:

* Learnt how to better use emacs. :)
* How path evaluation works, well... most of it.
* Filesystem and mount related concepts.
* _rename_r is not mv of linux
* How to submit a patch (v1).
* How not to submit a patch (v1)
* Destroying the coverage infrastructure and getting it back. :D
* Creating a "PR" and submitting the patch at bugzilla

Next:

fifo/pipe routines

Getting started with Coverage on RTEMS

Code coverage is a form of white box testing, where a piece of code is said to be 100% covered if the whole of the code is executed at least once. Of the several criteria [0], RTEMS uses the object coverage => every instruction (in the assembly code) must be executed.

As [1] indicates, coverage does not prove correctness, but, in my opinion, being vigilant while writing the coverage test cases could help uncovering a lot of bugs.

Profiles:

There are several profiles used in RTEMS. These profiles indicate what is being tested. Is it some part of RTEMS or the whole of it? Do we include POSIX library or not? And so on...

Profiles are formed based the following:
Part of the code considered for coverage: Core / Developmental (includes core)
Optimization level: Optimized (Os), Not optimized (O2)
Posix Enabled / Posix Disabled

Getting the tools in place:

You first need to get a emulator which provides coverage information.

Qemu-Couverture:

I am using Couverture [2] for coverage purpose of pc386 platform.
Couverture can be obtained using

git clone http://forge.open-do.org/anonscm/git/couverture-qemu/couverture-qemu.git

The usual configure, make, make install would complete the installation. Make sure (by changing the $PATH) that 'type qemu' leads to the newly obtained qemu.

Right now, I am using the qemu from [5].

Skyeye:

Skyeye is another one of such coverage enabled emulator. I will be updating this section in a while, after I get a hands on experience on it.

How to "do coverage" on RTEMS?

Get the rtems-testing module from the cvs repository. Down the tree, rtems-coverage/, we see the do_coverage, which is the script used to perform a coverage run.

Before one starts off with the do_coverage, we need to localize the environment, by modifying the VERSIONS-COVERAGE file. Once this is done, you need to choose from one of the profiles mentioned above. The flow performed by do_coverage is as follows:

- Clean up the build and test (executables) directory
- Updating and building the covmerge itself
- Updating RTEMS
- Configuring RTEMS
- Build RTEMS
- Copy tests
- Running the tests (the longest stride) (with coverage details enabled from the simulator)
- Analysing the coverage output from the simulator and generating a unified coverage map for each symbol
- Putting the output in a more readable form

All this in just one step: ./do_coverage -B

Output of coverage and what it means:

- annotated.txt / annotated.html: Annotated source code, with details on each line whether the line was executed or not
- branch.txt / branch.html: Branch always taken / never taken data
- configuration.txt: To indicate the configuration with which the run was performed
- index.html: the index for all the reports/results from the coverage run
- NotReferenced: List of symbols that were never referenced even once (Eyes widen)
- row.html: the results of the run, in a single row, that need to go into the site
- sizes.txt / sizes.html: file to indicate how much amount of a symbol is still uncovered
- summary.txt
- symbolSummary.txt: Details about each symbol, and the uncovered code in that routine
- testsWithNoEndOfTest.txt: Indicates the tests that did not end as expected
- uncovered.txt: details about uncovered code, and corresponding routines / symbols

Approach

Start with a coverage result, either from the rtems site::coverage results [3] or by running the coverage tests for a specific profile

Choose the next set of routines / next routine one needs to work on either by -
- Discussion with mentor / Dr. Joel [4]
Check the sizes and the possibility to write coverage code for the routines
Find a place to put your code
- Either a new test under one of the directories @ testsuites/ (where should you put it? Ask your mentor / make a best guess!)
- Patch an existing test to improve the coverage
- Run coverage
- Check the target code. Is it covered? Any improvement? Other possible tests?
- Patch / introduce new test
and so on...

With this, we now can perform any coverage enabled runs and check coverage status / improvement.

Next up: How did the _rename_r coverage go? Its here.

[0] - http://www.rtems.com/wiki/index.php/Coverage_Analysis_Theory#Object_Coverage
[1] - http://www.rtems.com/wiki/index.php/RTEMS_Coverage_Analysis#Applying_Coverage_Analysis_to_RTEMS
[2] - http://www.open-do.org/projects/couverture/
[3] - http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/
[4] - joel.sherrill AT OARcorp.com
[5] - http://www.rtems.org/ftp/private/coverage/

Getting started with RTEMS

Here are some notes to help (anybody and me) getting started with RTEMS. In case you get stuck, the best place to ask for help would be the IRC - #rtems on freenode

Setting up the development environment.

The best start would be to see something working. This requires setting the development environment and running the "Hello World" application consequently.

To have a development environment up and running you would need

• A development PC, running on Linux (preferably an RPM based system. Eg: Fedora)
• Or you could use vmware [0] or virtualbox [1] with fedora from [2] or [3] on it
• If you are comfortable with building tool chains from source, you could manage with Ubuntu (debian based) too.
• Internet connectivity

I would best guide you to relevant links on the web and fill in the missing gaps, if any instead of putting down the whole thing here. (Documentation reuse, like code reuse :) )

1. Building the toolchain

If you are keen on working on Debian based systems, see [4]
If you are using rpm based systems, [5] (Damn easy, I tell you)

• Get the file: rtems-X.XX-yum-conf-.YY.noarch.rpm (XX - RTEMS version, YY - OS version). Eg: I got the rpm at [6]
• rpm -ivh above_file
  
• Get the file : rtems-X.XX-release-.YY.noarch.rpm (XX - RTEMS version, YY - OS version). Eg: I got the rpm at [7]
• rpm -ivh above_file
• yum install rtems-4.9-auto*
• yum install rtems-4.9-* ; Eg: yum install rtems-4.9-sparc* would install the required toolchain for i386 platform

With this, you are done with installing the toolchain required for a particular platform. With the above steps, the binaries were in /opt/rtems-4.10/bin/

• extend the $PATH variable to include the path where all the rtems-specific compilers, gdbs and other binaries are present, which in this case would be /opt/rtems-4.10/bin
  

2. Getting RTEMS:

• Have a look at [8]
• Use cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS login ; which uses the required CVSROOT and tries to log you in. If prompted for a password, press return (no password)
• Get rtems by issuing cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P rtems
• To get a particular branch, use cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -r rtems-x-y-branch rtems

Once the above steps are done and successful, you will have a directory in the location where you issued the check out command called rtems.

3. Building rtems:

• Follow [9]
• Heads up: One MUST build RTEMS for the same platform you got / built the toolchain for. Eg: Both RTEMS (below) and the toolchain have to be i386 specific or sparc specific or arm specific or
  

Some notes about the build steps:

• I started with rtems-4.10 and it requires >autoconf-2.64. When I was building, Fedora had autoconf-2.63 and I had to get 2.64. So make sure you have 2.64
  
• Firstly, while issuing the configure, you have to the target using --target=XXX, enable some corresponding BSP using --enable-rtemsbsp=YYY and the location where RTEMS needs to be installed using --prefix=. The XXX and YYY are best obtained from the README.configure in the rtems root directory
• make
• make install (might require su rights, depending on install path)

Eg: My configure looked like:
../configure --target=sparc-rtems4.10 --enable-rtemsbsp=sis --prefix=/opt/rtems-4.10/rtems/

Getting example applications:

• You can get the example applications, by checking out the examples directory from CVS by cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P examples
  

Compiling applications:

• Reference [10]
• move to the directory hello_world_c
• before you make, you need to set the RTEMS_MAKEFILE_PATH as per [10]
• issue make there
  

Eg: With the above configuration, the RTEMS_MAKEFILE_PATH needs to be set to /opt/rtems-4.10/rtems/sparc-rtems4.10/sis/

Getting ready for the 1st run:

Of the several possible bsps/platforms, I am putting down two of them below which I have had tried: sparc and pc386

Lets start of with the easier one, which uses a simulator with gdb

1. sparc

• make your that the $PATH variable now can reach the rtems-specific compilers, gdbs and other binaries
• Run the sparc gdb: sparc-rtems4.10-gdb o-optimize/hello.exe
• set target as simulator, load the program and run; tar sim; start; load o-optimize/hello.exe; start
• You should now be seeing the output of hello.exe

So, what you did till now was running the test.c, compiled for sparc platform on a sparc simulator using the sparc specific gdb, that came along with the toolchain you installed

2. pc386 using qemu

• Make sure you have build the rtems for i386 / pc386. Eg:
  
• You now need qemu, a simulator, which simulates many platforms [11]. We will be using it to simulate the pc386 for us; yum install qemu should do the trick
• Download the images [12] that qemu provides and test if the qemu installation works well
• To quickly use and test qemu, [13]
• Get the gcc-testing folder from the rtems site (CVS access): cvs -d :pserver:anoncvs@www.rtems.com:/usr1/CVS -z 9 co -P gcc-testing
• Lets consider a folder (call it root)
  
• Get the rtems-boot.img from [14] and rtems-grub.cfg from [15] and place it in this directory
• Edit rtems-grub.cfg to have valid boot entries
  
• Also place the executable build for pc386 in this directory
  
• Invoke qemu while in this directory as qemu -boot a -fda rtems-boot.img -hda fat :. (boot from floppy(a), use current directory as hda, with these make sure your rtems-grub.cfg has the valid root as (hd0,0) and the valid path for the executables (hd0,0)/

Eg:
Have hello.exe you built for i386, rtems-boot.img and rtems-grub.cfg in a directory
rtems-grub.cfg would look like
set default=0

menuentry "RTEMS - Hello" {
set root=(hd0,0)
multiboot (hd0,0)/hello.exe
}
Use qemu -boot a -fda rtems-boot.img -hda fat:. and you should see the hello.exe output


3. An extension: Using gdb with qemu

Here is a small, but powerful extension to the above method (with qemu) where one can debug programs for pc386 using the i386-specific gdb

Reference: [17]
All the steps are the same as for the above method, except for the following:

• Invoke qemu with the additional switch -s (meaning wait for a gdb connection on localhost:1234)
• Invoke i386-rtems4.10-gdb
• On gdb prompt, connect to remote target, as tar remote localhost:1234
• On this one can put breakpoints, watch registers and all the other debugging possibilities

4. Using the gcc-testing rtems-testing module:

Another way to get things working is what I call "the easy Dr.Joel" way where all is ready, put things in place and bamm-you are good to go.

Reference: [16] - works like magic...!

Notes:

• In the sim-scripts/pc386, you might want to change the shell you want to use. I had to change it to /bin/bash

• Also, to see some "output" in front of your eyes, just change the way qemu is invoked to something similar to what was done in the previous method

Another method to see things running is using the qemu-nic in the qemu-support directory.

• You need to have the ~/qemu and the pc386_fda in that directory. Also the ~/qemu/hd should be present
  
• Copy the executable built for i386 into ~/qemu/hd as test.exe
  
• In qemu-nic, change the QEMUDIR to $HOME
• Invoke qemu-nic
• You should now be seeing the output of hello.exe

Next up: Getting started with coverage

Links and references: Accessed on 9th May, 2010

[0] - http://www.vmware.com/
[1] - http://www.virtualbox.org/
[2] - http://www.thoughtpolice.co.uk/vmware/#fedora12
[3] - http://www.rtems.org/wiki/index.php/Virtual_Machines_for_RTEMS_Development
[4] - http://www.rtems.com/wiki/index.php/Building_the_RTEMS_toolset_on_Ubuntu
[5] - http://www.rtems.com/wiki/index.php/APT/Yum_Repository
[6] - http://www.rtems.org/ftp/pub/rtems/linux/4.10/fedora/12/i386/rtems-4.10-yum-conf-0.20100423.0-1.fc12.noarch.rpm
[7] - http://www.rtems.org/ftp/pub/rtems/linux/4.10/fedora/12/i386/rtems-4.10-release-0.20100423.0-1.fc12.noarch.rpm
[8] - http://www.rtems.com/wiki/index.php/RTEMS_CVS_Repository
[9] - http://www.rtems.org/wiki/index.php/Quick_Start
[10] - http://www.rtems.org/wiki/index.php/Example_Application_Compiling
[11] - http://wiki.qemu.org/Main_Page
[12] - http://wiki.qemu.org/Download#QEMU_disk_images
[13] - http://wiki.qemu.org/download/qemu-doc.html#pcsys_005fquickstart
[14] - http://www.rtems.org/ftp/pub/rtems/people/chrisj/grub/rtems-boot.img
[15] - http://www.rtems.org/ftp/pub/rtems/qemu/i386-pc/rtems-grub.cfg
[16] - http://www.rtems.com/wiki/index.php/QEMU#Using_the_gcc-testing_Module
[17] - http://www.rtems.com/wiki/index.php/QEMU#Debugging_with_QEMU

Kick-off

Business, straight away!

Let us try to answer a few obvious questions, if this is the first time you are hearing the words RTEMS or GSoC.

RTEMS: In short, RTEMS is an real-time operating system supporting several open API and interface standards. Multi-tasking, the POSIX 1003.1b API standard, several networking standards are a few among several other supported features. To know more about RTEMS and to have a look at its exhaustive feature set, I would direct you to [0] and [1].

GSoC: Google Summer of Code, in short GSoC is what I would call an opportunity for students worldwide (almost) to get engaged with the world of open source software during the summer and also get paid (bonus!). You could have a look at [2] and for people who do not prefer reading text, [3] or [4].

This summer (2010), I have got the opportunity to work with RTEMS in the gsoc '10 program. A brief about the project and planned schedule is available at [5].

Till now,

- Met a lot of interesting people @ #rtems
- Some hands on RTEMS, thanks to the guided execution of "Hello World"
- Many iterations of change-review of the proposal
- Submission of proposal
- Some more work with RTEMS - Coverage analysis and attempts to improve coverage figures. These were partly successful, but no patches submitted yet.

Next,

- Submission of patch (hopefully the first one into RTEMS repo)
- Continuous interaction with the community and mentors via IRC / mailing list
- Submission of documents to google [6]
- Deciding the next chunk of code to be covered

Next up: Getting started with RTEMS

Links and References: Accessed on 8th May, 2010

[0] - www.rtems.org
[1] - http://www.rtems.org/wiki/index.php/RTEMSFeatures
[2] - http://code.google.com/soc/
[3] - http://www.youtube.com/watch?v=vBRRR0BQyz0&feature=player_embedded#!
[4] - http://www.youtube.com/watch?v=f-hYJAfovm8&feature=channel
[5] - http://sites.google.com/site/bharathswiki/home/actualstuff/from-the-world-of-foss/google-summer-of-code/RTEMS-GSoC2010-Proposal-Bharath-TestCoverage.pdf?attredirects=0&d=1
[6] - http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2010/faqs#student_docs