SCons Magic Glob

SCons is a great tool for building projects in C/C++. However I ran across a little problem the other day. I was using glob to get lists of source files from the source directory to then build into a library.

However some of those sources were auto-generated. Previously to get around this I would maintain a list of the auto generated sources and add them to the list of those discovered by glob. This was ugly but worked until I decided to change the build directory for the library. Then glob didn't work at all. Why was this? Well, it is because SCons copies the SConscript file to the build directory before executing it. glob then works on the build directory and as there are no files there yet it returns nothing.

There must be an elegant way to solve this and it turns out after a day of banging around there sort of is. On the SCons wiki there are a few solutions to the problem none of which seemed to work when I tried them so I rolled my own based upon what I could glean from the wiki.

However before I show the code I want to list my requirements. I want a Glob function that will return a list of files matching a pattern. The list must include files that

- permanently exist in the source/build directory.
- may be generated in the source/build directory from
previously executed SCons build declarations.

This will be powerful because our Glob will return lists of files some of which may not actually exist but that SCons has promised to build. This will allow us to avoid the messy problem of carrying around lists of files.

Now the code

import fnmatch
import glob
import os

# Magic glob retrieves all files that
# exist in the source directory or will
# be generated by previously defined
# rules.
#
# Arguments
#
#     patten   -  glob pattern with no path information
#     dir      -  the relative dir to search for the
#                 pattern.
#
# XXX Need to implement recursive search as
# well as absolute directories
def Glob( pattern, dir = "."):

    # Fix up root relative directories
    if dir.startswith ("#"):
        dir = os.path.join (str (Dir ("#")), dir[1:])

    def search_static(node, pattern, base = ''):
        dir = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"

        results = []
        files = glob.glob(os.path.join(dir, pattern))
        for r in files:
            if r.startswith (base):
                r = r[len(base):]
            results.append(r)
        return results


    def search_dynamic(node, pattern, base = ''):
        results = []
        node = Dir(node)
        node_s = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"
        l = len (node_s)
        for f in node.all_children():
            f = str(f)
            if fnmatch.fnmatch (f, pattern):
                # append the src to the results
                if f.startswith(base):
                    f = f[len(base):]
                results.append (f)

        # This seems to be necessary to refresh the
        # object after a call to all_children()
        node.clear()

        return results

    def is_abs(path):
        return os.path.abspath(dir) == dir


    # The current build directory
    bd = Dir(".")
    build_dir_s = str(bd)
    build_dir_l = len(build_dir_s)


    # The current source directory
    sd = bd.srcnode()
    src_dir_s = str(sd)
    src_dir_l = len(src_dir_s)

    results = []

    if is_abs(dir):
        # Only search the static and
        # dynamic for this directory
        # and return the absolute
        # paths
        results +=  search_static(dir, pattern)
        results +=  search_dynamic(dir, pattern)
    else:
        # Search the static and dynamic
        # for both build and source
        # and return relative paths

        # Build dir
        bd  = Dir(".")
        bds = str(bd)
        bd_search = os.path.join(bds, dir)

        # Source dir
        sd  = bd.srcnode()
        sds = str(sd)
        sd_search = os.path.join(sds, dir)

    #  results += search_static (bd_search, pattern, bd )

        results += search_static (sd_search, pattern, sd )

        results += search_dynamic (bd_search, pattern, bd)

        results += search_dynamic (sd_search, pattern, sd)

    # Fix up ./ prefixes
    res = []
    for r in results:
        if r.startswith ('./'):
            r = r[2:]
        res.append (r)

    # Uniquify the list
    files = {}
    for f in res:
       files[f] = None

    res = [ f for f in files.iterkeys() ]

    return res

Return('Glob')

You can get a handle to this function in your SConstruct file by

Glob = SConscript('scons_glob.py')

#auto generate a file
Command ("foo.c", "touch foo.c")

#get a list of files
files = Glob ("*.c")

Will return "foo.c" even if it has not been built yet.

Using doxygen and Eclipse for requirements capture

I have been looking around for a good tool for requirements capture but it has been right under my nose. A combination of C++ namespaces and doxygen comments is a superb solution. I am currently using the Eclipse CDT with vxWorks 6.5 which adds the extra benefit or refactoring requirement ids.

Here is a simple requirements file as a C++ header file

/** @addtogroup requirements
 * @{
 */

/** @file requirements.hpp
 *
 */

/** requirements type */
typedef int rq;

/** com */
namespace Com {
    /** xtargets */
    namespace Xtargets {
        /** client */
        namespace Client {

            /** Make sure the product works */
            rq r1;

            /** Ensure customers are happy */
            rq r2;

            /** Get rich from selling many copies */
            rq r3;
        }
        namespace Kernel {

            /** Hide bugs from customers */
            rq r1;

            /** All actions must occur simultaneously
             * and in zero time. */
            rq r2;

            /** Use quantum features of the compiler when
             * environment variables MAGIC is set. */
            rq r3;

        }
    }
}
/** @} */

You don't really use this file as a header file but just take advantage of Doxygen and C++ notation to do the heavy work for you. If using an IDE then you get for free a nice requirements browser using the symbol browser.

When using requirements traceability is important so you may want to refer to certain requirements from comments in your source code. You can do this by using Doxygen notation to create auto links.

/** Some other comment in the code.
 *
 * See Com::Xtargets::Client::r1
 */

Now if you need to change the identifier for requirement r1 then you can use the refactoring capabilities of your IDE if available. In Eclipse you can change the name of the variable and ask it change all references to it in source code and source comments. You thus have a fully traceable requirements management solution without any fancy new tools to setup sans Doxygen and your IDE.

If using Eclipse I recommend setting up a sub project of your main project to house your requirements psuedo header files. When it comes to refactoring this allows Eclipse to find all the sources that might refer to the specific requirements.

I use the nice Doxygen plugin for Eclipse call Eclox to make life a little easier.