SCons Magic Glob

SCons is a great tool for building projects in C/C++. However I ran across a little problem the other day. I was using glob to get lists of source files from the source directory to then build into a library.

However some of those sources were auto-generated. Previously to get around this I would maintain a list of the auto generated sources and add them to the list of those discovered by glob. This was ugly but worked until I decided to change the build directory for the library. Then glob didn't work at all. Why was this? Well, it is because SCons copies the SConscript file to the build directory before executing it. glob then works on the build directory and as there are no files there yet it returns nothing.

There must be an elegant way to solve this and it turns out after a day of banging around there sort of is. On the SCons wiki there are a few solutions to the problem none of which seemed to work when I tried them so I rolled my own based upon what I could glean from the wiki.

However before I show the code I want to list my requirements. I want a Glob function that will return a list of files matching a pattern. The list must include files that

- permanently exist in the source/build directory.
- may be generated in the source/build directory from
previously executed SCons build declarations.

This will be powerful because our Glob will return lists of files some of which may not actually exist but that SCons has promised to build. This will allow us to avoid the messy problem of carrying around lists of files.

Now the code

import fnmatch
import glob
import os

# Magic glob retrieves all files that
# exist in the source directory or will
# be generated by previously defined
# rules.
#
# Arguments
#
#     patten   -  glob pattern with no path information
#     dir      -  the relative dir to search for the
#                 pattern.
#
# XXX Need to implement recursive search as
# well as absolute directories
def Glob( pattern, dir = "."):

    # Fix up root relative directories
    if dir.startswith ("#"):
        dir = os.path.join (str (Dir ("#")), dir[1:])

    def search_static(node, pattern, base = ''):
        dir = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"

        results = []
        files = glob.glob(os.path.join(dir, pattern))
        for r in files:
            if r.startswith (base):
                r = r[len(base):]
            results.append(r)
        return results


    def search_dynamic(node, pattern, base = ''):
        results = []
        node = Dir(node)
        node_s = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"
        l = len (node_s)
        for f in node.all_children():
            f = str(f)
            if fnmatch.fnmatch (f, pattern):
                # append the src to the results
                if f.startswith(base):
                    f = f[len(base):]
                results.append (f)

        # This seems to be necessary to refresh the
        # object after a call to all_children()
        node.clear()

        return results

    def is_abs(path):
        return os.path.abspath(dir) == dir


    # The current build directory
    bd = Dir(".")
    build_dir_s = str(bd)
    build_dir_l = len(build_dir_s)


    # The current source directory
    sd = bd.srcnode()
    src_dir_s = str(sd)
    src_dir_l = len(src_dir_s)

    results = []

    if is_abs(dir):
        # Only search the static and
        # dynamic for this directory
        # and return the absolute
        # paths
        results +=  search_static(dir, pattern)
        results +=  search_dynamic(dir, pattern)
    else:
        # Search the static and dynamic
        # for both build and source
        # and return relative paths

        # Build dir
        bd  = Dir(".")
        bds = str(bd)
        bd_search = os.path.join(bds, dir)

        # Source dir
        sd  = bd.srcnode()
        sds = str(sd)
        sd_search = os.path.join(sds, dir)

    #  results += search_static (bd_search, pattern, bd )

        results += search_static (sd_search, pattern, sd )

        results += search_dynamic (bd_search, pattern, bd)

        results += search_dynamic (sd_search, pattern, sd)

    # Fix up ./ prefixes
    res = []
    for r in results:
        if r.startswith ('./'):
            r = r[2:]
        res.append (r)

    # Uniquify the list
    files = {}
    for f in res:
       files[f] = None

    res = [ f for f in files.iterkeys() ]

    return res

Return('Glob')

You can get a handle to this function in your SConstruct file by

Glob = SConscript('scons_glob.py')

#auto generate a file
Command ("foo.c", "touch foo.c")

#get a list of files
files = Glob ("*.c")

Will return "foo.c" even if it has not been built yet.

Recursively pcode a directory

Say you have a directory hierarchy with lots of m files you wish to pcode. This will do it.

function recursive_pcode(path)
    if nargin == 0
        path = fileparts(mfilename('fullpath'));
    end
    files = dir(path);
    for i = 1:length(files)
        file = files(i);
        file_name = fullfile(path,file.name);
        if is_subdir(path, file)
            recursive_pcode(file_name);
        elseif ism(file)
            pcode(file_name,'-INPLACE');
        end
    end

end


function out = ism(file)
    out = ~isempty(regexp(file.name,'\.m$'));
end

function out = is_subdir(currentdir, file)
    if file.isdir
        out = true;
        if strcmp(file.name, currentdir)
            out = false;
        elseif strcmp(file.name, '..')
            out = false;
        elseif strcmp(file.name, '.')
            out = false;
        end
    else
        out = false;
    end
end

Use it like

recursive_pcode('d:\users\brad\mfiles');