SCons Magic Glob

SCons is a great tool for building projects in C/C++. However I ran across a little problem the other day. I was using glob to get lists of source files from the source directory to then build into a library.

However some of those sources were auto-generated. Previously to get around this I would maintain a list of the auto generated sources and add them to the list of those discovered by glob. This was ugly but worked until I decided to change the build directory for the library. Then glob didn't work at all. Why was this? Well, it is because SCons copies the SConscript file to the build directory before executing it. glob then works on the build directory and as there are no files there yet it returns nothing.

There must be an elegant way to solve this and it turns out after a day of banging around there sort of is. On the SCons wiki there are a few solutions to the problem none of which seemed to work when I tried them so I rolled my own based upon what I could glean from the wiki.

However before I show the code I want to list my requirements. I want a Glob function that will return a list of files matching a pattern. The list must include files that

- permanently exist in the source/build directory.
- may be generated in the source/build directory from
previously executed SCons build declarations.

This will be powerful because our Glob will return lists of files some of which may not actually exist but that SCons has promised to build. This will allow us to avoid the messy problem of carrying around lists of files.

Now the code

import fnmatch
import glob
import os

# Magic glob retrieves all files that
# exist in the source directory or will
# be generated by previously defined
# rules.
#
# Arguments
#
#     patten   -  glob pattern with no path information
#     dir      -  the relative dir to search for the
#                 pattern.
#
# XXX Need to implement recursive search as
# well as absolute directories
def Glob( pattern, dir = "."):

    # Fix up root relative directories
    if dir.startswith ("#"):
        dir = os.path.join (str (Dir ("#")), dir[1:])

    def search_static(node, pattern, base = ''):
        dir = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"

        results = []
        files = glob.glob(os.path.join(dir, pattern))
        for r in files:
            if r.startswith (base):
                r = r[len(base):]
            results.append(r)
        return results


    def search_dynamic(node, pattern, base = ''):
        results = []
        node = Dir(node)
        node_s = str(node)
        base = str(base)
        if not base.endswith('/') and base != '':
            base = base + "/"
        l = len (node_s)
        for f in node.all_children():
            f = str(f)
            if fnmatch.fnmatch (f, pattern):
                # append the src to the results
                if f.startswith(base):
                    f = f[len(base):]
                results.append (f)

        # This seems to be necessary to refresh the
        # object after a call to all_children()
        node.clear()

        return results

    def is_abs(path):
        return os.path.abspath(dir) == dir


    # The current build directory
    bd = Dir(".")
    build_dir_s = str(bd)
    build_dir_l = len(build_dir_s)


    # The current source directory
    sd = bd.srcnode()
    src_dir_s = str(sd)
    src_dir_l = len(src_dir_s)

    results = []

    if is_abs(dir):
        # Only search the static and
        # dynamic for this directory
        # and return the absolute
        # paths
        results +=  search_static(dir, pattern)
        results +=  search_dynamic(dir, pattern)
    else:
        # Search the static and dynamic
        # for both build and source
        # and return relative paths

        # Build dir
        bd  = Dir(".")
        bds = str(bd)
        bd_search = os.path.join(bds, dir)

        # Source dir
        sd  = bd.srcnode()
        sds = str(sd)
        sd_search = os.path.join(sds, dir)

    #  results += search_static (bd_search, pattern, bd )

        results += search_static (sd_search, pattern, sd )

        results += search_dynamic (bd_search, pattern, bd)

        results += search_dynamic (sd_search, pattern, sd)

    # Fix up ./ prefixes
    res = []
    for r in results:
        if r.startswith ('./'):
            r = r[2:]
        res.append (r)

    # Uniquify the list
    files = {}
    for f in res:
       files[f] = None

    res = [ f for f in files.iterkeys() ]

    return res

Return('Glob')

You can get a handle to this function in your SConstruct file by

Glob = SConscript('scons_glob.py')

#auto generate a file
Command ("foo.c", "touch foo.c")

#get a list of files
files = Glob ("*.c")

Will return "foo.c" even if it has not been built yet.

Ruby style Python

After upsetting the Gods by getting Ruby to look a a bit like python, I thought about trying to offend in the opposite direction.

The most distinguishing thing about Ruby of course is the block notation which everybody, except pythonistas, love. They have lambda but it is just allows a single expression. Anonymous blocks have been deliberately left out of the language but with some hackery it is possible to get close to what we want.

The solution still has some cruft but it is pure python and just uses that standard toolkit of for loops, generators, iterators and decorators.

The trick is to decorate an iterator so that it has a few more desired features. Ie we need to be able to return values from the body of the for loop back into the generator.

The decorator adds an extra ret function parameter to the head of the parameter list of the generator function. This enables the generator to provide a final value to the top level code.

It also adds another r parameter to the head of the parameter list in the body of the for loop so that it can return values to the generator after every iteration.

from ruby import rubyfi

@rubyfi
def inject(ret, iter, a = 0):
   for x in iter:
      a = yield a, x
   ret(a)

@rubyfi
def collect(ret, iter):
   r = []
   for x in iter:
      x = yield x,
      r.append(x)
   ret(r)

# Do a sum over a range
for r, s, x in inject ( range (1,6), 1 ) :
   r(s+x)
print r()

# Generate a list
for r, s, x in inject ( range (1,6), [] ) :
   s.append(x*x)
   r(s)
print r()

# Collect items
for r, x in collect ( range (1,6) ) :
   r(x+x)
print r()

which generates

>> python iterator.py
16
[1, 4, 9, 16, 25]
[2, 4, 6, 8, 10]

The library to make it work (ruby.py) is

class Interceptor:

   def value(self, *v):
      if len(v) > 0:
         self.val = v[0]
      return self.val

   def __init__(self, generator):
      self.generator = generator
      self.val = None

   def __iter__(self):
      return self

   def next(self):

      def c(*args, **kwargs):
         return self.value(*args, **kwargs)

      y = self.generator.send(self.val)

      r = [c]
      r.extend(y)
      return r


def rubyfi(function):

      def wrapper(*args, **kw):

         g = []

         def ret(val):
            g[0].value(val)

         generator = function(ret, *args, **kw)

         g.append ( Interceptor(generator))

         return g[0]

      return wrapper

The critical trick I discovered was switching the next method which is called by default by the for loop mechanism into a call to the send method of the original generator. This allows me to feed values back into the iteration from the for loopp.

It is still uglier than Ruby but it can be done

:)

Python style Ruby.

Following from a thread on comp.lang.ruby I wondered what it would take to make Ruby more like Python. Yuk you say! Well probably but here is a start.

I updated the code after reading Giles Bowkett's" response to my post about needing the *.pyrb files. After a little poking around and finding hyphen-ruby I discovered a smarter way to do this.

I also made some other improvements to allow deeper dedenting. The trick to the new version is to use the __END__ keyword to block the Ruby scanner and then reload the current file after doing the preprocessing.

Also you only need the __END__ thingie in the first file that is loaded. All subsequent loads will do the preprocessing automatically.



test.pyrb

require 'pyrb.rb'
__END__

def foo:
    [1,2,3,4].each do |i|:
        puts i
        [1,2,3,4].each do |j|:
            puts i
        if i == 2 :
            puts "foo"
        else:
            puts "bar"


foo

pyruby.rb

module PyRuby
    public


    def self.require(base_name)
        if $".include? base_name
            return false
        else
            file = find_file base_name
            if file
                load(file)
                $" << base_name
                return true
            end
        end
    end

    def self.load(file)
        if file
            txt = PyRuby.pyrb_convert file
            begin
                Object.module_eval txt, file, 1
            rescue SyntaxError => e
                $stderr.print "Preprocessing Fail: " + $!
                $stderr.print "-----------------------------------\n"
                $stderr.print txt
                raise
            end
            return true
        end
        false
    end

    def self.find_file(fname)
        if FileTest.exist?(fname)
            return fname
        else
            $:.each do |path|
                path = File.join(path, fname)
                for ext in [ '.rb' ] do
                    p = path + ext
                    return p if FileTest.exist?(p)
                end
            end
        end
    end

    def self.pyrb_convert file
        txt = ""
        stack = []
        found_pyrb_req = false
        do_preprocess = false

        File.open file do |fh|


            fh.each_line do |l|
                l.chomp!
                if l =~ /^require\s*'pyrb.rb'\s*$/
                    found_pyrb_req = true
                end
                if l =~ /^__END__\s*$/ && found_pyrb_req
                    do_preprocess = true
                    txt = ''
                elsif do_preprocess

                    # Do the indent preprocessing
                    indent = (l.gsub /\S.*/,'').length
                    stack.reverse.each do |pindent|
                        if indent <= pindent
                            case l.gsub /^\s*/,''
                            when /(else|end|rescue).*/
                            else
                                txt << (" " * pindent) + "end\n"
                            end
                            stack.pop
                        end
                    end
                    if l =~ /:\s*$/
                        stack.push indent
                        l.gsub! /:\s*$/,''
                    end
                    txt << l << "\n"


                else
                    # Treat the code as normal Ruby code
                    # and pass straight through.
                    txt << l << "\n"
                end
            end
        end
        if do_preprocess
            while not stack.empty?
                txt << (" " * stack.pop) + "end\n"
            end
        end
        txt
    end
end

alias __pyrb_load load

def load base_name
    PyRuby.load base_name
end

def require base_name
    PyRuby.require base_name
end

PyRuby.load $0

( updated 29/05/2007 )