Serializing Enumerable

The following code demonstrates a memory saving technique for using a sequence of Enumerable methods on an input array. Turns out that you can have your cake and eat it as the speed performance is comparable but the memory consumption should be less

Library

require 'rubygems'
require 'ruby-prof'
require 'enumerator'

class Serializer

    attr_accessor :obj

    def collect(*args, &block)
        @obj.collect!(*args, &block)
    end

    def select(*args, &block)
        @obj = @obj.select(*args, &block)
    end

   def self.flatten(o)
       length = o.inject(0) {|m,i|m+i.length}
       out = Array.new(length)
       base = 0
       o.each do |sub|
           top = base + sub.length
           out[base..top]=sub
           base = top
       end

   end
end

module Enumerable

   def flatten
       length = @obj.inject(0) {|m,i|m+i.length}
       out = Array.new(length)
       base = 0
       @obj.each do |sub|
           out[base..base+sub.length]=sub
           base = base + sub.length
       end

   end
    def serially(slice=50, &b)
        t = []
        o = Serializer.new
        self.each_slice(slice) do |x|
            o.obj = x
            o.instance_eval(&b)
            t << o.obj
        end
        Serializer.flatten t
    end
end

Example

require 'serially'
require 'benchmark'

Benchmark.bmbm(15) do |bench|
    bench.report("serially") do
        a = (1..900000)

        b = a.serially(1000) do
            select do |x|
                if x > 500
                    true
                else
                    false
                end
            end
            collect do |x|
            "0x" + x.to_s
            end
            select do |x|
                if x == "0x15"
                    false
                else
                    true
                end
            end
        end
    end


    bench.report("classic") do
        a = (1..900000)

        b = a.select do |x|
            if x > 500
                true
            else
                false
            end
        end.collect do |x|
                "0x" + x.to_s
        end.select do |x|
            if x == "0x15"
                false
            else
                true
            end
        end
    end
end

#Results

Rehearsal --------------------------------------------------
serially         7.410000   0.120000   7.530000 (  7.731576)
classic          5.890000   0.080000   5.970000 (  6.123102)
---------------------------------------- total: 13.500000sec

                     user     system      total        real
serially         6.500000   0.000000   6.500000 (  6.581172)
classic          3.160000   0.010000   3.170000 (  3.199095)

Serialization and deserialization of matlab data

To serialize any matlab data type create a file called mserialize.c and mdeserialize and fill it thus

#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs,
                 const mxArray *prhs[])
{
   plhs[0] = (mxArray *) mxSerialize(prhs[0]);
}

#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs,
                 const mxArray *prhs[])
{
   plhs[0] = ( mxArray * ) mxDeserialize(mxGetData(prhs[0]), mxGetNumberOfElements(prhs[0]));
}

Note the above code uses unsupported matlab API so don't bug Mathworks tech support if it doesn't work or your matlab session crashes or your computer explodes :)

You use the functions thus

foo.a = 10

data = mserialize(foo)

bar = mdeserialize(data)

##################

Following from a post from CSSM by Dr Tristram J. Scott is another technique. I quote the whole post here

Related to this, I think, is the (also undocumented) save stdio command:

>> save stdio x

This sends x to stdout with a bit of header information, in much the same
format as the save command might have done. There is a corresponding load
command.

This allows you to use MATLAB in a pipe in the way which is implemented for the MATLAB engine under UNIX.

If you start a MATLAB engine using a tee you can see a bit of what goes on

#define MYMATLAB "tee /tmp/f1 | matlab -nojvm | tee /tmp/f2 -"
   if (!(ep = engOpen(MYMATLAB))) {
      syslog(LOG_ERR, "Can't start MATLAB engine.");
      syslog(LOG_ERR, "Exiting.");
      exit(EXIT_FAILURE);
   }

Send it a command such as x = rand(3), then use engGetVariable(ep, "x") to get MATLAB to send the variable through the pipe up to your C code, ready for mxGetPr() or whatever to use it. Examine the /tmp/f1 and /tmp/f2 files
to see the wrapper which the engine puts around all of your engEvalString() calls.