Using Rake for Erlang Unit Testing

My previous Erlang/Yaws project, dayfindr.com, is ticking along nicely:

$ yaws --status --id dayfindr
Uptime: 33 Days, 13 Hours, 38 Minutes

The feedback has been mostly positive, although I've had a "That's a very irritating site." comment. Well, you can't please everyone!

Following on the experience gained with dayfindr, I'm starting a new Lyme project which will be using some of the OTP design principles (Think of OTP as Erlang application design patterns). Using OTP will give me hot upgrades, which dayfindr lacks at the moment. The first step that I encountered towards this goal was getting my directory structure to conform to the OTP application structure:

/project      [The project root]
   /ebin      [The compiled .beam files]
   /src       [The source .erl files]
   /include   [Include files like record definitions]

I've been using the Makefile that's provided in the Erlang book, which is quite simple and compiles the beam files to the same directory as the source folder, which doesn't comply to the required directory structure. Now, I could just update the Makefile, but Makefiles have a cryptic syntax that I don't really want to spend time learning. Plus, they're difficult to debug in my experience. So I Googled around a bit for erlang Makefiles with little to disappointing success. Then I saw an interesting link:

Building Erlang with Rake

When they make a movie of my life and this moment, you will hear an orchestra and a choir of baritones singing "Eureka" when I click on the link.

Rake is an Ruby equivalent of Make, and more. It took some effort to get it working, since I had rake 0.7.1 on my machine, but trying to find the problem taught me a bit of Ruby in the process. Upgrading to 0.7.3 solved the problem. Sean's Rakefile compiles your src files into the ebin directory very nicely! After tinkering around with Rake, I realised that it's a really nice tool:

• It has a nice mix of declarative and imperative code. You can define rules (e.g. always compile .erl to .beam), or tasks (which can be imperative, e.g. running unit tests).
• You can use the full power of Ruby, and don't need to learn Make.
• It has syntax that is very close to the domain (i.e. it's a good DSL)
• It's easy to debug, since you can use the normal Ruby puts functions etc.
• How you set up you dependencies is completely up to you, e.g. you can have different rules for files that conform to different regular expressions.

After I got this working, inspired with confidence, I decided to integrate my unit testing into the Rakefile. I think it's important to note at this point that I have less than a day's Ruby experience, and it was easy to get this working. Hacking a Makefile would probably have taken me hours and hours. The idea is that the test task is dependent on the compile task, so that if you do a "rake test", it will compile anything that's new and run the unit tests. You can just compile with "rake compile".

In order to get this going I created two Erlang files, foo.erl and bar.erl. Here's bar.erl, which contains two functions, and a test (EUnit)for each function. One of the tests will fail:

-module(bar).
-export([bar1/0, bar2/0]).

-ifdef(EUNIT).
-include_lib("eunit/include/eunit.hrl").
-endif.


bar1() ->
  ok.

bar2() ->
  okk.

-ifdef(EUNIT).
bar1_test_() ->
  [
    ?_assert(ok == bar1())
  ].
-endif.

-ifdef(EUNIT).
bar2_test_() ->
  [
    ?_assert(ok == bar2())
  ].
-endif.

Notice that the unit test code is only compiled into the beam file when the EUNIT flag is set. You can set this in the Rakefile. The unit tests are in the same source file, so we can also test non-exported functions.

Now let's look at the Rakefile:

require 'rake/clean'

INCLUDE = "include"

ERLC_FLAGS = "-I#{INCLUDE} +warn_unused_vars +warn_unused_import"

SRC = FileList['src/*.erl']
OBJ = SRC.pathmap("%{src,ebin}X.beam")

CLEAN.include("ebin/*.beam")

directory 'ebin'

rule ".beam" => ["%{ebin,src}X.erl"] do |t|
  sh "erlc -D EUNIT -pa ebin -W #{ERLC_FLAGS} -o ebin #{t.source}"
end

task :compile => ['ebin'] + OBJ

task :default => :compile

task :run_tests => [:compile] do
  puts "Modules under test:"
  OBJ.each do |obj|
    obj[%r{.*/(.*).beam}]
    mod = $1
    test_output = `erl -pa ebin -run #{mod} test -run init stop`

    if /\*failed\*/ =~ test_output
      test_output[/(Failed.*Aborted.*Skipped.*Succeeded.*$)/]
    else
      test_output[/1>\s*(.*)\n/]
    end

    puts "#{mod}: #{$1}"
  end
end

The juicy bits are the rule and the "run_tests" task. The rule states "for each file ending in .erl, compile the beam file using erlc and put the beam file in ebin". The run_tests task starts up an erlang runtime for each module, and calls test() for that module. The test output is captured and parsed using regular expressions. I know the Ruby code can be improved, so comments are most welcome.

Here's what happens when I compile and run the tests:

$ rake clean
(in /home/bjnortier/development/project1)

$ rake
(in /home/bjnortier/development/project1)
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/foo.erl
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/bar.erl

$rake run_tests
(in /home/bjnortier/development/project1)
Modules under test:
foo: All 2 tests successful.
bar: Failed: 1.  Aborted: 0.  Skipped: 0.  Succeeded: 1.

At this points I have cleaning, compiling and running the tests for each module. Nice. I'm quite pleased with how it works at the moment, but there's still quite a bit of work to be done:

• Concatenating the running of the unit tests into one no-shell erlang execution. Actually running the tests are very fast, but the shell termination takes about a second or so. Thus, for each module, there is a approximately a second of extra time. Running all the tests in the same erlang session will require some more text parsing, but it's do-able.
  
• Displaying the failed tests. Seeing "Failed: 1" is not very useful for determining what went wrong, so I'll update the parsing to include the failures and errors.
• Probably change the "run_tests" task to just "test"
• Continuous integration that hooks into version control.
  
  

If there is significant progress I'll post the results. I hope you can use some of it :)