Wednesday, 12 November 2008

Does Robert Mugabe use Powerpoint?

I think most people will agree that Robert Mugabe is a terrible dictator who has scant disregard for the people he is supposed to be serving. The same goes for the likes of Stalin, Hitler, Idi Amin etc. That being said, I think most people will also agree that one of the reasons of their rise to power was specifically their ability to influence people. In other words

People listen[ed] to them, and they're not using Powerpoint

My experience is that every time someone want to say something to more than one person, there has to be a Powerpoint presentation to go along with it. And more often than not, this attempt at communication fails miserably, or succeeds only in some rudimentary fashion. 

Some of the causes of these failures are:
  • People are reading the slides, so they're not listening to the speaker.
  • The presenter is reading the slides, so there is no connection with the audience.
  • The bullet-pointed slides are so mind-numbingly boring that the audience members have gone to their happy place or are thinking of what to cook/order/kill for dinner and of course, they're not listening.
In contrast, I'll mention two very successful entrepeneurs and their presentation styles: Steve Jobs (Apple, Pixar, General Motors?) and Elon Musk (PayPal, Tesla Motors, SpaceX). In business it is absolutely essential that you can effectively communicate your ideas to people. And when you audience is holding the purse strings, and you can't convince them of your business ideas, you're doomed to become a (perhaps very successful) hobbyist.

I went to a talk by Elon Musk on his company SpaceX a few months ago. The projector had been set up and the presentation was ready to proceed. The first slide was already on the screen, with text along the lines of "Elon Musk, SpaceX" with very nice picture of a rocket (on a Pacific island if I remember correctly). 

Elon Musk came in and talked a bit. After a few minutes he proceeded to questions, and these went on for more than an hour. The first slide was the only slide. Maybe the host created it when he realised there were no slides. "Are you crazy? No slides?! But how will you...."

The talk was very effective, he had communicated his points, and gave the audience exactly what they wanted, since they were asking him what they wanted to know!

The second example is Steve Jobs. I remember watching the first iPhone Keynote speech and how I got swept up in the presentation. It was a highly effective communication (helped by a great product). Let's look at some of the slides he used:







Do you see lots of bullet points? The last slide has the most text on it, but that slide is in the minority, and the bullets are actually very short. The first two slides had simple images, and minimal text, and I can assure you that people were listening to what he was saying!

Do you think the response would have been the same if his slides looked like this:

instead? I'm asleep already.

My highly opinionated advice for presentations would be:
  • Don't use PowerPoint/Keynote/[insert slide software here] to write your presentation. Decide what you want to say and how you want it to flow and only prepare your slides once you know the content of your presentation. 
  • Don't use your slides as notes, have seperate notes if you don't know the topic that well or need some backup information.
  • Don't read you slides.
  • Don't have lots of bullet points on each slide.
  • You don't need to use slides. Sometimes a white board is good enough, and in most cases it is probably better.
What do you think?


Monday, 8 September 2008

Iterative and Agile - the Venn Diagram of truth

I'm seeing a bit of this lately:

"Iterative software development is Agile software development."

and I have to say that I disagree quite strongly. The Venn diagram actually looks more like this:


This post has been prompted by two things. First, a few weeks ago a SCRUMmer made some comments along the following lines:
  • "We don't allow changing the iteration plan once the iteration has begun" 
  • "All the requirements for the stories for the iteration must have been gathered before it starts."
  • "If the customer want something else, they have to wait for the next iteration"
And some articles/blogs about iteration lengths that jogged my memory about what I disagreed with. The root of my discomfort in equating iterative development to agile development is because

agile software development is a set of values, it is not a process. 

In fact, one of the core values of Agile development is to not value you process too much. There are some processes that try to embody Agile values, and they have different interpretations and practicalities around those values, but these processes are not what defines Agile development. 

Here's a reminder of what it means to be agile:
  • Individuals and interactions over processes and tools 
  • Working software over comprehensive documentation 
  • Customer collaboration over contract negotiation 
  • Responding to change over following a plan 
It's very possible to do iterative development without embodying any of these values. Treating each iteration like a waterfall with the usual waterfall suspects (requirements up front, little room for manoevering, opressive specification and documentation) is a case in point. And to me, valuing your plan over your customer by not responding to their needs is not Agile.

You are, of course, invited to disagree...

Wednesday, 3 September 2008

Me and my dog on Google Chrome

Like every man and his dog I feel compelled to say something about Chrome, Google's new browser. Most impressions I have read have been favourable, and my personal experience so far is very positive. Most reviews I have scanned have been thumbs-up, and have focussed on the usual suspects, i.e. rendering and javascript performance, stability, usability etc. etc.

For me, there is something subtle and extremely powerful that comes with Chrome. It fields a combination of features with very powerful implications. A "whole being more that the sum of the parts" kind of situation.

In the past year or two there has been a lot of focus in the "richer" web experience, and on ways of getting the web to feel more like the desktop. Newish technologies like Flex and Silverlight have received significant attention, and DHTML (Javascript+CSS+DOM+HTML) seems to have become the cousin that you are obliged to invite to your wedding party (or application party), but don't really want to. I believe that part of the reason for this has been Javascript performance, but there are several efforts underway (e.g. TraceMonkey and now Chrome's V8 Javascript engine) that will bring huge leaps in performance to Javascript. Also, I prefer to support non-proprietary technologies such as DHTML over vendor-based alternatives like Flex and Silverlight.

The second feature that's important in Chrome is process isolation for tabs. This enables each tab to be completely isolated from other processes. One tab going down doesn't take the whole browser with it (which is something that has plagued my Firefox quite a bit).

Thirdly, we have "application-like" OS integration in Chrome. Chrome has the ability to make a single tab look like an application window, with normal window decorations:


In your taskbar it looks like an application, there is  quick lauch icon for you app:

There's also a Start Menu item (not shown) and a Desktop icon:


The combination of these three is a significant step towards blurring the lines between the desktop and the web. Chrome is an application platform with (potentially) desktop-like application performance (DHTML + V8), process isolation (a key feaure of an application platform) and the web apps look like desktop apps (Operating system integration). And it's all standards based.

There have been some rumours of a Google OS...

You're looking at it.

Thursday, 24 July 2008

Scalaris Released

If you were at the Erlang Exchange in London last month, you should know that one of the hottest talks was given by Alexander Reinefeld, "Building a transactional distributed data store with Erlang"

Joe Armstrong seems to like it too:

"I might be wrong, but my gut feeling is that what Alexander Reinefeld showed us will be the first killer application in Erlang."

"So my take on this is that this is one of the sexiest applications I've seen in many a year. I've been waiting for this to happen for a long while. The work is backed by quadzillion Ph.D's and is really good believe me."

- http://armstrongonsoftware.blogspot.com/2008/06/itching-my-programming-nerve.html

Well, seems like Scalaris has been released! The link went up on the website in the last 24-48 hours. I haven't had the time to look at it yet, but you can grab it here.

Wednesday, 2 July 2008

Notes from Erlang Exchange

I was at the Erlang Exchange in London last week, which I really enjoyed. It was really great to see Joe Armstrong, and to see the faces behind the projects, e.g. Claes Wikström (Yaws, Bluetail, Kreditor, Tail-f), Steve Vinoski, Eric Merrit and Martin Logan (Erlware), Matthias Radestock (LShift & RabbitMQ), Mickaël Rémond (ejabberd) etc. etc.

Something that Joe said reminded me of a comment that Keith made about 6 months ago. I think this is something that gets lost in the hype around Erlang and multicore computing:

Erlang was designed to program fault tolerant systems.

It was not designed to program multicore computers. It was designed 15-20 years ago when multicore-everywhere was not even on the horizon. As a side-effect, it is a good language for multicore systems, since the only way to actually achieve fault tolerance is to have independant processes (read here for some more on this).

I think it's important for us as to remember to advocate this, and instead of marketing Erlang/OTP as "the application system for multicore", we should be marketing it as "the application system for fault tolerance". Oh, and by the way "this is one of the few systems that will actually use those cores that are coming your way".

The second point that I have been pondering, was brought on my Gordon Guthrie's talk about Erlang/OTP vs Google Apps as an application engine. At this moment, there aren't many application systems to choose from, Erlang/OTP, Google Apps and Amazon EC2 come to mind. In the context of the talk, an Application System (A/S) would be something that takes care of a lot of the non-functional requirements of you system, e.g. reliability, scalability (+distribution) etc.

For some businesses, using Google Apps or EC2 would be a good fit, but then you're tied into the value chain of that business, and in some instances you just cannot (for legal and other reasons) put your data on servers you don't own. If you go the OTP route you have full control over your application(s), but there's a lot of extra infrastructure that you will have to supply yourself.

If you start thinking of Erlang/OTP in this way, you realise that the terms should be reversed and it should actually be OTP/Erlang. The platform is actually the important thing. The platform is the thing that gives you the reliability, scalability, distribution and hot code swapping etc. that you require. Erlang becomes this awesome language that you use to build things on the platform. Looking at things from a different perspective can lead to some interesting insights.

If I'm standing at this vantage point, heated discussions like the recent meme-storm in the blogosphere of "Erlang vs Scala" become much less relevant. Is there a platform that gives you reliability, scalability, distribution, hot code replacement that uses Scala as a language? No? Oh. What a pity...

Tuesday, 17 June 2008

Migrating a native Erlang interface to RESTful Mochiweb (with a bit of TDD)

The title is a bit of a mouthful, but it does contain in essence what I will show you:

1. I will convert an existing native erlang CRUD interface (a simple client-server) to use a HTTP layer.
2. The HTTP calls will be RESTful.
3. I will be using Mochiweb.
4. I will show you how to do a bit of Test-Driven-Development (TDD) for this exercise using EUnit.

As an introduction, I suggest you read Steve Vinoski's "RESTful Services with Erlang and Yaws", and "A RESTful web service demo in yaws" by Nick Gerakines. They are both excellent posts, and Nick's post has a lot of interesting details, including some OTP goodness. Also check out Daryn's posts for a Rails twist on this topic.

I will NOT be showing much error handling, simply returning a 501 response if the REST Url has
some mistake in it.

This is the plan:
1. We will start with a working set of tests for a native CRUD interface. I will not show you the actual implementation, but it is simple enough to do yourself using a process dictionary.
2. I will show a very simple REST interface with Mochiweb, that just returns the request method type as a plaintext string. This will be tested using Inets, the native Erlang OTP internet module.
3. We will write a translators to and from plaintext and native erlang terms (again, unit tested).
4. We will combine all these into the REST interface, tested using Inets again.

Right, here is the unit tests for the CRUD interface:
crud_test_() ->   
{setup,
fun() -> start() end,
fun(_) -> stop() end,
fun(_) ->
[
?_assert(ok == create(#person{id = 1})),
?_assert(already_exists == create(#person{id=1})),
?_assert(#person{id=1} == retrieve(1)),
?_assert(ok == update(#person{id = 1, name="Ben"})),
?_assert(#person{id=1, name="Ben"} == retrieve(1)),
?_assert(ok == delete(1)),
?_assert(undefined == delete(2)),
?_assert(undefined == update(#person{id = 2}))
]
end}.
The CRUD interface is container in a module "crud", and start() and stop() are methods to start and stop the CRUD server. For EUnit, you can define a test with a setup, teardown and tests, and this is the form that I've used here. There are many forms of tests for EUnit, and I suggest you consult the EUnit documentation (contained in the EUnit distribution).

EUnit automatically creates a test() function on the module when you include "eunit.hrl", so let's run the crud tests:

75> crud:test().
All 8 tests successful.
ok
We know that are CRUD interface is working OK.

Onto Step 2! If you're not familiar with Mochiweb, it's a lightweight web server developed by the guys at Mochi Media. It's very easy to integrate into you application, and has very little configuration.

Here are the tests for the simple Mochiweb demo:

-define(URL, "http://127.0.0.1:8888").

rest_server_test_() ->
{setup,
fun() -> inets:start(), start_simple() end,
fun(_) -> inets:stop(), stop_simple() end,
fun(_) ->
[
?_assert(http_result('GET') =:= "GET"),
?_assert(http_result('PUT') =:= "PUT"),
?_assert(http_result('POST') =:= "POST"),
?_assert(http_result('DELETE') =:= "DELETE")
]
end}.

parse_result(Result) ->
{ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
ResultBody.

rest_server_test_() ->
{setup,
fun() -> inets:start(), start_simple() end,
fun(_) -> inets:stop(), stop_simple() end,
fun(_) ->
[
?_assert(http_result('GET') =:= "GET"),
?_assert(http_result('PUT') =:= "PUT"),
?_assert(http_result('POST') =:= "POST"),
?_assert(http_result('DELETE') =:= "DELETE")
]
end}.
start_simple() starts the simple version of the server and stop_simple() stops it. We're just using the base URL, and the server just returns a plain-text string of the request method. The http:request() functions are part of Inets (documentation here). You will notice that the put and post functions have extra parameters (which we will use later for the request body).

Here's the solution:

start_simple() ->
mochiweb_http:start(
[{ip, "127.0.0.1"},
{loop, {?MODULE, simple_response}}]).
stop_simple() ->
mochiweb_http:stop().


simple_response(Req, Method) ->
Req:ok({"text/plain", atom_to_list(Method)}).
simple_response(Req) ->
simple_response(Req, Req:get(method)).
We've started Mochiweb, and told it that the "simple_response" function in the current module should be used to handle requests. It takes one parameter, which contains the request data. The data can be queried with different methods to extract data from it, e.g. Req:get(method), gives you the method used.

The Req:ok() function is used to respond to a request, and we simply return plain text (with the text/plain MIME type).

And let's run the tests:

77> rest_server:test().
...
All 4 tests successful.
ok
Nice. The next bit of code we need is to translate Erlang terms to and from plain text. We'll use Base64 encoding for this. Erlang also has very hand term to binary and binary to term functions that we can use to achieve the translation.

The tests for the translation:

term_to_plaintext_test_() ->
A = anatom,
B = {a, 23, "abc"},
C = {props, [{a,3},{b,4}]},
[
?_assert(A == ptt(ttp(A))),
?_assert(B == ptt(ttp(B))),
?_assert(C == ptt(ttp(C)))
].


Where "ttp" would be "term_to_plaintext" and "ptt" is
"plaintext_to_term". Here's the implementation:

ttp(Term) ->
base64:encode_to_string(term_to_binary(Term)).

ptt(PlainText) ->
binary_to_term(base64:decode(PlainText)).


And the test result:

77> rest_server:test().
...
All 7 tests successful.
ok
(7 tests, since we have the first 4 and now the extra 3). Now things are getting a bit more complicated. We now have to ensure that the response to the request is converted from the native Erlang terms to plaintext, this result is then returned, and we also need a function to convert the request result into the native form.

Here we go:

result_to_terms(Result) ->
{ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
ptt(ResultBody).

rest_api('GET', Path) ->
Result = http:request(get, {?URL ++ Path, []}, [], []),
result_to_terms(Result);
rest_api('DELETE', Path) ->
Result = http:request(delete, {?URL ++ Path, []}, [], []),
result_to_terms(Result).

rest_api('POST', Path, Data) ->
Result = http:request(post, {?URL ++ Path, [], [], ttp(Data)}, [], []),
result_to_terms(Result);
rest_api('PUT', Path, Data) ->
Result = http:request(put, {?URL ++ Path, [], [], ttp(Data)}, [], []),
result_to_terms(Result).

crud_server_test_() ->
{setup,
fun() -> crud:start(), inets:start(), start() end,
fun(_) -> stop(), inets:stop(), crud:stop() end,
fun(_) ->
[
?_assert(ok == rest_api('POST', "/person", #person{id = 1})),
?_assert(already_exists == rest_api('POST', "/person", #person{id = 1})),
?_assert(#person{id=1} == rest_api('GET', "/person/1")),
?_assert(ok == rest_api('PUT', "/person/1", #person{name="Ben"})),
?_assert(#person{id=1, name="Ben"} == rest_api('GET', "/person/1")),
?_assert(ok == rest_api('DELETE', "/person/1")),
?_assert(undefined == rest_api('DELETE', "/person/1")),
?_assert(undefined == rest_api('PUT', "/person/1", #person{id = 2}))
]
end}.
There is a bit of duplication here, but for illustration I've kept the functions seperate. You will notice that the data is converted to plaintext for the PUT and POST requests. result_to_terms() converts the HTTP result from the plain text to the native Erlang terms.

You can scroll back to the original CRUD tests, and notice that the tests do exactly the same thing as on the CRUD interface, but we would have to do a "GET" + "/person/1" to get person 1, instead of doing a crud:retrieve(1).

The mapping between the CRUD and REST method are as follows:
GET <=> RETRIEVE
POST <=> CREATE
PUT <=> UPDATE
DELETE <=>DELETE

Here's the final product:

start() ->
mochiweb_http:start(
[{ip, "127.0.0.1"},
{loop, {?MODULE, crud_response}}]).
stop() ->
mochiweb_http:stop().


crud_response(Req, 'GET', "/person/" ++ IdString) ->
Id = list_to_integer(IdString),
Response = crud:retrieve(Id),
Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'DELETE', "/person/" ++ IdString) ->
Id = list_to_integer(IdString),
Response = crud:delete(Id),
Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'POST', "/person") ->
Body = Req:recv_body(),
Person = ptt(Body),
Response = crud:create(Person),
Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'PUT', "/person/" ++ IdString) ->
Id = list_to_integer(IdString),
Body = Req:recv_body(),
PersonWithNewValues = ptt(Body),
UpdatedPerson = #person{id = Id,
name = PersonWithNewValues#person.name,
email_address = PersonWithNewValues#person.email_address},
Response = crud:update(UpdatedPerson),
Req:ok({"text/plain", ttp(Response)});

crud_response(Req, _Method, Path) ->
Req:respond({501, [], Path}).

crud_response(Req) ->
crud_response(Req, Req:get(method), Req:get(path)).


Once again, there is some duplication, but it's easier to grasp when we split the functions. The GET and DELETE requests are simple to handle, since we just convert the Id string to an integer (which could throw an exception, you would have to handle that in some way), and call the CRUD interface.

POST is not much more complicated, we just get the body of the request using Req:recv_body().

The PUT function has a problem. There is duplication of the Id of the person, in both the URL "/person/1", and the actual term, #person{id=1,...}. I'm not sure how to handle this, comments are welcome. Perhaps you can generate an error response if the Ids don't match. Or you can make sure the posted record does NOT contain an Id field, and return an error if it does.

The solution as I've given it, uses the fields in the record, and ignores the Id of the record, using the Id in the URL instead.

The last function generates an error if the request could not be matched.

And the moment of truth:
81> rest_server:test().
...
All 15 tests successful.
ok
Looks so simple now, but I can asure you that it took some effort to get all the tests to pass!

Well I hope I've shown you something that you didn't know before, or even encouraged you to learn some Erlang.

Comments and criticisms are most welcome :)

Sunday, 8 June 2008

Restful Charts

What would you normally expect of a graph-generation library? I'm talking about the pie charts, bar charts etc. You would normally expect an output to some file format, it being PNG or SVG if you're lucky. BUT, gchartrb doesn't do this, it creates a URL that you would use to GET the graph from the Google charts API.

So what?

Well, it's interesting because it means that some people at least consider that generating the graph using a REST API is as reliable as disk, and as convenient as a local library. I've used Google charts before, but I've never thought about integrating it into another program as a service, and I think it's pretty neat. And easy.

I'm placing my chips on REST.

Wednesday, 7 May 2008

Parallel Quicksort in Erlang - Part II

In my previous post, Jaksa pointed out that there's a problem, in that although there may be multiple processes, they will be blocked and waiting (sequencially) for the first peval to complete, then the second peval, etc. Which will result in a pseudo-parallel depth-first quicksort, instead of a breadth-first quicksort. This is the offending line:


peval(fun pqsort/1, Left) ++ [Pivot] ++ peval(fun pqsort/1, Right).

I was hoping Erlang would be clever enough to execute the two functions in parallel, but that's a fairly unreasonable (and completely incorrect) assumption!

But how to fix it? There is a parallel implementation of the map function in Joe's book, and Montsamu has a variation of it on his blog:


-module(plists).
-export([pmap/2]).

pmap(F, L) ->
S = self(),
Pids = lists:map(fun(I) -> spawn(fun() -> pmap_f(S, F, I) end) end, L),
pmap_gather(Pids).

pmap_gather([H|T]) ->
receive
{H, Ret} -> [Ret|pmap_gather(T)]
end;
pmap_gather([]) ->
[].

pmap_f(Parent, F, I) ->
Parent ! {self(), (catch F(I))}.

The pmap_gather function is the interesting one. It will traverse the list of Pids, and wait for each to send back its result. Using pmap, we update pqsort as follows:


pqsort([]) -> [];
pqsort([Pivot]) -> [Pivot];
pqsort([Pivot|Rest]) ->
io:format("+", []),
Left = [X || X <- Rest, X < Pivot],
Right = [Y || Y <- Rest, Y >= Pivot],
[SortedLeft, SortedRight] = plists:pmap(fun pqsort/1, [Left, Right]),
io:format("-", []),
SortedLeft ++ [Pivot] ++ SortedRight.

And with a perfectly unbalanced (for quicksort) list to sort, we expect 1 process, then 3 as the first spawns 2 new ones, then down again to zero:


(emacs@21ccw.blogspot.com)47> pqsort:pqsort([4,2,1,3,6,5,7]).
+++---[1,2,3,4,5,6,7]

For the same example we had last time, sorting 100 random numbers, the number of concurrent process graph during execution looks like this:



Much better!

Thursday, 24 April 2008

Parallel Quicksort in Erlang

Jaksa is doing some posts on parallel graph algorithm in Java using fork/join, and he mentions quicksort:

"The equivalent of the hello world for parallel languages is the quicksort algorithm"

Now I'm tempted to say something like "Oh yea? You know how many lines of code quicksort is in Erlang? 3!" Yes really:


qsort([]) -> [];
qsort([Pivot|Rest]) ->
qsort([ X || X <- Rest, X < Pivot]) ++ [Pivot] ++ qsort([ Y || Y <- Rest, Y >= Pivot]).


But this would be wrong, as the terseness of the quicksort implementation says more about the power of list comprehensions than about Erlang's concurrency-supporting language features.

So then let's see how you could implement a concurrent quicksort in Erlang, which will say something about the way in which we can implement concurrency in Erlang. We'll start with the original Wikipedia quicksort from above, and alter it a little bit to make it more obvious what I'm going to do next:


qsort([]) -> [];
qsort([Pivot|Rest]) ->
Left = [X || X <- Rest, X < Pivot],
Right = [Y || Y <- Rest, Y >= Pivot],
qsort(Left) ++ [Pivot] ++ qsort(Right).


Note the two calls to qsort at the end. Since there is no shared state between these two functions, they can be executed in parallel, and the results can be concatenated to yield the sorted result. Depending on the distribution of the input, each of these calls could be split up into 2 more qsort calls etc. etc. Note that you could do a hybrid recursive-tail-recursive solution which sorts the smallest partition using normal recursion and the largest using tail-recusrion, but that is not the aim here.

Let's assume that we already have a function that executes any input function in a separate Erlang process called "peval". (Note Erlang processes are extremely light-weight and take very little time to construct. They're are not modelled as operating system processes). peval takes two arguments, a function to execute, and the arguments to the function. When called, it looks like a normal function call:


(emacs@21ccw.blogspot.com)38> pqsort:peval(fun(X) -> X*X end, 13).
169


But, in the background, it spawns a process and the function is evaluated on this new process. The result is then returned to the calling process. Using peval, we now have a parallel quicksort:


pqsort([]) -> [];
pqsort([Pivot|Rest]) ->
Left = [X || X <- Rest, X < Pivot],
Right = [Y || Y <- Rest, Y >= Pivot],
peval(fun pqsort/1, Left) ++ [Pivot] ++ peval(fun pqsort/1, Right).


Now there is a significant observation to make. The concurrent and non-concurrent versions look remarkably similar, we only had to substitute the original function evaluations with parallel versions. This is one of the reasons why I think functional languages are better suited to concurrency that OO ones.

Let's look at peval. peval creates a new process using spawn_link. spawn_link() is used instead of spawn(), so that the creator process receives exception when the new child process throws an exception.


peval(Fun, Args) ->
Pid = spawn_link(fun() -> wait() end),
Pid ! {self(), Fun, Args},
receive
{Pid, R} -> R
end.

wait() ->
receive
{From,Fun,Args} ->
From ! {self(), Fun(Args)}
end.


That's it! Parallel quicksort in about 15 lines of Erlang code. If I add a "+" output to console on every entry to peval, and "-" on every exit from peval, you get the following output:


(emacs@21ccw.blogspot.com)18> L1 = lists:map(fun(X) -> trunc(random:uniform()*100) end, lists:seq(1,100)).
[57,33,2,85,59,65,61,71,83,75,94,94,19,5,79,53,31,54,39,61,
61,79,55,6,37,35,86,31,88|...]

(emacs@21ccw.blogspot.com> pqsort:pqsort(L1).
+++-++++-+--++-+++++-+--+--+--+++-+--++-+------+++
+-++-+---+++-+--+---++-+-----+++++-+--++-+---+++++
-+--+--+++-+--+---++-+----++-++-++-+------++++-+--
++++-+--++-+---++++-+--++-+---++++-+--+++-++-+---+
++-+--+----+-----+++-+++-+--+++-++-+---+----++-+--
--
[2,4,5,6,7,9,11,13,14,17,18,19,20,21,22,24,27,31,32,33,35,
37,38,39,40,42,44,47,48|...]

(emacs@21ccw.blogspot.com)5> pqsort:integrate_output("+++-++++-+--++-+++++-+--+--+--+++-+--++-+------+++").
0123234565654565678910910989878767898987898987654345...


You can see that for the first few pevals, by integrating over the output, that number of parallel processes goes up to around 8-10. This will depend on the distribution, and the relative times it takes to initialise processes to doing the actual computation etc.

Now that we've got pqsort going, I'm looking forward to doing the parallel graph algorithms of Jaksa's fork/join graph algorithms. There will be an obstacle to ensure that no nodes are visited more than once, but this could (possibly) be solved by partitioning the graph into subgraphs first where no children have multiple parents. We'll see...

Update (7 May 2008): This implementation is wrong! (see comments). There's a follow-up post here: Parallel Quicksort in Erlang - Part II

Friday, 18 April 2008

IE7 Page Zoom Broken

We use XPlanner at work for some projects, and we noticed that there seems to be a problem with the hyperlinks in Internet Explorer (version 7.0.5730.11) when the pages are zoomed in (you can zoom pages in IE with Ctrl +). The horizontal areas for the links were not aligned to the actual text, which means that some links seem can't be clicked correctly, but you can click white space (look carefully at the first image, you can see the normal I-beam cursor):





I shouldn't be surprised, since it does so badly in the Acid3 test. I couldn't find any other reports of this issue, and will be happy to link to them if you post a comment.

P.S. Firefox, Opera and Safari don't have this problem.

Thursday, 10 April 2008

Getting started with distributed Erlang - Mnesia table relocation

Mnesia is a distributed database that forms part of the Erlang release. One of the features that I think is potentially powerful, is transparent table relocation across machines. With Mnesia, you can replicate tables to any nodes you wish in your network, and Mnesiatakes care of all the back end bits for you. With "transparent", I mean that you don't need to do anything in your clients to make them "aware" of the new tables. Reads that were taking place from a table on one machine, will now be distributed across multiple nodes (where the nodes reside on single or multiple machines).

I wanted to see how difficult it is to achieve this. For the setup, I installed two virtual Ubuntu 7.10 machines using VMware Player. You can get images for most Ubuntu distros at http://isv-image.ubuntu.com/vmware/. FYI, the username and password for these images is ubuntu:ubuntu. I named the two nodes

node1.21ccw.blogspot.com and
node2.21ccw.blogspot.com

You'll need to edit the network configurations with the IP addresses if you want to reproduce this experiment. If you need some help, post a question as comment :)

I now had two machines that could ping each other using the full names, and a warm and fuzzy feeling inside:


The next step was to start up an Erlang node on each machine. There's a catch here though. I got some problems using erl -sname, probably because of the way I set up the hostnames of the machines. So, I had to specify the fully qualified names manually:


ubuntu@node1:~/node1$ erl -name 'node1@node1.21ccw.blogspot.com'
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.5.5 (abort with ^G)


ubuntu@node2:~/node2$ erl -name 'node2@node2.21ccw.blogspot.com'
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.5.5 (abort with ^G)
(node1@node1.21ccw.blogspot.com)1> nodes().
[]


Notice the output of the nodes() command. This will return a list of other Erlang nodes that this node is aware of. Initially there's no awareness. To let a node know of another node, you can use net_adm:ping/1 to ping the other node. Both nodes will then become be aware of each other:


(node1@node1.21ccw.blogspot.com)4> net_adm:ping('node2@node2.21ccw.blogspot.com').
pong

(node1@node1.21ccw.blogspot.com)5> nodes().
['node1@node1.21ccw.blogspot.com']


(node2@node2.21ccw.blogspot.com)1> nodes().
['node1@node1.21ccw.blogspot.com']


Cool. Now the nodes know of each other. To get Mnesia started, you have to create a schema on each node. A schema is located on the file system, in the same location where the actual disc-copies of tables will reside. [node()|nodes()] creates a list of the current node and all the other connected nodes. ls() shows the directory that Mnesia has created for the database.


(node1@node1.21ccw.blogspot.com)5> mnesia:create_schema([node()|nodes()]).
ok

(node1@node1.21ccw.blogspot.com)6> ls().
Mnesia.node1@node1.21ccw.blogspot.com
ok



(node2@node2.21ccw.blogspot.com)2> ls().
Mnesia.node2@node2.21ccw.blogspot.com
ok


Now we have to start Mnesia on both nodes. You will notice that when we do an mnesia:info on node2 at this point, that it shows both nodes as being running database nodes.



(node1@node1.21ccw.blogspot.com)8> mnesia:start().
ok



(node2@node2.21ccw.blogspot.com)3> mnesia:start().
ok

(node2@node2.21ccw.blogspot.com)4> mnesia:info().
...
running db nodes = ['node1@node1.21ccw.blogspot.com','node2@node2.21ccw.blogspot.com']
...

Next we'll create an actual database table, and populate it with some data. We define a record using rd(), then create a table on node1 (by default, this table will reside in RAM and have a disc copy), write a record to it and then read the record again. The primary key of the table is the first field of the record, i.e. the name.


(node1@node1.21ccw.blogspot.com)9> rd(person, {name, email_address}).
person

(node1@node1.21ccw.blogspot.com)10> mnesia:create_table(person, [{attributes, record_info(fields, person)}, {disc_copies, [node()]}]).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)11> mnesia:transaction(fun() -> mnesia:write(#person{name = "John", email_address = "john@21ccw.blogspot.com"}) end).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)14> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[#person{name = "John",email_address = "john@21ccw.blogspot.com"}]}

(node1@node1.21ccw.blogspot.com)15> mnesia:info(). ...
...
[{'node1@node1.21ccw.blogspot.com',disc_copies}] = [person]
...


What happens when we do the same read on node2? Remember that node has access to the person table only via the network, since it resides in RAM and on disc on node1.

node2@node2.21ccw.blogspot.com)5> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[{person,"John","john@21ccw.blogspot.com"}]}
Nice. Mnesia has transparently read the record from a table that's on another machine :)

Now we decide to copy the table to node2. This requires a single command. Mnesia does the copying of the actual data for you to the other machine, and when you look at the file system on node2, there will now be "person.DCD" file, which is the disc copy of the table.


(node1@node1.21ccw.blogspot.com)15> mnesia:add_table_copy(person, 'node2@node2.21ccw.blogspot.com', disc_copies).
{atomic,ok}



(node2@node2.21ccw.blogspot.com)9> ls("Mnesia.node2@node2.21ccw.blogspot.com").
DECISION_TAB.LOG LATEST.LOG person.DCD
schema.DAT
ok


At this point, when you do a query on the person table, the actual data can come from either node. I'm not sure how Mnesia decides how to distribute the data, that's something to investigate further.

Since the table is resident on both nodes, we can actually delete it from node1, and doing a read on node1 will now read the table over the network from node2:


(node1@node1.21ccw.blogspot.com)23> mnesia:del_table_copy(person, node()).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)19> mnesia:info().
...
[{'node2@node2.21ccw.blogspot.com',disc_copies}] = [person]
...

(node1@node1.21ccw.blogspot.com)18> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[#person{name = "John",email_address = "john@21ccw.blogspot.com"}]}

Cool.

What I've show is how to start up an Erlang/Mnesia node on two machines that are networked together, create tables on either node, and move the tables to other nodes by copying and then deleting them. Mnesia has the ability to configure tables to be RAM only, RAM and disc and disc only, which gives you lots of power for optimisation. Couple this with the fact that you can change your configuration dynamically and you have powerful, dynamically configurable distributed database!

Wednesday, 2 April 2008

Using Rake for Erlang Unit Testing

My previous Erlang/Yaws project, dayfindr.com, is ticking along nicely:
$ yaws --status --id dayfindr
Uptime: 33 Days, 13 Hours, 38 Minutes
The feedback has been mostly positive, although I've had a "That's a very irritating site." comment. Well, you can't please everyone!

Following on the experience gained with dayfindr, I'm starting a new Lyme project which will be using some of the OTP design principles (Think of OTP as Erlang application design patterns). Using OTP will give me hot upgrades, which dayfindr lacks at the moment. The first step that I encountered towards this goal was getting my directory structure to conform to the OTP application structure:


/project [The project root]
/ebin [The compiled .beam files]
/src [The source .erl files]
/include [Include files like record definitions]

I've been using the Makefile that's provided in the Erlang book, which is quite simple and compiles the beam files to the same directory as the source folder, which doesn't comply to the required directory structure. Now, I could just update the Makefile, but Makefiles have a cryptic syntax that I don't really want to spend time learning. Plus, they're difficult to debug in my experience. So I Googled around a bit for erlang Makefiles with little to disappointing success. Then I saw an interesting link:

Building Erlang with Rake

When they make a movie of my life and this moment, you will hear an orchestra and a choir of baritones singing "Eureka" when I click on the link.

Rake is an Ruby equivalent of Make, and more. It took some effort to get it working, since I had rake 0.7.1 on my machine, but trying to find the problem taught me a bit of Ruby in the process. Upgrading to 0.7.3 solved the problem. Sean's Rakefile compiles your src files into the ebin directory very nicely! After tinkering around with Rake, I realised that it's a really nice tool:
  • It has a nice mix of declarative and imperative code. You can define rules (e.g. always compile .erl to .beam), or tasks (which can be imperative, e.g. running unit tests).
  • You can use the full power of Ruby, and don't need to learn Make.
  • It has syntax that is very close to the domain (i.e. it's a good DSL)
  • It's easy to debug, since you can use the normal Ruby puts functions etc.
  • How you set up you dependencies is completely up to you, e.g. you can have different rules for files that conform to different regular expressions.
After I got this working, inspired with confidence, I decided to integrate my unit testing into the Rakefile. I think it's important to note at this point that I have less than a day's Ruby experience, and it was easy to get this working. Hacking a Makefile would probably have taken me hours and hours. The idea is that the test task is dependent on the compile task, so that if you do a "rake test", it will compile anything that's new and run the unit tests. You can just compile with "rake compile".

In order to get this going I created two Erlang files, foo.erl and bar.erl. Here's bar.erl, which contains two functions, and a test (EUnit)for each function. One of the tests will fail:


-module(bar).
-export([bar1/0, bar2/0]).

-ifdef(EUNIT).
-include_lib("eunit/include/eunit.hrl").
-endif.


bar1() ->
ok.

bar2() ->
okk.

-ifdef(EUNIT).
bar1_test_() ->
[
?_assert(ok == bar1())
].
-endif.

-ifdef(EUNIT).
bar2_test_() ->
[
?_assert(ok == bar2())
].
-endif.

Notice that the unit test code is only compiled into the beam file when the EUNIT flag is set. You can set this in the Rakefile. The unit tests are in the same source file, so we can also test non-exported functions.

Now let's look at the Rakefile:


require 'rake/clean'

INCLUDE = "include"

ERLC_FLAGS = "-I#{INCLUDE} +warn_unused_vars +warn_unused_import"

SRC = FileList['src/*.erl']
OBJ = SRC.pathmap("%{src,ebin}X.beam")

CLEAN.include("ebin/*.beam")

directory 'ebin'

rule ".beam" => ["%{ebin,src}X.erl"] do |t|
sh "erlc -D EUNIT -pa ebin -W #{ERLC_FLAGS} -o ebin #{t.source}"
end

task :compile => ['ebin'] + OBJ

task :default => :compile

task :run_tests => [:compile] do
puts "Modules under test:"
OBJ.each do |obj|
obj[%r{.*/(.*).beam}]
mod = $1
test_output = `erl -pa ebin -run #{mod} test -run init stop`

if /\*failed\*/ =~ test_output
test_output[/(Failed.*Aborted.*Skipped.*Succeeded.*$)/]
else
test_output[/1>\s*(.*)\n/]
end

puts "#{mod}: #{$1}"
end
end

The juicy bits are the rule and the "run_tests" task. The rule states "for each file ending in .erl, compile the beam file using erlc and put the beam file in ebin". The run_tests task starts up an erlang runtime for each module, and calls test() for that module. The test output is captured and parsed using regular expressions. I know the Ruby code can be improved, so comments are most welcome.

Here's what happens when I compile and run the tests:


$ rake clean
(in /home/bjnortier/development/project1)

$ rake
(in /home/bjnortier/development/project1)
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/foo.erl
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/bar.erl

$rake run_tests
(in /home/bjnortier/development/project1)
Modules under test:
foo: All 2 tests successful.
bar: Failed: 1. Aborted: 0. Skipped: 0. Succeeded: 1.


At this points I have cleaning, compiling and running the tests for each module. Nice. I'm quite pleased with how it works at the moment, but there's still quite a bit of work to be done:
  • Concatenating the running of the unit tests into one no-shell erlang execution. Actually running the tests are very fast, but the shell termination takes about a second or so. Thus, for each module, there is a approximately a second of extra time. Running all the tests in the same erlang session will require some more text parsing, but it's do-able.
  • Displaying the failed tests. Seeing "Failed: 1" is not very useful for determining what went wrong, so I'll update the parsing to include the failures and errors.
  • Probably change the "run_tests" task to just "test"
  • Continuous integration that hooks into version control.

If there is significant progress I'll post the results. I hope you can use some of it :)

Tuesday, 1 April 2008

The MacbookPro Index

I'm looking into prices for a Macbook Pro, so that I can bolster my own prediction. I'm aware of the fact that for some reason, they are a bit more expensive in the UK than in the US (even without sales tax/VAT). I'm from South Africa, so I can organise for someone to bring me one, but I would never have guessed that it would be cheaper to buy it there than in the UK, and even cheaper than the states if you ignore sales tax/VAT.

Here's a summary of the prices for the 3 models in Us Dollars (USD), Canadian Dollars (CAD), Punds Sterling (GBP), South African Rand (ZAR) and Hong Kong Dollars (HKD). Prices were taken from the official apple websites, except in South African where the ZA Store is one of the official retailers.









15" 2.4GHz15"2.5 GHz17" 2.5GHz
US199924992799
CAD209925992899
GBP129915991799
ZAR184992299925999
HKD154001920021500


Using the following VAT and exchange rates (as on 31/3/2008 and 1/4/2008. A quick investigation revealed Hong Kong at 0%. I could be wrong):







CountryVAT
Exchange Rate
USD0
1
CAD0
1.02
GBP18
0.50
ZAR14
8.12
HKD0
7.79


we get the pre-tax cost in US Dollars:








15" 2.4GHz15"2.5 GHz17" 2.5GHz
US199924992799
CAD2049.82538.092813.05
GBP2197.882705.473043.86
ZAR1997.932483.942807.95
HKD1977.642465.632760.99


At a glance, the UK price is consistently higher than any others, and remember this is without VAT. UK VAT is at 17.5%, the highest of the 5, which will make the post-tax prices even higher.

And if you prefer a pretty picture (using Google Charts API):


And now onto the MacbookPro index. If you haven't heard about the Big Mac index, it is an informal way of measuring purchasing power parity between countries. It can be used to estimate if a currency is over or undervalued, since a Big Mac is pretty much the same in every country. It is calculated by dividing the price of a Big Mac in one country, by that it another country, and comparing this value to the actual exchange rate.

For the MacbookPro index, I've used an average price (non-weighted for simplicity), and here are the results:







CurrencyUnder valuation [%]
CAD0.02
GBP8.91
ZAR-0.1
HKD-1.27


and the graph:


You could conclude that the pound is undervalued by about 9%. I wouldn't. I would conclude that Macs are overpriced in the UK. I think I'll get mine elsewhere...

Wednesday, 19 March 2008

The Mac is Back*

* No, not John McCain, I'm talking about the Apple Mac.

I'm going to make a wild prediction:

The Apple Mac will increase it's market share by more than 50% over the next two years from 4.8% to at least 7.5 %.

I'm basing my speculation on some figures:
And some opinion:

First, the Halo effect. Apple now has a solid (well integrated) product offering with the Mac line, iPods, iPhones and iTunes (music and movies). Consumers who are exposed to any of these products have (mostly) good experiences, which means they get warm and fuzzy inside and buy more Apple products. (but NOT shares. Hmmmm). Why are their product doing so well despite their higher prices? My first axiom:

People are always willing to pay more for a better quality product

Secondly, some perception. I work in technology (creating software), and I would hope that we have a better idea of where the market is heading that the average consumer. I would also like to think that we have some influence over how people will be using their computers in the future, especially though the products and services that we create. I'm seeing more and more Macs being used by the creators of software, especially by those who are more discerning, and my perception is that there is an increasingly positive view of the Mac and Mac OS X in the amongst software creators. Couple that with the buzz that has been created by the iPhone and the iPhone SDK (which delivers a platform for writing software for the iPhone by 3rd party developers), and the future looks rosy.

Lastly, Windows Vista. No matter what Microsoft says, it hasn't been a big success. All the problems associated with it's (much delayed) launch, the quality of the product, the whole "Vista Capable" fiasco etc. has created a unique opportunity for Mac OS X.

Let's wait and see...

Thursday, 6 March 2008

Acid3 Test released

What is Acid3? From Wikipedia: "Acid3 is a test suite that checks how well a web browser follows certain web standards, especially relating to the DOM and JavaScript."

There's some synopsis here and on the wikipedia page

Here are the results of the browsers that I have on my work machine (Windows XP). Note that this is a score out of 100:



No surprise that IE, the bastion of web standards, fails so miserably. I'm actually surprised that Firefox is doing better than Opera, since my subjective opinion at the moment is that Opera renders most pages better than Firefox (except the Google ones, which is not much of a surprise since Google funds a large chunk of Firefox development). Also, the first time I tried the test in Opera 9.26, it crashed the browser. And it just crashed again now with Acid3 open.

All these 4 browsers have big releases coming up, so it remains to be seen who become the new king of the Acid hill. Webkit based browsers like Safari seems to be in the early lead with scores of 90/100 floating around...

Wednesday, 5 March 2008

Apache + Erlang for dynamic web content

It dawned on me this morning that there is a way of using Apache (or any other web server for that matter) with Erlang to create dynamic web content...

How? Use FUSE. There is an implementation of FUSE for Erlang by the Dukes of Erl, called fuserl. If you haven't heard of FUSE, it's a way to make anything you want look like a file system. By using fuserl, you can make an Mnesia database (just one example) look like a file system. You can map queries to directories, and the files that are "listed" can contain the rows of your tables. But you can structure your filesystem in any way you want!

Take for example YoutubeFS: "YoutubeFS enables you to browse your favorite Youtube videos locally on your desktop without going to the youtube website. Just create a youtube account and add videos to your playlists, favorites list or subscribe to different channels. YoutubeFS then enables you to automatically load these videos to a local folder on your desktop. You can then view these videos (using a browser) as if they are local files."

If you extend this idea, you can imagine that you can point your Apache server to a FUSE filesystem location which generates the "contents" of the file system dynamically. To Apache it looks like it's serving normal files, but behind the scenes you can have a distributed Erlang system generating content for you...

This wouldn't work so well if you actually want to use parameters in your http queries, but then I would use Yaws with an appmod if that's what you need.

P.S. You don't have to use Erlang either, you can write FUSE file systems in many other languages...

Monday, 25 February 2008

The time has come

I've been working on an Erlang project at home for a while, and it has reached the first public version! Here it is http://www.dayfindr.com. It's a simple utility that can make your life easier.





It ticks some of the boxes that Paul Graham related recently in Six Principles for Making things:

(a) simple solutions I hope so. This was one of the big design principles - to make it as simple as possible. That's why, for example, it leverages email instead of using some fancy social networking/address book features. There's also no login, since your email address already identifies you. TripIt uses the same idea
(b) to overlooked problems nothing that I know of addresses it
(c) that actually need to be solved, and This tries to help with a real problem that I've encountered myself. Feedback on the idea has been quite positive, so I know it's something that people will find useful!
(d) deliver them as informally as possible Hope so
(e) starting with a very crude version 1, then Relatively crude
(f) iterating rapidly. We'll see!

It's implemented on a Lyme stack (Linux, Yaws, Mnesia and Erlang). This project has generated some interesting questions, that I will relate in a series of posts. For example:

Why Erlang?
Why not ErlyWeb?
Why not Rails?
Why did I abandon the fully functional Lisp version?

So, use it, it's free. If you like it (or more importantly if you don't), lemme know at feedback@dayfindr.com

P.S. To the Internet Explorer users: You will notice that there's a javascript error on the page, because some function is "not implemented", which means you'll see check boxes and not coloured boxes in the calendar. The current versions of Opera, Firefox and Safari have no problem with it. Maybe you should consider an upgrade...

Friday, 22 February 2008

Lyme vs Lamp IV




The first Lyme vs Lamp comparison is here!

Let's recap. I have created a single web page, in Lyme and Lamp, based on database queries. The query is from a single table with 1000 "blog" entries, with Id, Timestamp, Title and Content fields. The web page displays the last 10 entries in the table.

I use Tsung to generate users at a progressive rate, starting from 2 users per second up to 1000 users per second:



Subjecting each web application stack to this test, they compare as follows:


But what does it mean?

It looks like Lyme and Lamp perform the same up until 240 seconds into the simulations, which is where we step up from 66 to 100 requests per second. At about that point, Lyme starts to handle less transactions per second, and the mean transaction time increases dramatically.

Where's the bottleneck? Good question. I suspect it's in the database query, but this is just a shot in the dark. Why could it be? Well, in the PHP version, it's very simple to retrieve the last 10 entries:

SELECT * FROM blog_entries ORDER BY `ID` DESC LIMIT 0 , 10


Whereas with Mnesia, I'm not aware of a better way to retrieve the last 10 rows, other than to iterate backwards from the end of the table:


last_N_entries(_Key, 0, Acc) ->
lists:reverse(Acc);
last_N_entries(Key, N, Acc) ->
{atomic, {Entry, NextKey}} =
mnesia:transaction(
fun() ->
[Entry] = mnesia:read({blog_entry, Key}),
NextKey = mnesia:prev(blog_entry, Key),
{Entry, NextKey}
end),
last_N_entries(NextKey, N-1, [Entry|Acc]).



last_N_entries(N) ->
{atomic, LastKey} = mnesia:transaction(fun() -> mnesia:last(blog_entry) end),
last_N_entries(LastKey, N, []).


I might try some different approaches to the Mnesia query (using ram copies instead of disk copies for example), but I'd rather put some effort into making the experiment a bit more focussed. I'd like to especially see what happens when multiple cores are thrown at the problem, but more importantly, what happens when the transactions become concurrent.

I have now realised that this is scenario is essentially a sequential load test, which will not show up Lyme's strengths. The pages render rather quickly, so there is no concurrent page rendering. Thus, I think the next step should be to create another scenario that will create concurrent page requests...

Tuesday, 19 February 2008

Lyme vs Lamp III



Lyme vs Lamp continues...

I've been spending some time figuring out how Tsung outputs the data, how gnuplot works and how to create my own graphs using the Tsung output data. I now have a graph that shows the throughput rate (Kbits per second) and the arrival rate of users.

When you set up the testing scenarios using Tsung, you can specify the arrival rate of new users. I've set up a simple progression, that changes every 60 seconds, going from 2 users per second up to 1000 users per second. You can see the progression in the graph below.

The page that is being requested is generated from an Mnesia database, with Yaws. The database contains 1000 made-up blog "postings", and the page requested renders the last 5 postings. The machine is quite old, a 2004 laptop actually, and Tsung is also running on the same machine as Lyme.

I've plotted the data throughput rate against the arrival rate. (I've used Inkscape to polish the gnuplot SVG output. I really love it and use it all the time). Here's the result:


As you can see, the throughput rate increases proportionally to the user arrival rate. The "server" manager to handle 200 users/sec, but with 500 requests per second the increase in rate is not proportional. Looks like the max has been reached. Increasing the request rate to 1000 requests shows no difference in throughput. Please comment if you have any more insight into the data.

I should be able to get a PHP version soon, so I can compare it against something...

Thursday, 14 February 2008

Lyme vs Lamp II - The first graph arrives

The first graph of the Lyme vs Lamp debate has arrived!




What does it mean? It means that I've got the LYME stack working with a prototype mnesia-backed web page, and Tsung is doing something. But that's about it for now, the actual results are almost irrelevant...

Lyme vs Lamp I


As part of a presentation at SPA2008 that I'm involved in, I'm doing a bit of load testing on Lamp and Lyme. LAMP is Linux + Apache + MySql + PHP, and LYME is Linux + Yaws + Mnesia + Erlang. (Mnesia is the Erlang database, Yaws is the Erlang web server).

Interestingly, it would seem that in community there is now a precedent to name erlang projects after diseases/conditions, since yaws and lyme are both diseases. AND mnesia used to be called amnesia. Maybe I'll develop a killer app in Erlang called ankylosing spondylitis. Just kidding.

So, there is a rather well-known comparison of Apache and Yaws, but I'd like to go a step further. I'd like to know how a complete web application stack performs under load testing. As an initial comparison, I'll do Lyme and Lamp, and then move on to some others. I would like to have 3 scenarios, a static page only, a dynamic page and a dynamic page with a database backend.

Exactly how to construct these scenarios are still unclear, so suggestions are welcome.

Why test? Why yet another performance benchmark?
- Because it's easy. It's much easier to test something (relatively) objectively, and then wave the results in the air to prove your point. It's much harder to debate the merits of technical choices in the real world, where we have constraints such as budget, skills availability, culture, vested interests etc.
- It creates conversation. Which is a good thing. No flame wars please.

I'll be using Ubuntu 7.10 and Tsung and will publish everything, including configuration file, source files etc. etc.

Updates:
Lyme vs Lamp II - The first graph arrives
Lyme vs Lamp III
Lyme vs Lamp IV

Monday, 11 February 2008

The Visitor Pattern eliminates enums from the Observer Pattern

To avoid boolean parameters, every now and then in a moment of weakness I do something like this:


public void addFile(File file, InputOutput inputOrOutput) {
switch(inputOrOutput) {
case InputOrOutput.INPUT: ... break;
case InputOrOutput.OUTPUT: ... break;
default: assert false;
}
}


Which is not a very good solution. There are at least two problems:

  • I have to maintain an enum

  • There's a possibility that the parameter is null, which creates an opportunity for an error condition


I can achieve the same result by using two functions:

public void addInputFile(File file) {
...
}

public void addOutputFile(File file) {
...
}


This solves the two problems and and also satisfies the readability requirement.

How does this relate to Observers? When I use the Java Observer interface and Observable class, I usually end up with a class defining the chunk of data that describes the change. This chunk is the delta of the object state from the last notification. And a object of this class is received by the observers.

This delta class can take this form:


public class ModelObservedData {

enum Type { ADDED, REMOVED };

// Members that contain the data
...

public ModelObservedData (Type type, Data data) {
...
}

public Type getType() {
...
}

public Data getData() {
...
}

}

and then in the observer

...

update(Observable o, Object arg) {
// Assert on observable source and type or use an if when observing
// multiple sources

ModelObservedData a data = (ModelObservedData)arg;
switch(data.getType()) {
case ADDED: doAdded(data.getData()) break;
case REMOVED: doRemoved(data.getData()) break;
default: assert false;
}

}


I don't find this very elegant. Now, considering the whole multiple dispatch and visitor pattern that I wrote about, there is a more elegant solution.

Essentially, you visit the observed data instead of getting the type of the delta and switching your implementation on the type. The observed data now takes this form:


class ModelObservedData {

...

public static interface IVisitor {

void dataAdded(Data data);
void dataRemoved(Data data);

}

public void accept(IVisitor visitor) {
if (...) {
visitor.dataAdded(addedData);
} else if (...) {
visitor.dataRemoved(removedData);
}
}
}


The client implements the IVisitor interface and the update with the switch becomes an acceptance.


...

update(Observable o, Object arg) {
// Assert on observable source and type or ifs when observing
// multiple sources
ModelObservedData data = (ModelObservedData)arg;
data.accept(this);
}

void dataAdded(Data data) {
...
}

void dataRemoved(Data data) {
...
}


This mechanism doesn't have the switch in the observer, there's no more enum, and the observed data can better encapsulate the state that defines the delta.

The last bit would be to eliminate constructors of the observed data that are difficult to read. For example, you would have one constructor that receives two data parameters, the added and removed data, and stores them as fields. But then the client needs to know the ordering, and have null values in the constructor. Instead, make that constructor private, and create two factory methods that describe the construction process better:


class ModelObservedData {

Data added;
Data removed;

public static ModelObservedData createAdded(Data data)
return new ModelObservedData (data, null /* removed */);
}


public static ModelObservedData createRemoved(Data data) {
return new ModelObservedData (null /* added */, data);
}

private ModelObservedData (Data dataAdded, Date dataRemoved) {
this.dataAdded = dataAdded;
this.dataRemoved = dataRemoved;
}
}


You now have a more elegant solution (I think), with less possibility for error states and no enum to maintain. Also, the switching that each client would have to implement is now implemented only once in the observed class, so there is less duplication.

Thursday, 24 January 2008

Single Dispatch, Multiple dispatch and the Visitor Pattern

Every now and then the question of the Visitor pattern comes up, and sometimes we forget why we need the visitor pattern in the first place! The Visitor pattern is a way to implement double dispatch in single dispatch languages like C++ and Java.

Single Dispatch

The best method that I've discovered for remembering what single dispatch is, is by relating it to compile time vs. runtime function resolution, or overloading vs. overriding.

When you do overloading, the function (or method) that will be executed within the calling context is determined at compile time. Overriding is the ability to choose the method to execute at runtime (it's called single dispatch since you can only use one parameter to the function to determine which one to execute, which is the implicit "this" parameter in curly bracket languages).

Example

I'll illustrate with an example. Let's say that you have application that manages you share portfolio. You have two types of shares, a normal share and a preference share, and you wish to generate a report on the value of your portfolio. The value of a preference share is calculated differently to that of a normal share.

In Java, we'll have something like this:

class Share {
}

class PreferenceShare extends Share {
}


class PortfolioWriter {

public void write(Share share) {
System.out.println("Share");
}

public void write(PreferenceShare preferenceShare) {
System.out.println("PreferenceShare");
}
}


public class SingleDispatch {

public static void main(String[] args) {

PortfolioWriter writer = new PortfolioWriter();

Share share = new Share();
writer.write(share);

PreferenceShare preferenceShare = new PreferenceShare();
writer.write(preferenceShare);

Share shareReferenceToPreferenceShare = new PreferenceShare();
writer.write(shareReferenceToPreferenceShare); // 1
}
}



And here's the output:

$ java SingleDispatch
Share
PreferenceShare
Share


Looking at line marked with "// 1" in the code:
The method that is called on the writer object is determined at compile time, and at compile time the parameter is a Share reference. It's not possible for the compiler to know that the reference actually point to an instance of PreferenceShare!

Lisp

In Lisp, you have multiple dispatch, so you can do something like this


(defgeneric write (generator instrument))

(defmethod write ((g portfolio-generator) (s share))
(format t "portfolio-generator: a share"))

(defmethod write((g portfolio-generator) (p preference-share))
(format t "portfolio-generator: a preference share"))


And when you call "(write ...)" with a report-generator instance and either a share or preference-share instance, you will get the expected result!


In fact, in Lisp, if you have two (or more) type hierarchies you can choose to specialize the method at any point in the hierarchy of any of the parameters. This is very powerful, and the C++/Java object model looks like Lisp's poor cousin in comparison.

Conclusion

You can imagine the limitation of single dispatch when you have a collection of objects that conform to some interface. If you want to do something on each instance (like generate a report), but you don't want to inject that logic into each class, you have to use a visitor pattern to get around the single dispatch limitation.


Wikipedia references

Visitor Pattern
Single Dispatch
Multiple Dispatch

Wednesday, 23 January 2008

Eclipse RCP - Customizing the context menu in the Common Navigator

When you use the Common Navigator as a way to navigate your project, some of your book and web sources will indicate that you should include org.eclipse.ui.navigator.resources.* in you viewer's action binding. This adds all the actions of the common navigator to the context menu. You'll observe that there are quite a few. To see which action providers are included:


  1. Add the Plug-ins view: Window -> Show View -> Other -> PDE -> Plug-ins
  2. In the Plug-ins view, navigate to org.eclipse.ui.navigator.resources and double-click it, which will open the plug-in editor.
  3. Go to the "Extensions" page.
  4. Open org.eclipse.ui.navigator.navigatorContent




Now you'll see 6 Action Providers listed, org.eclispe.ui.internal.navigator.resources.actions.OpenActionProvider, ...actions.NewActionProvider etc, and selecting each will show it's id on the right hand side, e.g. org.eclipse.ui.navigtor.resources.OpenActions for the OpenActionProvider. These are the ids you should use instead of …resources.* in your own plug-in's navigator viewer action bindings to select the ones you want. Say, for example you want all of them except the New provider (because you have your own New con-text menu), your extensions view should look something like this:




Voila! No more New menu!

Eclipse RCP - Where is my editor?

Is you Eclipse plugin/ECP editor not displaying? Have you spend hours checking and rechecking and poring through books and the internet and you're still frustrated?

The solution might be simple! If you don't have an icon specified for the editor, it may not display. The plugin.xml editor doesn't indicate that it's a required field, so I'm assuming it's a bug. A VERY frustrating bug...