21st Century Code Works

Goodbye Blogger

noreply@blogger.com (Benjamin Nortier) — Thu, 04 Feb 2010 17:40:00 +0000

2010 has brought a few changes. I'm now a freelance software engineer with my own company, 1011 Ltd, based in London. This is also ideal opportunity to do something I've been planning for a while - stop using Blogger.

Blogger used to be fine, but it just hasn't kept up. Wordpress, for example, (and there are plenty of others) offers so much more in terms of usability, customizability and administration. Some real pain points for me are:

In Blogger you have to work with a minuscule editing window that you cannot enlarge. Wordpress has a fullscreen mode.
You can download and install Wordpress to your own domain, even modify it as you like, and have ownership of all your posts.
Blogger has doesn't have proper previews using the blog template, whereas the preview in Wordpress is an actual preview.
At this very moment the image feature is broken
The html in uneditable

It feels to me like Google bought Blogger and then just stop working on it. A bit like Microsoft disbanding the IE team in 2004.

Find me at my new blog, www.1011ltd.com/web/blog

[Update: And as you can see, the Spam protection is woeful]

My Google Wave Client

noreply@blogger.com (Benjamin Nortier) — Sun, 23 Aug 2009 15:40:00 +0000

Google released a reference implementation of a Wave client and server in July, written in Java, that you can download and experiment with. When Mickaël Rémond wrote about using the reference implementation with ejabberd, I decided that it was time to get my hands dirty.

I got the reference implementation working, and I could chat with multiple participants in the console clients. However, I wanted to know more about what was happening behind the scenes in the client, and more importantly, the server (and how operation tranforms work).

So I decided to write my own client in Erlang that communicates with the reference server. This way I could investigate what was happening in the client and the server, and gain more knowledge about Wave. It required a bit of reverse engineering, and messing around with protocol buffers (the client communicates with the server using these), but it works nicely.

I've created a video to demonstrate it. Happy watching!

Vimeo link

P.S. I used ngerakines's implementation of Erlang prototocol buffers, and the Nitrogen Web Framework.

P.P.S. I'm going on holiday for 2 weeks, so if I don't reply to comments, please be patient. I will do so when I get back...

Google OS - I believe I was right

noreply@blogger.com (Benjamin Nortier) — Wed, 08 Jul 2009 09:04:00 +0000

disclosure: I own Google shares

I'm not one for blowing my own trumpet, but sometimes it feels good to be right. A month ago I blogged about HTML5, and I said the following:

"I believe it is part of Google's long term strategy to make the browser the de-facto application platform. To displace Windows as the dominant aplication platform. You can see it already with Chrome, how the tabs on the browser have become the new taskbar, and the integration of browser tabs as "applications" into the host OS. For example, features like local file storage and the canvas provide a path for Google Docs to become more like MS Office. You can be sure that some of the drive behind the HTML5 standard is coming from the Google docs team."

Yesterday Google announced Google OS: http://googleblog.blogspot.com/2009/07/introducing-google-chrome-os.html. Google OS will be based on Chrome and a Linux kernel.

From the announcement:

"For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. And of course, these apps will run not only on Google Chrome OS, but on any standards-based browser on Windows, Mac and Linux thereby giving developers the largest user base of any platform."

I will admit that I didn't think they would actually write an OS to support this strategy, but the strategy is clear. Unfortunately we'll have to wait at least a year to see it in action...

Erlang Factory 2009 - New Kids on the Erlang Block

noreply@blogger.com (Benjamin Nortier) — Thu, 02 Jul 2009 20:32:00 +0000

I went to Erlang Factory again this year thanks to my employer, and it was even more exciting than last year's event. There were some great speakers (Joe Armstrong - entertaining as ever, Simon Peyton Jones), I got to meet some interesting people (e.g. Rusty Klophaus, creator of the Nitrogen web framework) and I got to see some awesome projects. Although Erlang is experiencing a surge of interest, it is by no means "new". it has been used for years in robust and highly scalable applications. That is why I refer to the "new" - as opposed to the solid Erlang VM, tools, libraries and frameworks that have been around for at least a decade.

So, wo are these the New Kids on the Block? These guys?

Nice hair! No, not them. Here are some of the New Kids on the Erlang Block:

Heroku

Heroku is a platform for deploying Ruby or Rails web apps. They've built their routing mesh in Erlang. This routing mesh is used to load-balance and route requests to their application work units (dynos). This is a really cool piece of cloud infrastructure.

HyperNumbers

HyperNumbers is a startup that is "doing for numbers what hypertext did for text". Think GoogleDocs, but each cell can be a resource that's located on another spreadsheet. Coming soon.

CouchDB

CouchDB is a non-relational document database written in Erlang, and it is going from strength to strength. Look at this graph from Google Trends:

If you don't know what CouchDB is, you should definitely have a look it. Even is it's just to make you challenge you preconceptions around databases, and relational ones in particular. There is also a book coming out soon. It's open source.

RabbitMQ

RabbitMQ is an open source, high-performance enterprise messaging system, and an implementation of the messaging standard AQMP. It is written in Erlang. I've heard rumors that ActiveMQ (Java) has 20 times more lines of code. (NB: If anyone can give me a reference or link to proof I would be grateful). Update: See comments for line counts.

Nitrogen Web Framework

This is my favourite web framework. It's a real Web 2.0 framework, and it's highly productive. Some of the main features are:

Event-Driven Development
Brainlessly Easy Ajax
Ridiculously Simple Comet
Complex Interfaces: Dragging, Dropping, and Sorting
Flexible Templating
Data Binding
Erlang Power

I've used this to create my own pet Erlang project, opinion8r.com. Check it out. It's also open source.

Smarkets

I didn't go to this talk, but I've been to one given last year at the Erlang London User group. Smarkets is an online betting exchange which allows users to bet on anything they wish. It is implemented in Erlang and close to launch.

Campfire

Campfire is one of the applications from the guys at 37Signals. If you don't know who 37 Signals are, they created Ruby on Rails. The functional C polling for Campfire service that was being used was replaced by an Erlang implementation:

The Erlang implementation is as fast as the C implementation, modular and extensible, and much easier to administer with 1 OS process instead of 80.

ProcessOne - OneTeam Media Server

"OneTeam Media Server is a new project launched by ProcessOne which is a Media server for Flash clients implementing the FLV and RTMP protocol. The project is designed so that it can work hand in hand with ejabberd, ProcessOne Instant Messaging platform."

This one is very exciting. The media server has Flash in the front end, and Erlang in the back. It makes thing like writing video chat in the browser ridiculously easy. And it's open source! Look out for it in July.

Webmachine

Webmachine is the best way to implement a REST interface to your Erlang application that I've seen to date. I've blogged about REST and Erlang in the past, but instead of rolling your own, you should look at this. Especially considering how it handles all the response codes correctly, does all the correct caching responses etc. Another open source project.

And the point is...?

I think it's a very positive indicator for the future of Erlang that it is being used in these new and exciting areas and companies. It is performing very well in the field of high-scalability infrastructure. It is well suited to Cloud computing. But most importantly, very clever and innovative people are using it and loving it. That, above all else, means the future looks good for Erlang.

P.S. I'm sure I missed some great talks again this year, but lucky for me and you, you can find the talks and videos here.

Look out RIAs, HTML5 is coming

noreply@blogger.com (Benjamin Nortier) — Fri, 05 Jun 2009 22:50:00 +0000

Google showing off Wave, has [re]ignited some discussion around HTML5. I read an article on InfoQ, Is Google Wave Going to Have an Impact on RIA/Silverlight? Which struck me as a bit of an odd question. Tim Heuer (in the Silverlight camp) is correct in asking "is HTML5 really what people are talking about here?" and towards the end of the article, the correct question is polled:

"What negative impact will HTML 5/Wave going to have on RIA/Silverlight?" [sic]

At the time the results of the poll were:

Major: 66
Significant: 115
Small: 112
None: 43
Total votes: 336

I was a bit surprised by the poll results. I voted "Major", and I find it hard to understand how someone could vote "None". None?! Really? I do realise that this is partly due to the fact that most people, including me, think they are above average (I just discovered that it is also called the Lake Wobegon effect. Cute). There is also some confusion about including Wave in the question, which has nothing to do with the issue. I'll try and convince you of the potential impact...

Why RIAs exist

The existence of Rich Internet Applications can be explained by the word "Rich" in the name. At some point in the past few years, people started to realise that HTML didn't keep up with the explosion in bandwidth and processing power available to users, and that the web experience could be "richer". This was especially obvious in areas such as not having the ability to to arbitrary on-screen animantions and rendering on a "canvas", the ability to store data locally on user's hard drives and drag-and-drop, amongst others. The lack of these features in HTML created a gap in the market, but only whilst browser technology continues to lag. This gap has been exploited by Flex, Silverlight and JavaFX.

This situation is changing.

Enter HTML5

Some of the features in HTML5 will add some really exciting possibilities to applications running on the browser platform:

Video

Although I wouldn't categorise video as a major driver for RIA technologies, it is important to mention in relation to Adobe flash, which is the dominant technology for delivering internet video. YouTube is already delivering non-flash internet video in mobile devices. The introduction of the tag will reduce the pressure of deploying flash on every platform, which could have an effect on the usage of Flex. Consider that Flash is not available on the iPhone: if you have a choice of developing your application on just one technology (HTML5) for both desktop and mobile, or choosing 2, which would cost less?

Canvas drawing (immediate mode drawing)

Canvas drawing will enable application developers to fill a gaping hole in native browser capabilities - gaming. The gaming industry is bigger than the movie industry, and the availability of a virtually universally available standard platform will not got unnoticed by game developers. Of course this capability doesn't only apply to gaming.

Local storage

As long as there's no capability for storing data offline, web applications continue to be considered the poor cousin of desktop applications. Google documents are not availble offline without Google Gears. Having local storage available in a standardised way is a big attraction for application developers. Gears might work for Gooogle, but not everyone is so keen on having a dependency on a non-standard technology.

Some other disavantages to RIAs which could favour HTML5 applications

- RIAs are usually everything or nothing. They don't play well with embedding RIA functionality into existing web pages. HTML5 allows you to grow your application, creating a richer and richer experience as you go along.
- Some RIAs have proprietary technologies.

- RIAs require an extra deployment step on top of deploying the browser. Browsers implementing HTML5 will provide a more standardised way of deploying applications to users, also in the corporate environment where locked-down systems are common. Choosing a technology that doesn't require another installation step is a big attraction for corporate IT departments.

In the long term

There another significant point to make.

I believe it is part of Google's long term strategy to make the browser the de-facto application platform. To displace Windows as the dominant aplication platform. You can see it already with Chrome, how the tabs on the browser have become the new taskbar, and the integration of browser tabs as "applications" into the host OS. For example, features like local file storage and the canvas provide a path for Google Docs to become more like MS Office. You can be sure that some of the drive behind the HTML5 standard is coming from the Google docs team.

I'm not the only one thinking along these lines: http://industry.bnet.com/technology/10002039/googles-long-shot-at-kicking-microsoft-off-the-desktop/

In closing

I'm not trying to say that HTML5 will eliminate RIA technologies, but I do believe that thinking that its impact on RIA will be "small" or "none" is absolutely ridiculous.

There are some big players pushing the browser to become the dominant application platform. HTML5 is a big step in that direction.

If you disagree, I look forward to your comments!

The Law of Conservation of Misery

noreply@blogger.com (Benjamin Nortier) — Wed, 20 May 2009 18:29:00 +0000

At university I had a professor who would jokingly refer to his Law of Conservation of Misery:

The total amount of misery in a system is constant

which implies that when you try and decrease the misery in one aspect of a system, you will increase the misery in some of the other aspects.

There are many types of misery. In software engineering projects these could include non-maintainability, non-performance and budget constraints.

Personally, I think people are too prone to think that there are zero-sum monsters lurking under each bed, so my own take on the "law" would be:

The total amount of misery in a system is > 0

Which implies that you can make tradeoffs that decrease the total amount of misery, but you can never eliminate it completely. A bad implementation of a system will have a higher total amount of misery, and a good one will have less, but you will always have at least a little bit!

How to send email via Gmail using Erlang

noreply@blogger.com (Benjamin Nortier) — Tue, 19 May 2009 18:19:00 +0000

One of my pet projects, www.dayfindr.com, integrates with email to send notifications to users.

I use Google Apps for email infrastructure, so you need an SMTP client that supports TLS. At the time, I couldn't find a simple Erlang SMTP client that could handle TLS, so I used a command-line SMTP client.

For my new pet project, for want of a better name temporarily called The Isabelle Project, I need to add some email functionality. This time I would prefer to use an Erlang solution with proper error handling and logging.

I looked at the SMTP protocol on Wikipedia, and it didn't seem to difficult. Erlang's built-in ssl module also seemed to support TLS. So, with a bit of trial and error, here's the result:


-module(smtp).
-export([connect/0]).

connect() ->
    {ok, Socket} = ssl:connect("smtp.gmail.com", 465, [{active, false}], 1000),
    recv(Socket),
    send(Socket, "HELO localhost"),
    send(Socket, "AUTH LOGIN"),
    send(Socket, binary_to_list(base64:encode("___@gmail.com"))),
    send(Socket, binary_to_list(base64:encode("johngalt"))),
    send(Socket, "MAIL FROM: <___@gmail.com>"),
    send(Socket, "RCPT TO:<___@gmail.com>"),
    send(Socket, "DATA"),
    send_no_receive(Socket, "From: <___@gmail.com>"),
    send_no_receive(Socket, "To: <___@gmail.com>"),
    send_no_receive(Socket, "Date: Tue, 15 Jan 2008 16:02:43 +0000"),
    send_no_receive(Socket, "Subject: Test message"),
    send_no_receive(Socket, ""),
    send_no_receive(Socket, "This is a test"),
    send_no_receive(Socket, ""),
    send(Socket, "."),
    send(Socket, "QUIT"),
    ssl:close(Socket).

send_no_receive(Socket, Data) ->
    ssl:send(Socket, Data ++ "\r\n").


send(Socket, Data) ->
    ssl:send(Socket, Data ++ "\r\n"),
    recv(Socket).

recv(Socket) ->
    case ssl:recv(Socket, 0, 1000) of
 {ok, Return} -> io:format("~p~n", [Return]);
 {error, Reason} -> io:format("ERROR: ~p~n", [Reason])
    end.

And the output from the Erlang shell:


3> application:start(ssl).
ok
4> smtp:connect().
"220 mx.google.com ESMTP y37sm613282mug.19\r\n"
"250 mx.google.com at your service\r\n"
"334 VXNlcm5hbWU6\r\n"
"334 UGFzc3dvcmQ6\r\n"
"235 2.7.0 Accepted\r\n"
"250 2.1.0 OK y37sm613282mug.19\r\n"
"250 2.1.5 OK y37sm613282mug.19\r\n"
"354  Go ahead y37sm613282mug.19\r\n"
"250 2.0.0 OK 1242683885 y37sm613282mug.19\r\n"
"221 2.0.0 closing connection y37sm613282mug.19\r\n"
ok

The only tricky bit is that for the AUTO LOGIN, the received text and the username and password you send is base-64 encoded. By default the connect is active=false, which means the responses are send to the creating process directly. Using passive mode requires explicit receiving of the response using ssl:recv/2

You'll have to handle errors better if you use this in production, but the basic protocol is pretty straightforward...

Is a Square a Rectangle?

noreply@blogger.com (Benjamin Nortier) — Fri, 02 Jan 2009 11:31:00 +0000

I pre-ordered the Clojure book today, so I downloaded Clojure and had a peek at something that I know is interesting - "multimethods". I came across the following example statement:

(derive ::square ::rect)

i.e. a Square IS-A Rectangle in the Object-Oriented sense.

This rang a bell for me. Not a nice melodious one, but one that's a little off-key. A good teacher asked some colleagues and me the question "Is a Square a Rectangle?" a few years ago, and the answer wasn't all that clear ti us. But he showed us a realitively easy way to determine to some extent the validity of any IS-A relationship.

I will try and show you this principle by way of actually designing this hierarchy. First we design a simple Rectangle in Ruby:


class RectangleTest < Test::Unit::TestCase
  def test_width_height

    # Given
    r = Rectangle.new(1,1)

    # Then
    assert_equal 1, r.width
    assert_equal 1, r.height

    # When
    r.width = 5

    # Then
    assert_equal 5, r.width
    assert_equal 1, r.height

    # When
    r.height = 2
    
    # Then
    assert_equal 5, r.width
    assert_equal 2, r.height

  end
end

If you think it's strange that I use a test as a design, I suggest you read this. I'm not even going to show the implementation since it's so simple. Next we will design our Square class:


class SquareTest < Test::Unit::TestCase
  def test_width_height

    # Given
    s = Square.new(3)

    # Then
    assert_equal 3, s.width
    assert_equal 3, s.height

    # When
    s.width = 5

    # Then
    assert_equal 5, s.width
    assert_equal 5, s.height

    # When
    s.height = 2
    
    # Then
    assert_equal 2, s.width
    assert_equal 2, s.height

  end
end

And you can see that we make sure that when either the width or height has been set, we get the same value for both the width and height respectively. The same goes for the single-parameter constructor. Again, the implementation is trivial.

Now, you might think that this design is OK, but it's not. We have to ensure that our inheritance doesn't break the Liskov Substritution Principle.

From Wikipedia:

"Liskov's notion of 'subtype' is based on the notion of substitutability; that is, if S is a subtype of T, then objects of type T in a program may be replaced with objects of type S without altering any of the desirable properties of that program (e.g., correctness)."

This means that for our designs, the unit test for the Rectangle should pass when I substitute the Rectangle instance with a Square instance. Will it? No it won't. Because in the rectangle test, we ensure that when the width is set, the height is not affected, and vice versa. Using a Square instance here will break the test.

Another way of looking at it is to think of the contract of the setters, and the pre and post conditions of those contracts. According to the Liskov princple, (again, from Wikipedia):

Preconditions cannot be strengthened in a subclass.
Postconditions cannot be weakened in a subclass.

If we design the Square to force the width to be set when setting the height (and vice versa), we are weakening the postconditions of the Rectangle methods. I.e. the post condition that states that either the width or height should not be altered when setting the height or width respectively has been weakened.

In conclusion, here we have a Square that is NOT a Rectangle. The lesson to be learnt is that (unfortunately for us), IS-A in the real world or other domains does not necessarily imply IS-A in the OO world!

P.S. If you remove the setters from the Rectangle, and only allow the fields to be set during construction, there is no violation of the Liskov Substitution Principle :)

Does Robert Mugabe use Powerpoint?

noreply@blogger.com (Benjamin Nortier) — Wed, 12 Nov 2008 09:07:00 +0000

I think most people will agree that Robert Mugabe is a terrible dictator who has scant disregard for the people he is supposed to be serving. The same goes for the likes of Stalin, Hitler, Idi Amin etc. That being said, I think most people will also agree that one of the reasons of their rise to power was specifically their ability to influence people. In other words

People listen[ed] to them, and they're not using Powerpoint

My experience is that every time someone want to say something to more than one person, there has to be a Powerpoint presentation to go along with it. And more often than not, this attempt at communication fails miserably, or succeeds only in some rudimentary fashion.

Some of the causes of these failures are:

People are reading the slides, so they're not listening to the speaker.
The presenter is reading the slides, so there is no connection with the audience.
The bullet-pointed slides are so mind-numbingly boring that the audience members have gone to their happy place or are thinking of what to cook/order/kill for dinner and of course, they're not listening.

In contrast, I'll mention two very successful entrepeneurs and their presentation styles: Steve Jobs (Apple, Pixar, General Motors?) and Elon Musk (PayPal, Tesla Motors, SpaceX). In business it is absolutely essential that you can effectively communicate your ideas to people. And when you audience is holding the purse strings, and you can't convince them of your business ideas, you're doomed to become a (perhaps very successful) hobbyist.

I went to a talk by Elon Musk on his company SpaceX a few months ago. The projector had been set up and the presentation was ready to proceed. The first slide was already on the screen, with text along the lines of "Elon Musk, SpaceX" with very nice picture of a rocket (on a Pacific island if I remember correctly).

Elon Musk came in and talked a bit. After a few minutes he proceeded to questions, and these went on for more than an hour. The first slide was the only slide. Maybe the host created it when he realised there were no slides. "Are you crazy? No slides?! But how will you...."

The talk was very effective, he had communicated his points, and gave the audience exactly what they wanted, since they were asking him what they wanted to know!

The second example is Steve Jobs. I remember watching the first iPhone Keynote speech and how I got swept up in the presentation. It was a highly effective communication (helped by a great product). Let's look at some of the slides he used:

Do you see lots of bullet points? The last slide has the most text on it, but that slide is in the minority, and the bullets are actually very short. The first two slides had simple images, and minimal text, and I can assure you that people were listening to what he was saying!

Do you think the response would have been the same if his slides looked like this:

instead? I'm asleep already.

My highly opinionated advice for presentations would be:

Don't use PowerPoint/Keynote/[insert slide software here] to write your presentation. Decide what you want to say and how you want it to flow and only prepare your slides once you know the content of your presentation.
Don't use your slides as notes, have seperate notes if you don't know the topic that well or need some backup information.
Don't read you slides.
Don't have lots of bullet points on each slide.
You don't need to use slides. Sometimes a white board is good enough, and in most cases it is probably better.

What do you think?

Iterative and Agile - the Venn Diagram of truth

noreply@blogger.com (Benjamin Nortier) — Mon, 08 Sep 2008 12:31:00 +0000

I'm seeing a bit of this lately:

"Iterative software development is Agile software development."

and I have to say that I disagree quite strongly. The Venn diagram actually looks more like this:

This post has been prompted by two things. First, a few weeks ago a SCRUMmer made some comments along the following lines:

"We don't allow changing the iteration plan once the iteration has begun"
"All the requirements for the stories for the iteration must have been gathered before it starts."
"If the customer want something else, they have to wait for the next iteration"

And some articles/blogs about iteration lengths that jogged my memory about what I disagreed with. The root of my discomfort in equating iterative development to agile development is because

agile software development is a set of values, it is not a process.

In fact, one of the core values of Agile development is to not value you process too much. There are some processes that try to embody Agile values, and they have different interpretations and practicalities around those values, but these processes are not what defines Agile development.

Here's a reminder of what it means to be agile:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

It's very possible to do iterative development without embodying any of these values. Treating each iteration like a waterfall with the usual waterfall suspects (requirements up front, little room for manoevering, opressive specification and documentation) is a case in point. And to me, valuing your plan over your customer by not responding to their needs is not Agile.

You are, of course, invited to disagree...

Me and my dog on Google Chrome

noreply@blogger.com (Benjamin Nortier) — Wed, 03 Sep 2008 16:29:00 +0000

Like every man and his dog I feel compelled to say something about Chrome, Google's new browser. Most impressions I have read have been favourable, and my personal experience so far is very positive. Most reviews I have scanned have been thumbs-up, and have focussed on the usual suspects, i.e. rendering and javascript performance, stability, usability etc. etc.

For me, there is something subtle and extremely powerful that comes with Chrome. It fields a combination of features with very powerful implications. A "whole being more that the sum of the parts" kind of situation.

In the past year or two there has been a lot of focus in the "richer" web experience, and on ways of getting the web to feel more like the desktop. Newish technologies like Flex and Silverlight have received significant attention, and DHTML (Javascript+CSS+DOM+HTML) seems to have become the cousin that you are obliged to invite to your wedding party (or application party), but don't really want to. I believe that part of the reason for this has been Javascript performance, but there are several efforts underway (e.g. TraceMonkey and now Chrome's V8 Javascript engine) that will bring huge leaps in performance to Javascript. Also, I prefer to support non-proprietary technologies such as DHTML over vendor-based alternatives like Flex and Silverlight.

The second feature that's important in Chrome is process isolation for tabs. This enables each tab to be completely isolated from other processes. One tab going down doesn't take the whole browser with it (which is something that has plagued my Firefox quite a bit).

Thirdly, we have "application-like" OS integration in Chrome. Chrome has the ability to make a single tab look like an application window, with normal window decorations:

In your taskbar it looks like an application, there is quick lauch icon for you app:

There's also a Start Menu item (not shown) and a Desktop icon:

The combination of these three is a significant step towards blurring the lines between the desktop and the web. Chrome is an application platform with (potentially) desktop-like application performance (DHTML + V8), process isolation (a key feaure of an application platform) and the web apps look like desktop apps (Operating system integration). And it's all standards based.

There have been some rumours of a Google OS...

You're looking at it.

Scalaris Released

noreply@blogger.com (Benjamin Nortier) — Thu, 24 Jul 2008 11:46:00 +0000

If you were at the Erlang Exchange in London last month, you should know that one of the hottest talks was given by Alexander Reinefeld, "Building a transactional distributed data store with Erlang"

Joe Armstrong seems to like it too:

"I might be wrong, but my gut feeling is that what Alexander Reinefeld showed us will be the first killer application in Erlang."

"So my take on this is that this is one of the sexiest applications I've seen in many a year. I've been waiting for this to happen for a long while. The work is backed by quadzillion Ph.D's and is really good believe me."

- http://armstrongonsoftware.blogspot.com/2008/06/itching-my-programming-nerve.html

Well, seems like Scalaris has been released! The link went up on the website in the last 24-48 hours. I haven't had the time to look at it yet, but you can grab it here.

Notes from Erlang Exchange

noreply@blogger.com (Benjamin Nortier) — Wed, 02 Jul 2008 11:09:00 +0000

I was at the Erlang Exchange in London last week, which I really enjoyed. It was really great to see Joe Armstrong, and to see the faces behind the projects, e.g. Claes Wikström (Yaws, Bluetail, Kreditor, Tail-f), Steve Vinoski, Eric Merrit and Martin Logan (Erlware), Matthias Radestock (LShift & RabbitMQ), Mickaël Rémond (ejabberd) etc. etc.

Something that Joe said reminded me of a comment that Keith made about 6 months ago. I think this is something that gets lost in the hype around Erlang and multicore computing:

Erlang was designed to program fault tolerant systems.

It was not designed to program multicore computers. It was designed 15-20 years ago when multicore-everywhere was not even on the horizon. As a side-effect, it is a good language for multicore systems, since the only way to actually achieve fault tolerance is to have independant processes (read here for some more on this).

I think it's important for us as to remember to advocate this, and instead of marketing Erlang/OTP as "the application system for multicore", we should be marketing it as "the application system for fault tolerance". Oh, and by the way "this is one of the few systems that will actually use those cores that are coming your way".

The second point that I have been pondering, was brought on my Gordon Guthrie's talk about Erlang/OTP vs Google Apps as an application engine. At this moment, there aren't many application systems to choose from, Erlang/OTP, Google Apps and Amazon EC2 come to mind. In the context of the talk, an Application System (A/S) would be something that takes care of a lot of the non-functional requirements of you system, e.g. reliability, scalability (+distribution) etc.

For some businesses, using Google Apps or EC2 would be a good fit, but then you're tied into the value chain of that business, and in some instances you just cannot (for legal and other reasons) put your data on servers you don't own. If you go the OTP route you have full control over your application(s), but there's a lot of extra infrastructure that you will have to supply yourself.

If you start thinking of Erlang/OTP in this way, you realise that the terms should be reversed and it should actually be OTP/Erlang. The platform is actually the important thing. The platform is the thing that gives you the reliability, scalability, distribution and hot code swapping etc. that you require. Erlang becomes this awesome language that you use to build things on the platform. Looking at things from a different perspective can lead to some interesting insights.

If I'm standing at this vantage point, heated discussions like the recent meme-storm in the blogosphere of "Erlang vs Scala" become much less relevant. Is there a platform that gives you reliability, scalability, distribution, hot code replacement that uses Scala as a language? No? Oh. What a pity...

Migrating a native Erlang interface to RESTful Mochiweb (with a bit of TDD)

noreply@blogger.com (Benjamin Nortier) — Tue, 17 Jun 2008 13:07:00 +0000

The title is a bit of a mouthful, but it does contain in essence what I will show you:

1. I will convert an existing native erlang CRUD interface (a simple client-server) to use a HTTP layer.
2. The HTTP calls will be RESTful.
3. I will be using Mochiweb.
4. I will show you how to do a bit of Test-Driven-Development (TDD) for this exercise using EUnit.

As an introduction, I suggest you read Steve Vinoski's "RESTful Services with Erlang and Yaws", and "A RESTful web service demo in yaws" by Nick Gerakines. They are both excellent posts, and Nick's post has a lot of interesting details, including some OTP goodness. Also check out Daryn's posts for a Rails twist on this topic.

I will NOT be showing much error handling, simply returning a 501 response if the REST Url has
some mistake in it.

This is the plan:
1. We will start with a working set of tests for a native CRUD interface. I will not show you the actual implementation, but it is simple enough to do yourself using a process dictionary.
2. I will show a very simple REST interface with Mochiweb, that just returns the request method type as a plaintext string. This will be tested using Inets, the native Erlang OTP internet module.
3. We will write a translators to and from plaintext and native erlang terms (again, unit tested).
4. We will combine all these into the REST interface, tested using Inets again.

Right, here is the unit tests for the CRUD interface:

crud_test_() ->   
  {setup,
   fun() -> start() end,
   fun(_) -> stop() end,
   fun(_) ->
           [
            ?_assert(ok == create(#person{id = 1})),
            ?_assert(already_exists == create(#person{id=1})),
            ?_assert(#person{id=1} == retrieve(1)),
            ?_assert(ok == update(#person{id = 1, name="Ben"})),
            ?_assert(#person{id=1, name="Ben"} == retrieve(1)),
            ?_assert(ok == delete(1)),
            ?_assert(undefined == delete(2)),
            ?_assert(undefined == update(#person{id = 2}))
           ]
   end}.

The CRUD interface is container in a module "crud", and start() and stop() are methods to start and stop the CRUD server. For EUnit, you can define a test with a setup, teardown and tests, and this is the form that I've used here. There are many forms of tests for EUnit, and I suggest you consult the EUnit documentation (contained in the EUnit distribution).

EUnit automatically creates a test() function on the module when you include "eunit.hrl", so let's run the crud tests:


75> crud:test().
All 8 tests successful.
ok

We know that are CRUD interface is working OK.

Onto Step 2! If you're not familiar with Mochiweb, it's a lightweight web server developed by the guys at Mochi Media. It's very easy to integrate into you application, and has very little configuration.

Here are the tests for the simple Mochiweb demo:


-define(URL, "http://127.0.0.1:8888").

rest_server_test_() ->
    {setup,
     fun() -> inets:start(), start_simple() end,
     fun(_) -> inets:stop(), stop_simple() end,
     fun(_) ->
             [
              ?_assert(http_result('GET') =:= "GET"),
              ?_assert(http_result('PUT') =:= "PUT"),
              ?_assert(http_result('POST') =:= "POST"),
              ?_assert(http_result('DELETE') =:= "DELETE")
             ]
     end}.

parse_result(Result) ->
    {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
    ResultBody.

rest_server_test_() ->
    {setup,
     fun() -> inets:start(), start_simple() end,
     fun(_) -> inets:stop(), stop_simple() end,
     fun(_) ->
             [
              ?_assert(http_result('GET') =:= "GET"),
              ?_assert(http_result('PUT') =:= "PUT"),
              ?_assert(http_result('POST') =:= "POST"),
              ?_assert(http_result('DELETE') =:= "DELETE")
             ]
     end}.

start_simple() starts the simple version of the server and stop_simple() stops it. We're just using the base URL, and the server just returns a plain-text string of the request method. The http:request() functions are part of Inets (documentation here). You will notice that the put and post functions have extra parameters (which we will use later for the request body).

Here's the solution:


start_simple() ->
    mochiweb_http:start(
      [{ip, "127.0.0.1"},
       {loop, {?MODULE, simple_response}}]).
stop_simple() ->
    mochiweb_http:stop().
    

simple_response(Req, Method) ->
    Req:ok({"text/plain", atom_to_list(Method)}).
simple_response(Req) ->
    simple_response(Req, Req:get(method)).

We've started Mochiweb, and told it that the "simple_response" function in the current module should be used to handle requests. It takes one parameter, which contains the request data. The data can be queried with different methods to extract data from it, e.g. Req:get(method), gives you the method used.

The Req:ok() function is used to respond to a request, and we simply return plain text (with the text/plain MIME type).

And let's run the tests:


77> rest_server:test().
...
All 4 tests successful.
ok

Nice. The next bit of code we need is to translate Erlang terms to and from plain text. We'll use Base64 encoding for this. Erlang also has very hand term to binary and binary to term functions that we can use to achieve the translation.

The tests for the translation:


term_to_plaintext_test_() ->
    A = anatom,
    B = {a, 23, "abc"},
    C = {props, [{a,3},{b,4}]},
    [
     ?_assert(A == ptt(ttp(A))),
     ?_assert(B == ptt(ttp(B))),
     ?_assert(C == ptt(ttp(C)))
    ].

Where "ttp" would be "term_to_plaintext" and "ptt" is
"plaintext_to_term". Here's the implementation:


ttp(Term) ->
    base64:encode_to_string(term_to_binary(Term)).

ptt(PlainText) ->
    binary_to_term(base64:decode(PlainText)).

And the test result:


77> rest_server:test().
...
All 7 tests successful.
ok

(7 tests, since we have the first 4 and now the extra 3). Now things are getting a bit more complicated. We now have to ensure that the response to the request is converted from the native Erlang terms to plaintext, this result is then returned, and we also need a function to convert the request result into the native form.

Here we go:


result_to_terms(Result) ->
  {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
  ptt(ResultBody).

rest_api('GET', Path) ->
    Result = http:request(get, {?URL ++ Path, []}, [], []),
    result_to_terms(Result);
rest_api('DELETE', Path) ->
    Result = http:request(delete, {?URL ++ Path, []}, [], []),
    result_to_terms(Result).

rest_api('POST', Path, Data) ->
    Result = http:request(post, {?URL ++ Path, [], [], ttp(Data)}, [], []),
    result_to_terms(Result);
rest_api('PUT', Path, Data) ->
    Result = http:request(put, {?URL ++ Path, [], [], ttp(Data)}, [], []),
    result_to_terms(Result).

crud_server_test_() ->
    {setup,
     fun() -> crud:start(), inets:start(), start() end,
     fun(_) -> stop(), inets:stop(), crud:stop() end,
     fun(_) ->
      [
       ?_assert(ok == rest_api('POST', "/person", #person{id = 1})),
       ?_assert(already_exists == rest_api('POST', "/person", #person{id = 1})),
       ?_assert(#person{id=1} == rest_api('GET', "/person/1")),
       ?_assert(ok == rest_api('PUT', "/person/1", #person{name="Ben"})),
       ?_assert(#person{id=1, name="Ben"} == rest_api('GET', "/person/1")),
       ?_assert(ok == rest_api('DELETE', "/person/1")),
       ?_assert(undefined == rest_api('DELETE', "/person/1")),
       ?_assert(undefined == rest_api('PUT', "/person/1", #person{id = 2}))
      ]
     end}.

There is a bit of duplication here, but for illustration I've kept the functions seperate. You will notice that the data is converted to plaintext for the PUT and POST requests. result_to_terms() converts the HTTP result from the plain text to the native Erlang terms.

You can scroll back to the original CRUD tests, and notice that the tests do exactly the same thing as on the CRUD interface, but we would have to do a "GET" + "/person/1" to get person 1, instead of doing a crud:retrieve(1).

The mapping between the CRUD and REST method are as follows:
GET <=> RETRIEVE
POST <=> CREATE
PUT <=> UPDATE
DELETE <=>DELETE

Here's the final product:


start() ->
    mochiweb_http:start(
      [{ip, "127.0.0.1"},
       {loop, {?MODULE, crud_response}}]).
stop() ->
    mochiweb_http:stop().


crud_response(Req, 'GET', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Response = crud:retrieve(Id),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'DELETE', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Response = crud:delete(Id),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'POST', "/person") ->
    Body = Req:recv_body(),
    Person = ptt(Body),
    Response = crud:create(Person),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'PUT', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Body = Req:recv_body(),
    PersonWithNewValues = ptt(Body),
    UpdatedPerson = #person{id = Id, 
                            name = PersonWithNewValues#person.name, 
                            email_address = PersonWithNewValues#person.email_address},
    Response = crud:update(UpdatedPerson),
    Req:ok({"text/plain", ttp(Response)});
        
crud_response(Req, _Method, Path) ->
    Req:respond({501, [], Path}).

crud_response(Req) ->
    crud_response(Req, Req:get(method), Req:get(path)).

Once again, there is some duplication, but it's easier to grasp when we split the functions. The GET and DELETE requests are simple to handle, since we just convert the Id string to an integer (which could throw an exception, you would have to handle that in some way), and call the CRUD interface.

POST is not much more complicated, we just get the body of the request using Req:recv_body().

The PUT function has a problem. There is duplication of the Id of the person, in both the URL "/person/1", and the actual term, #person{id=1,...}. I'm not sure how to handle this, comments are welcome. Perhaps you can generate an error response if the Ids don't match. Or you can make sure the posted record does NOT contain an Id field, and return an error if it does.

The solution as I've given it, uses the fields in the record, and ignores the Id of the record, using the Id in the URL instead.

The last function generates an error if the request could not be matched.

And the moment of truth:

81> rest_server:test().
...
All 15 tests successful.
ok

Looks so simple now, but I can asure you that it took some effort to get all the tests to pass!

Well I hope I've shown you something that you didn't know before, or even encouraged you to learn some Erlang.

Comments and criticisms are most welcome :)

Restful Charts

noreply@blogger.com (Benjamin Nortier) — Sun, 08 Jun 2008 21:19:00 +0000

What would you normally expect of a graph-generation library? I'm talking about the pie charts, bar charts etc. You would normally expect an output to some file format, it being PNG or SVG if you're lucky. BUT, gchartrb doesn't do this, it creates a URL that you would use to GET the graph from the Google charts API.

So what?

Well, it's interesting because it means that some people at least consider that generating the graph using a REST API is as reliable as disk, and as convenient as a local library. I've used Google charts before, but I've never thought about integrating it into another program as a service, and I think it's pretty neat. And easy.

I'm placing my chips on REST.

Parallel Quicksort in Erlang - Part II

noreply@blogger.com (Benjamin Nortier) — Wed, 07 May 2008 12:14:00 +0000

In my previous post, Jaksa pointed out that there's a problem, in that although there may be multiple processes, they will be blocked and waiting (sequencially) for the first peval to complete, then the second peval, etc. Which will result in a pseudo-parallel depth-first quicksort, instead of a breadth-first quicksort. This is the offending line:


peval(fun pqsort/1, Left) ++ [Pivot] ++ peval(fun pqsort/1, Right).

I was hoping Erlang would be clever enough to execute the two functions in parallel, but that's a fairly unreasonable (and completely incorrect) assumption!

But how to fix it? There is a parallel implementation of the map function in Joe's book, and Montsamu has a variation of it on his blog:


-module(plists).
-export([pmap/2]).

pmap(F, L) ->
    S = self(),
    Pids = lists:map(fun(I) -> spawn(fun() -> pmap_f(S, F, I) end) end, L),
    pmap_gather(Pids).

pmap_gather([H|T]) ->
    receive
        {H, Ret} -> [Ret|pmap_gather(T)]
    end;
pmap_gather([]) ->
    [].

pmap_f(Parent, F, I) ->
    Parent ! {self(), (catch F(I))}.

The pmap_gather function is the interesting one. It will traverse the list of Pids, and wait for each to send back its result. Using pmap, we update pqsort as follows:


pqsort([]) -> [];
pqsort([Pivot]) -> [Pivot];
pqsort([Pivot|Rest]) ->
    io:format("+", []),
    Left = [X || X <- Rest, X < Pivot],
    Right = [Y || Y <- Rest, Y >= Pivot],
    [SortedLeft, SortedRight] = plists:pmap(fun pqsort/1, [Left, Right]),
    io:format("-", []),
    SortedLeft ++ [Pivot] ++ SortedRight.

And with a perfectly unbalanced (for quicksort) list to sort, we expect 1 process, then 3 as the first spawns 2 new ones, then down again to zero:


(emacs@21ccw.blogspot.com)47> pqsort:pqsort([4,2,1,3,6,5,7]).
+++---[1,2,3,4,5,6,7]

For the same example we had last time, sorting 100 random numbers, the number of concurrent process graph during execution looks like this:

Much better!

Parallel Quicksort in Erlang

noreply@blogger.com (Benjamin Nortier) — Thu, 24 Apr 2008 12:36:00 +0000

Jaksa is doing some posts on parallel graph algorithm in Java using fork/join, and he mentions quicksort:

"The equivalent of the hello world for parallel languages is the quicksort algorithm"

Now I'm tempted to say something like "Oh yea? You know how many lines of code quicksort is in Erlang? 3!" Yes really:


qsort([]) -> [];
qsort([Pivot|Rest]) ->
    qsort([ X || X <- Rest, X < Pivot]) ++ [Pivot] ++ qsort([ Y || Y <- Rest, Y >= Pivot]).

But this would be wrong, as the terseness of the quicksort implementation says more about the power of list comprehensions than about Erlang's concurrency-supporting language features.

So then let's see how you could implement a concurrent quicksort in Erlang, which will say something about the way in which we can implement concurrency in Erlang. We'll start with the original Wikipedia quicksort from above, and alter it a little bit to make it more obvious what I'm going to do next:


qsort([]) -> [];
qsort([Pivot|Rest]) ->
    Left = [X || X <- Rest, X < Pivot],
    Right = [Y || Y <- Rest, Y >= Pivot],
    qsort(Left) ++ [Pivot] ++ qsort(Right).

Note the two calls to qsort at the end. Since there is no shared state between these two functions, they can be executed in parallel, and the results can be concatenated to yield the sorted result. Depending on the distribution of the input, each of these calls could be split up into 2 more qsort calls etc. etc. Note that you could do a hybrid recursive-tail-recursive solution which sorts the smallest partition using normal recursion and the largest using tail-recusrion, but that is not the aim here.

Let's assume that we already have a function that executes any input function in a separate Erlang process called "peval". (Note Erlang processes are extremely light-weight and take very little time to construct. They're are not modelled as operating system processes). peval takes two arguments, a function to execute, and the arguments to the function. When called, it looks like a normal function call:


(emacs@21ccw.blogspot.com)38> pqsort:peval(fun(X) -> X*X end, 13).
169

But, in the background, it spawns a process and the function is evaluated on this new process. The result is then returned to the calling process. Using peval, we now have a parallel quicksort:


pqsort([]) -> [];
pqsort([Pivot|Rest]) ->
    Left = [X || X <- Rest, X < Pivot],
    Right = [Y || Y <- Rest, Y >= Pivot],
    peval(fun pqsort/1, Left) ++ [Pivot] ++ peval(fun pqsort/1, Right).

Now there is a significant observation to make. The concurrent and non-concurrent versions look remarkably similar, we only had to substitute the original function evaluations with parallel versions. This is one of the reasons why I think functional languages are better suited to concurrency that OO ones.

Let's look at peval. peval creates a new process using spawn_link. spawn_link() is used instead of spawn(), so that the creator process receives exception when the new child process throws an exception.


peval(Fun, Args) ->
    Pid = spawn_link(fun() -> wait() end),
    Pid ! {self(), Fun, Args},
    receive 
        {Pid, R} -> R
    end.

wait() ->
    receive
        {From,Fun,Args} -> 
            From ! {self(), Fun(Args)}
    end.

That's it! Parallel quicksort in about 15 lines of Erlang code. If I add a "+" output to console on every entry to peval, and "-" on every exit from peval, you get the following output:


(emacs@21ccw.blogspot.com)18> L1 = lists:map(fun(X) -> trunc(random:uniform()*100) end, lists:seq(1,100)).
[57,33,2,85,59,65,61,71,83,75,94,94,19,5,79,53,31,54,39,61,
 61,79,55,6,37,35,86,31,88|...]

(emacs@21ccw.blogspot.com> pqsort:pqsort(L1).
+++-++++-+--++-+++++-+--+--+--+++-+--++-+------+++
+-++-+---+++-+--+---++-+-----+++++-+--++-+---+++++
-+--+--+++-+--+---++-+----++-++-++-+------++++-+--
++++-+--++-+---++++-+--++-+---++++-+--+++-++-+---+
++-+--+----+-----+++-+++-+--+++-++-+---+----++-+--
--
[2,4,5,6,7,9,11,13,14,17,18,19,20,21,22,24,27,31,32,33,35,
 37,38,39,40,42,44,47,48|...]

(emacs@21ccw.blogspot.com)5> pqsort:integrate_output("+++-++++-+--++-+++++-+--+--+--+++-+--++-+------+++").
0123234565654565678910910989878767898987898987654345...

You can see that for the first few pevals, by integrating over the output, that number of parallel processes goes up to around 8-10. This will depend on the distribution, and the relative times it takes to initialise processes to doing the actual computation etc.

Now that we've got pqsort going, I'm looking forward to doing the parallel graph algorithms of Jaksa's fork/join graph algorithms. There will be an obstacle to ensure that no nodes are visited more than once, but this could (possibly) be solved by partitioning the graph into subgraphs first where no children have multiple parents. We'll see...

Update (7 May 2008): This implementation is wrong! (see comments). There's a follow-up post here: Parallel Quicksort in Erlang - Part II

IE7 Page Zoom Broken

noreply@blogger.com (Benjamin Nortier) — Fri, 18 Apr 2008 17:13:00 +0000

We use XPlanner at work for some projects, and we noticed that there seems to be a problem with the hyperlinks in Internet Explorer (version 7.0.5730.11) when the pages are zoomed in (you can zoom pages in IE with Ctrl +). The horizontal areas for the links were not aligned to the actual text, which means that some links seem can't be clicked correctly, but you can click white space (look carefully at the first image, you can see the normal I-beam cursor):

I shouldn't be surprised, since it does so badly in the Acid3 test. I couldn't find any other reports of this issue, and will be happy to link to them if you post a comment.

P.S. Firefox, Opera and Safari don't have this problem.

Getting started with distributed Erlang - Mnesia table relocation

noreply@blogger.com (Benjamin Nortier) — Thu, 10 Apr 2008 13:35:00 +0000

Mnesia is a distributed database that forms part of the Erlang release. One of the features that I think is potentially powerful, is transparent table relocation across machines. With Mnesia, you can replicate tables to any nodes you wish in your network, and Mnesiatakes care of all the back end bits for you. With "transparent", I mean that you don't need to do anything in your clients to make them "aware" of the new tables. Reads that were taking place from a table on one machine, will now be distributed across multiple nodes (where the nodes reside on single or multiple machines).

I wanted to see how difficult it is to achieve this. For the setup, I installed two virtual Ubuntu 7.10 machines using VMware Player. You can get images for most Ubuntu distros at http://isv-image.ubuntu.com/vmware/. FYI, the username and password for these images is ubuntu:ubuntu. I named the two nodes

node1.21ccw.blogspot.com and
node2.21ccw.blogspot.com

You'll need to edit the network configurations with the IP addresses if you want to reproduce this experiment. If you need some help, post a question as comment :)

I now had two machines that could ping each other using the full names, and a warm and fuzzy feeling inside:

The next step was to start up an Erlang node on each machine. There's a catch here though. I got some problems using erl -sname, probably because of the way I set up the hostnames of the machines. So, I had to specify the fully qualified names manually:


ubuntu@node1:~/node1$ erl -name 'node1@node1.21ccw.blogspot.com'
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.5.5  (abort with ^G)


ubuntu@node2:~/node2$ erl -name 'node2@node2.21ccw.blogspot.com'
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.5.5  (abort with ^G)
(node1@node1.21ccw.blogspot.com)1> nodes().
[]

Notice the output of the nodes() command. This will return a list of other Erlang nodes that this node is aware of. Initially there's no awareness. To let a node know of another node, you can use net_adm:ping/1 to ping the other node. Both nodes will then become be aware of each other:


(node1@node1.21ccw.blogspot.com)4> net_adm:ping('node2@node2.21ccw.blogspot.com').
pong

(node1@node1.21ccw.blogspot.com)5> nodes().
['node1@node1.21ccw.blogspot.com']


(node2@node2.21ccw.blogspot.com)1> nodes().
['node1@node1.21ccw.blogspot.com']

Cool. Now the nodes know of each other. To get Mnesia started, you have to create a schema on each node. A schema is located on the file system, in the same location where the actual disc-copies of tables will reside. [node()|nodes()] creates a list of the current node and all the other connected nodes. ls() shows the directory that Mnesia has created for the database.


(node1@node1.21ccw.blogspot.com)5> mnesia:create_schema([node()|nodes()]).
ok

(node1@node1.21ccw.blogspot.com)6> ls().
Mnesia.node1@node1.21ccw.blogspot.com
ok


(node2@node2.21ccw.blogspot.com)2> ls().
Mnesia.node2@node2.21ccw.blogspot.com
ok

Now we have to start Mnesia on both nodes. You will notice that when we do an mnesia:info on node2 at this point, that it shows both nodes as being running database nodes.



(node1@node1.21ccw.blogspot.com)8> mnesia:start().
ok


(node2@node2.21ccw.blogspot.com)3> mnesia:start().
ok

(node2@node2.21ccw.blogspot.com)4> mnesia:info().
...
running db nodes   = ['node1@node1.21ccw.blogspot.com','node2@node2.21ccw.blogspot.com']
...

Next we'll create an actual database table, and populate it with some data. We define a record using rd(), then create a table on node1 (by default, this table will reside in RAM and have a disc copy), write a record to it and then read the record again. The primary key of the table is the first field of the record, i.e. the name.


(node1@node1.21ccw.blogspot.com)9> rd(person, {name, email_address}).
person

(node1@node1.21ccw.blogspot.com)10> mnesia:create_table(person, [{attributes, record_info(fields, person)}, {disc_copies, [node()]}]).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)11> mnesia:transaction(fun() -> mnesia:write(#person{name = "John", email_address = "john@21ccw.blogspot.com"}) end).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)14> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[#person{name = "John",email_address = "john@21ccw.blogspot.com"}]}

(node1@node1.21ccw.blogspot.com)15> mnesia:info().                                     ...
...
[{'node1@node1.21ccw.blogspot.com',disc_copies}] = [person]
...

What happens when we do the same read on node2? Remember that node has access to the person table only via the network, since it resides in RAM and on disc on node1.


node2@node2.21ccw.blogspot.com)5> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[{person,"John","john@21ccw.blogspot.com"}]}

Nice. Mnesia has transparently read the record from a table that's on another machine :)

Now we decide to copy the table to node2. This requires a single command. Mnesia does the copying of the actual data for you to the other machine, and when you look at the file system on node2, there will now be "person.DCD" file, which is the disc copy of the table.


(node1@node1.21ccw.blogspot.com)15> mnesia:add_table_copy(person, 'node2@node2.21ccw.blogspot.com', disc_copies).
{atomic,ok}


(node2@node2.21ccw.blogspot.com)9> ls("Mnesia.node2@node2.21ccw.blogspot.com").
DECISION_TAB.LOG     LATEST.LOG           person.DCD
schema.DAT
ok

At this point, when you do a query on the person table, the actual data can come from either node. I'm not sure how Mnesia decides how to distribute the data, that's something to investigate further.

Since the table is resident on both nodes, we can actually delete it from node1, and doing a read on node1 will now read the table over the network from node2:


(node1@node1.21ccw.blogspot.com)23> mnesia:del_table_copy(person, node()).
{atomic,ok}

(node1@node1.21ccw.blogspot.com)19> mnesia:info().
...
[{'node2@node2.21ccw.blogspot.com',disc_copies}] = [person]
...

(node1@node1.21ccw.blogspot.com)18> mnesia:transaction(fun() -> mnesia:read({person, "John"}) end).
{atomic,[#person{name = "John",email_address = "john@21ccw.blogspot.com"}]}

Cool.

What I've show is how to start up an Erlang/Mnesia node on two machines that are networked together, create tables on either node, and move the tables to other nodes by copying and then deleting them. Mnesia has the ability to configure tables to be RAM only, RAM and disc and disc only, which gives you lots of power for optimisation. Couple this with the fact that you can change your configuration dynamically and you have powerful, dynamically configurable distributed database!

Using Rake for Erlang Unit Testing

noreply@blogger.com (Benjamin Nortier) — Wed, 02 Apr 2008 13:37:00 +0000

My previous Erlang/Yaws project, dayfindr.com, is ticking along nicely:

$ yaws --status --id dayfindr
Uptime: 33 Days, 13 Hours, 38 Minutes

The feedback has been mostly positive, although I've had a "That's a very irritating site." comment. Well, you can't please everyone!

Following on the experience gained with dayfindr, I'm starting a new Lyme project which will be using some of the OTP design principles (Think of OTP as Erlang application design patterns). Using OTP will give me hot upgrades, which dayfindr lacks at the moment. The first step that I encountered towards this goal was getting my directory structure to conform to the OTP application structure:


/project      [The project root]
   /ebin      [The compiled .beam files]
   /src       [The source .erl files]
   /include   [Include files like record definitions]

I've been using the Makefile that's provided in the Erlang book, which is quite simple and compiles the beam files to the same directory as the source folder, which doesn't comply to the required directory structure. Now, I could just update the Makefile, but Makefiles have a cryptic syntax that I don't really want to spend time learning. Plus, they're difficult to debug in my experience. So I Googled around a bit for erlang Makefiles with little to disappointing success. Then I saw an interesting link:

Building Erlang with Rake

When they make a movie of my life and this moment, you will hear an orchestra and a choir of baritones singing "Eureka" when I click on the link.

Rake is an Ruby equivalent of Make, and more. It took some effort to get it working, since I had rake 0.7.1 on my machine, but trying to find the problem taught me a bit of Ruby in the process. Upgrading to 0.7.3 solved the problem. Sean's Rakefile compiles your src files into the ebin directory very nicely! After tinkering around with Rake, I realised that it's a really nice tool:

It has a nice mix of declarative and imperative code. You can define rules (e.g. always compile .erl to .beam), or tasks (which can be imperative, e.g. running unit tests).
You can use the full power of Ruby, and don't need to learn Make.
It has syntax that is very close to the domain (i.e. it's a good DSL)
It's easy to debug, since you can use the normal Ruby puts functions etc.
How you set up you dependencies is completely up to you, e.g. you can have different rules for files that conform to different regular expressions.

After I got this working, inspired with confidence, I decided to integrate my unit testing into the Rakefile. I think it's important to note at this point that I have less than a day's Ruby experience, and it was easy to get this working. Hacking a Makefile would probably have taken me hours and hours. The idea is that the test task is dependent on the compile task, so that if you do a "rake test", it will compile anything that's new and run the unit tests. You can just compile with "rake compile".

In order to get this going I created two Erlang files, foo.erl and bar.erl. Here's bar.erl, which contains two functions, and a test (EUnit)for each function. One of the tests will fail:



-module(bar).
-export([bar1/0, bar2/0]).

-ifdef(EUNIT).
-include_lib("eunit/include/eunit.hrl").
-endif.


bar1() ->
  ok.

bar2() ->
  okk.

-ifdef(EUNIT).
bar1_test_() ->
  [
    ?_assert(ok == bar1())
  ].
-endif.

-ifdef(EUNIT).
bar2_test_() ->
  [
    ?_assert(ok == bar2())
  ].
-endif.

Notice that the unit test code is only compiled into the beam file when the EUNIT flag is set. You can set this in the Rakefile. The unit tests are in the same source file, so we can also test non-exported functions.

Now let's look at the Rakefile:



require 'rake/clean'

INCLUDE = "include"

ERLC_FLAGS = "-I#{INCLUDE} +warn_unused_vars +warn_unused_import"

SRC = FileList['src/*.erl']
OBJ = SRC.pathmap("%{src,ebin}X.beam")

CLEAN.include("ebin/*.beam")

directory 'ebin'

rule ".beam" => ["%{ebin,src}X.erl"] do |t|
  sh "erlc -D EUNIT -pa ebin -W #{ERLC_FLAGS} -o ebin #{t.source}"
end

task :compile => ['ebin'] + OBJ

task :default => :compile

task :run_tests => [:compile] do
  puts "Modules under test:"
  OBJ.each do |obj|
    obj[%r{.*/(.*).beam}]
    mod = $1
    test_output = `erl -pa ebin -run #{mod} test -run init stop`

    if /\*failed\*/ =~ test_output
      test_output[/(Failed.*Aborted.*Skipped.*Succeeded.*$)/]
    else
      test_output[/1>\s*(.*)\n/]
    end

    puts "#{mod}: #{$1}"
  end
end

The juicy bits are the rule and the "run_tests" task. The rule states "for each file ending in .erl, compile the beam file using erlc and put the beam file in ebin". The run_tests task starts up an erlang runtime for each module, and calls test() for that module. The test output is captured and parsed using regular expressions. I know the Ruby code can be improved, so comments are most welcome.

Here's what happens when I compile and run the tests:


$ rake clean
(in /home/bjnortier/development/project1)

$ rake
(in /home/bjnortier/development/project1)
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/foo.erl
erlc -D EUNIT -pa ebin -W -Iinclude +warn_unused_vars +warn_unused_import -o ebin src/bar.erl

$rake run_tests
(in /home/bjnortier/development/project1)
Modules under test:
foo: All 2 tests successful.
bar: Failed: 1.  Aborted: 0.  Skipped: 0.  Succeeded: 1.

At this points I have cleaning, compiling and running the tests for each module. Nice. I'm quite pleased with how it works at the moment, but there's still quite a bit of work to be done:

Concatenating the running of the unit tests into one no-shell erlang execution. Actually running the tests are very fast, but the shell termination takes about a second or so. Thus, for each module, there is a approximately a second of extra time. Running all the tests in the same erlang session will require some more text parsing, but it's do-able.
Displaying the failed tests. Seeing "Failed: 1" is not very useful for determining what went wrong, so I'll update the parsing to include the failures and errors.
Probably change the "run_tests" task to just "test"
Continuous integration that hooks into version control.

If there is significant progress I'll post the results. I hope you can use some of it :)

The MacbookPro Index

noreply@blogger.com (Benjamin Nortier) — Tue, 01 Apr 2008 11:25:00 +0000

I'm looking into prices for a Macbook Pro, so that I can bolster my own prediction. I'm aware of the fact that for some reason, they are a bit more expensive in the UK than in the US (even without sales tax/VAT). I'm from South Africa, so I can organise for someone to bring me one, but I would never have guessed that it would be cheaper to buy it there than in the UK, and even cheaper than the states if you ignore sales tax/VAT.

Here's a summary of the prices for the 3 models in Us Dollars (USD), Canadian Dollars (CAD), Punds Sterling (GBP), South African Rand (ZAR) and Hong Kong Dollars (HKD). Prices were taken from the official apple websites, except in South African where the ZA Store is one of the official retailers.

	15" 2.4GHz	15"2.5 GHz	17" 2.5GHz
US	1999	2499	2799
CAD	2099	2599	2899
GBP	1299	1599	1799
ZAR	18499	22999	25999
HKD	15400	19200	21500

Using the following VAT and exchange rates (as on 31/3/2008 and 1/4/2008. A quick investigation revealed Hong Kong at 0%. I could be wrong):

Country	VAT	Exchange Rate
USD	0	1
CAD	0	1.02
GBP	18	0.50
ZAR	14	8.12
HKD	0	7.79

we get the pre-tax cost in US Dollars:

	15" 2.4GHz	15"2.5 GHz	17" 2.5GHz
US	1999	2499	2799
CAD	2049.8	2538.09	2813.05
GBP	2197.88	2705.47	3043.86
ZAR	1997.93	2483.94	2807.95
HKD	1977.64	2465.63	2760.99

At a glance, the UK price is consistently higher than any others, and remember this is without VAT. UK VAT is at 17.5%, the highest of the 5, which will make the post-tax prices even higher.

And if you prefer a pretty picture (using Google Charts API):

And now onto the MacbookPro index. If you haven't heard about the Big Mac index, it is an informal way of measuring purchasing power parity between countries. It can be used to estimate if a currency is over or undervalued, since a Big Mac is pretty much the same in every country. It is calculated by dividing the price of a Big Mac in one country, by that it another country, and comparing this value to the actual exchange rate.

For the MacbookPro index, I've used an average price (non-weighted for simplicity), and here are the results:

Currency	Under valuation [%]
CAD	0.02
GBP	8.91
ZAR	-0.1
HKD	-1.27

and the graph:

You could conclude that the pound is undervalued by about 9%. I wouldn't. I would conclude that Macs are overpriced in the UK. I think I'll get mine elsewhere...

The Mac is Back*

noreply@blogger.com (Benjamin Nortier) — Wed, 19 Mar 2008 14:35:00 +0000

* No, not John McCain, I'm talking about the Apple Mac.

I'm going to make a wild prediction:

The Apple Mac will increase it's market share by more than 50% over the next two years from 4.8% to at least 7.5 %.

I'm basing my speculation on some figures:

Mac market share went up 16% year-on-year (Feb 2008. From 4.4 to 4.8% )
Macs represented 14 percent of sales last month (Feb 2008)
Solid iPhone market share figures in the US (28% in the US for Q4 2007)

And some opinion:

First, the Halo effect. Apple now has a solid (well integrated) product offering with the Mac line, iPods, iPhones and iTunes (music and movies). Consumers who are exposed to any of these products have (mostly) good experiences, which means they get warm and fuzzy inside and buy more Apple products. (but NOT shares. Hmmmm). Why are their product doing so well despite their higher prices? My first axiom:

People are always willing to pay more for a better quality product

Secondly, some perception. I work in technology (creating software), and I would hope that we have a better idea of where the market is heading that the average consumer. I would also like to think that we have some influence over how people will be using their computers in the future, especially though the products and services that we create. I'm seeing more and more Macs being used by the creators of software, especially by those who are more discerning, and my perception is that there is an increasingly positive view of the Mac and Mac OS X in the amongst software creators. Couple that with the buzz that has been created by the iPhone and the iPhone SDK (which delivers a platform for writing software for the iPhone by 3rd party developers), and the future looks rosy.

Lastly, Windows Vista. No matter what Microsoft says, it hasn't been a big success. All the problems associated with it's (much delayed) launch, the quality of the product, the whole "Vista Capable" fiasco etc. has created a unique opportunity for Mac OS X.

Let's wait and see...

Acid3 Test released

noreply@blogger.com (Benjamin Nortier) — Thu, 06 Mar 2008 14:44:00 +0000

What is Acid3? From Wikipedia: "Acid3 is a test suite that checks how well a web browser follows certain web standards, especially relating to the DOM and JavaScript."

There's some synopsis here and on the wikipedia page

Here are the results of the browsers that I have on my work machine (Windows XP). Note that this is a score out of 100:

No surprise that IE, the bastion of web standards, fails so miserably. I'm actually surprised that Firefox is doing better than Opera, since my subjective opinion at the moment is that Opera renders most pages better than Firefox (except the Google ones, which is not much of a surprise since Google funds a large chunk of Firefox development). Also, the first time I tried the test in Opera 9.26, it crashed the browser. And it just crashed again now with Acid3 open.

All these 4 browsers have big releases coming up, so it remains to be seen who become the new king of the Acid hill. Webkit based browsers like Safari seems to be in the early lead with scores of 90/100 floating around...

Apache + Erlang for dynamic web content

noreply@blogger.com (Benjamin Nortier) — Wed, 05 Mar 2008 09:58:00 +0000

It dawned on me this morning that there is a way of using Apache (or any other web server for that matter) with Erlang to create dynamic web content...

How? Use FUSE. There is an implementation of FUSE for Erlang by the Dukes of Erl, called fuserl. If you haven't heard of FUSE, it's a way to make anything you want look like a file system. By using fuserl, you can make an Mnesia database (just one example) look like a file system. You can map queries to directories, and the files that are "listed" can contain the rows of your tables. But you can structure your filesystem in any way you want!

Take for example YoutubeFS: "YoutubeFS enables you to browse your favorite Youtube videos locally on your desktop without going to the youtube website. Just create a youtube account and add videos to your playlists, favorites list or subscribe to different channels. YoutubeFS then enables you to automatically load these videos to a local folder on your desktop. You can then view these videos (using a browser) as if they are local files."

If you extend this idea, you can imagine that you can point your Apache server to a FUSE filesystem location which generates the "contents" of the file system dynamically. To Apache it looks like it's serving normal files, but behind the scenes you can have a distributed Erlang system generating content for you...

This wouldn't work so well if you actually want to use parameters in your http queries, but then I would use Yaws with an appmod if that's what you need.

P.S. You don't have to use Erlang either, you can write FUSE file systems in many other languages...

The time has come

noreply@blogger.com (Benjamin Nortier) — Mon, 25 Feb 2008 18:25:00 +0000

I've been working on an Erlang project at home for a while, and it has reached the first public version! Here it is http://www.dayfindr.com. It's a simple utility that can make your life easier.

It ticks some of the boxes that Paul Graham related recently in Six Principles for Making things:

(a) simple solutions I hope so. This was one of the big design principles - to make it as simple as possible. That's why, for example, it leverages email instead of using some fancy social networking/address book features. There's also no login, since your email address already identifies you. TripIt uses the same idea
(b) to overlooked problems nothing that I know of addresses it
(c) that actually need to be solved, and This tries to help with a real problem that I've encountered myself. Feedback on the idea has been quite positive, so I know it's something that people will find useful!
(d) deliver them as informally as possible Hope so
(e) starting with a very crude version 1, then Relatively crude
(f) iterating rapidly. We'll see!

It's implemented on a Lyme stack (Linux, Yaws, Mnesia and Erlang). This project has generated some interesting questions, that I will relate in a series of posts. For example:

Why Erlang?
Why not ErlyWeb?
Why not Rails?
Why did I abandon the fully functional Lisp version?

So, use it, it's free. If you like it (or more importantly if you don't), lemme know at feedback@dayfindr.com

P.S. To the Internet Explorer users: You will notice that there's a javascript error on the page, because some function is "not implemented", which means you'll see check boxes and not coloured boxes in the calendar. The current versions of Opera, Firefox and Safari have no problem with it. Maybe you should consider an upgrade...