Sunday, 23 August 2009

My Google Wave Client

Google released a reference implementation of a Wave client and server in July, written in Java, that you can download and experiment with. When Mickaël Rémond wrote about using the reference implementation with ejabberd, I decided that it was time to get my hands dirty.

I got the reference implementation working, and I could chat with multiple participants in the console clients. However, I wanted to know more about what was happening behind the scenes in the client, and more importantly, the server (and how operation tranforms work).

So I decided to write my own client in Erlang that communicates with the reference server. This way I could investigate what was happening in the client and the server, and gain more knowledge about Wave. It required a bit of reverse engineering, and messing around with protocol buffers (the client communicates with the server using these), but it works nicely.

I've created a video to demonstrate it. Happy watching!

Vimeo link

P.S. I used ngerakines's implementation of Erlang prototocol buffers, and the Nitrogen Web Framework.

P.P.S. I'm going on holiday for 2 weeks, so if I don't reply to comments, please be patient. I will do so when I get back...


Wednesday, 8 July 2009

Google OS - I believe I was right

disclosure: I own Google shares

I'm not one for blowing my own trumpet, but sometimes it feels good to be right. A month ago I blogged about HTML5, and I said the following:

"I believe it is part of Google's long term strategy to make the browser the de-facto application platform. To displace Windows as the dominant aplication platform. You can see it already with Chrome, how the tabs on the browser have become the new taskbar, and the integration of browser tabs as "applications" into the host OS. For example, features like local file storage and the canvas provide a path for Google Docs to become more like MS Office. You can be sure that some of the drive behind the HTML5 standard is coming from the Google docs team."

Yesterday Google announced Google OS: http://googleblog.blogspot.com/2009/07/introducing-google-chrome-os.html. Google OS will be based on Chrome and a Linux kernel.

From the announcement:

"For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. And of course, these apps will run not only on Google Chrome OS, but on any standards-based browser on Windows, Mac and Linux thereby giving developers the largest user base of any platform."

I will admit that I didn't think they would actually write an OS to support this strategy, but the strategy is clear. Unfortunately we'll have to wait at least a year to see it in action...




Thursday, 2 July 2009

Erlang Factory 2009 - New Kids on the Erlang Block

I went to Erlang Factory again this year thanks to my employer, and it was even more exciting than last year's event. There were some great speakers (Joe Armstrong - entertaining as ever, Simon Peyton Jones), I got to meet some interesting people (e.g. Rusty Klophaus, creator of the Nitrogen web framework) and I got to see some awesome projects. Although Erlang is experiencing a surge of interest, it is by no means "new". it has been used for years in robust and highly scalable applications. That is why I refer to the "new" - as opposed to the solid Erlang VM, tools, libraries and frameworks that have been around for at least a decade.

So, wo are these the New Kids on the Block? These guys?


Nice hair! No, not them. Here are some of the New Kids on the Erlang Block:
Heroku is a platform for deploying Ruby or Rails web apps. They've built their routing mesh in Erlang. This routing mesh is used to load-balance and route requests to their application work units (dynos). This is a really cool piece of cloud infrastructure.

HyperNumbers is a startup that is "doing for numbers what hypertext did for text". Think GoogleDocs, but each cell can be a resource that's located on another spreadsheet. Coming soon.

CouchDB is a non-relational document database written in Erlang, and it is going from strength to strength. Look at this graph from Google Trends:


If you don't know what CouchDB is, you should definitely have a look it. Even is it's just to make you challenge you preconceptions around databases, and relational ones in particular. There is also a book coming out soon. It's open source.

RabbitMQ is an open source, high-performance enterprise messaging system, and an implementation of the messaging standard AQMP. It is written in Erlang. I've heard rumors that ActiveMQ (Java) has 20 times more lines of code. (NB: If anyone can give me a reference or link to proof I would be grateful). Update: See comments for line counts.

This is my favourite web framework. It's a real Web 2.0 framework, and it's highly productive. Some of the main features are:
  • Event-Driven Development
  • Brainlessly Easy Ajax
  • Ridiculously Simple Comet
  • Complex Interfaces: Dragging, Dropping, and Sorting
  • Flexible Templating
  • Data Binding
  • Erlang Power
I've used this to create my own pet Erlang project, opinion8r.com. Check it out. It's also open source.

I didn't go to this talk, but I've been to one given last year at the Erlang London User group. Smarkets is an online betting exchange which allows users to bet on anything they wish. It is implemented in Erlang and close to launch.

Campfire is one of the applications from the guys at 37Signals. If you don't know who 37 Signals are, they created Ruby on Rails. The functional C polling for Campfire service that was being used was replaced by an Erlang implementation:


The Erlang implementation is as fast as the C implementation, modular and extensible, and much easier to administer with 1 OS process instead of 80.

ProcessOne - OneTeam Media Server
"OneTeam Media Server is a new project launched by ProcessOne which is a Media server for Flash clients implementing the FLV and RTMP protocol. The project is designed so that it can work hand in hand with ejabberd, ProcessOne Instant Messaging platform."

This one is very exciting. The media server has Flash in the front end, and Erlang in the back. It makes thing like writing video chat in the browser ridiculously easy. And it's open source! Look out for it in July.

Webmachine is the best way to implement a REST interface to your Erlang application that I've seen to date. I've blogged about REST and Erlang in the past, but instead of rolling your own, you should look at this. Especially considering how it handles all the response codes correctly, does all the correct caching responses etc. Another open source project.

And the point is...?

I think it's a very positive indicator for the future of Erlang that it is being used in these new and exciting areas and companies. It is performing very well in the field of high-scalability infrastructure. It is well suited to Cloud computing. But most importantly, very clever and innovative people are using it and loving it. That, above all else, means the future looks good for Erlang.

P.S. I'm sure I missed some great talks again this year, but lucky for me and you, you can find the talks and videos here.

Friday, 5 June 2009

Look out RIAs, HTML5 is coming

Google showing off Wave, has [re]ignited some discussion around HTML5. I read an article on InfoQ, Is Google Wave Going to Have an Impact on RIA/Silverlight? Which struck me as a bit of an odd question. Tim Heuer (in the Silverlight camp) is correct in asking "is HTML5 really what people are talking about here?" and towards the end of the article, the correct question is polled:

"What negative impact will HTML 5/Wave going to have on RIA/Silverlight?" [sic]

At the time the results of the poll were:

Major: 66
Significant: 115
Small: 112
None: 43
Total votes: 336

I was a bit surprised by the poll results. I voted "Major", and I find it hard to understand how someone could vote "None". None?! Really? I do realise that this is partly due to the fact that most people, including me, think they are above average (I just discovered that it is also called the Lake Wobegon effect. Cute). There is also some confusion about including Wave in the question, which has nothing to do with the issue. I'll try and convince you of the potential impact...

Why RIAs exist


The existence of Rich Internet Applications can be explained by the word "Rich" in the name. At some point in the past few years, people started to realise that HTML didn't keep up with the explosion in bandwidth and processing power available to users, and that the web experience could be "richer". This was especially obvious in areas such as not having the ability to to arbitrary on-screen animantions and rendering on a "canvas", the ability to store data locally on user's hard drives and drag-and-drop, amongst others. The lack of these features in HTML created a gap in the market, but only whilst browser technology continues to lag. This gap has been exploited by Flex, Silverlight and JavaFX.

This situation is changing.

Enter HTML5

Some of the features in HTML5 will add some really exciting possibilities to applications running on the browser platform:

Video


Although I wouldn't categorise video as a major driver for RIA technologies, it is important to mention in relation to Adobe flash, which is the dominant technology for delivering internet video. YouTube is already delivering non-flash internet video in mobile devices. The introduction of the tag will reduce the pressure of deploying flash on every platform, which could have an effect on the usage of Flex. Consider that Flash is not available on the iPhone: if you have a choice of developing your application on just one technology (HTML5) for both desktop and mobile, or choosing 2, which would cost less?

Canvas drawing (immediate mode drawing)


Canvas drawing will enable application developers to fill a gaping hole in native browser capabilities - gaming. The gaming industry is bigger than the movie industry, and the availability of a virtually universally available standard platform will not got unnoticed by game developers. Of course this capability doesn't only apply to gaming.

Local storage

As long as there's no capability for storing data offline, web applications continue to be considered the poor cousin of desktop applications. Google documents are not availble offline without Google Gears. Having local storage available in a standardised way is a big attraction for application developers. Gears might work for Gooogle, but not everyone is so keen on having a dependency on a non-standard technology.

Some other disavantages to RIAs which could favour HTML5 applications

- RIAs are usually everything or nothing. They don't play well with embedding RIA functionality into existing web pages. HTML5 allows you to grow your application, creating a richer and richer experience as you go along.
- Some RIAs have proprietary technologies.
- RIAs require an extra deployment step on top of deploying the browser. Browsers implementing HTML5 will provide a more standardised way of deploying applications to users, also in the corporate environment where locked-down systems are common. Choosing a technology that doesn't require another installation step is a big attraction for corporate IT departments.

In the long term

There another significant point to make.

I believe it is part of Google's long term strategy to make the browser the de-facto application platform. To displace Windows as the dominant aplication platform. You can see it already with Chrome, how the tabs on the browser have become the new taskbar, and the integration of browser tabs as "applications" into the host OS. For example, features like local file storage and the canvas provide a path for Google Docs to become more like MS Office. You can be sure that some of the drive behind the HTML5 standard is coming from the Google docs team.


In closing

I'm not trying to say that HTML5 will eliminate RIA technologies, but I do believe that thinking that its impact on RIA will be "small" or "none" is absolutely ridiculous.

There are some big players pushing the browser to become the dominant application platform. HTML5 is a big step in that direction.

If you disagree, I look forward to your comments!

Wednesday, 20 May 2009

The Law of Conservation of Misery

At university I had a professor who would jokingly refer to his Law of Conservation of Misery:

The total amount of misery in a system is constant

which implies that when you try and decrease the misery in one aspect of a system, you will increase the misery in some of the other aspects.

There are many types of misery. In software engineering projects these could include non-maintainability, non-performance and budget constraints.

Personally, I think people are too prone to think that there are zero-sum monsters lurking under each bed, so my own take on the "law" would be:

The total amount of misery in a system is > 0

Which implies that you can make tradeoffs that decrease the total amount of misery, but you can never eliminate it completely. A bad implementation of a system will have a higher total amount of misery, and a good one will have less, but you will always have at least a little bit!

Tuesday, 19 May 2009

How to send email via Gmail using Erlang

One of my pet projects, www.dayfindr.com, integrates with email to send notifications to users.

I use Google Apps for email infrastructure, so you need an SMTP client that supports TLS. At the time, I couldn't find a simple Erlang SMTP client that could handle TLS, so I used a command-line SMTP client.

For my new pet project, for want of a better name temporarily called The Isabelle Project, I need to add some email functionality. This time I would prefer to use an Erlang solution with proper error handling and logging.

I looked at the SMTP protocol on Wikipedia, and it didn't seem to difficult. Erlang's built-in ssl module also seemed to support TLS. So, with a bit of trial and error, here's the result:


-module(smtp).
-export([connect/0]).

connect() ->
{ok, Socket} = ssl:connect("smtp.gmail.com", 465, [{active, false}], 1000),
recv(Socket),
send(Socket, "HELO localhost"),
send(Socket, "AUTH LOGIN"),
send(Socket, binary_to_list(base64:encode("___@gmail.com"))),
send(Socket, binary_to_list(base64:encode("johngalt"))),
send(Socket, "MAIL FROM: <___@gmail.com>"),
send(Socket, "RCPT TO:<___@gmail.com>"),
send(Socket, "DATA"),
send_no_receive(Socket, "From: <___@gmail.com>"),
send_no_receive(Socket, "To: <___@gmail.com>"),
send_no_receive(Socket, "Date: Tue, 15 Jan 2008 16:02:43 +0000"),
send_no_receive(Socket, "Subject: Test message"),
send_no_receive(Socket, ""),
send_no_receive(Socket, "This is a test"),
send_no_receive(Socket, ""),
send(Socket, "."),
send(Socket, "QUIT"),
ssl:close(Socket).

send_no_receive(Socket, Data) ->
ssl:send(Socket, Data ++ "\r\n").


send(Socket, Data) ->
ssl:send(Socket, Data ++ "\r\n"),
recv(Socket).

recv(Socket) ->
case ssl:recv(Socket, 0, 1000) of
{ok, Return} -> io:format("~p~n", [Return]);
{error, Reason} -> io:format("ERROR: ~p~n", [Reason])
end.


And the output from the Erlang shell:


3> application:start(ssl).
ok
4> smtp:connect().
"220 mx.google.com ESMTP y37sm613282mug.19\r\n"
"250 mx.google.com at your service\r\n"
"334 VXNlcm5hbWU6\r\n"
"334 UGFzc3dvcmQ6\r\n"
"235 2.7.0 Accepted\r\n"
"250 2.1.0 OK y37sm613282mug.19\r\n"
"250 2.1.5 OK y37sm613282mug.19\r\n"
"354 Go ahead y37sm613282mug.19\r\n"
"250 2.0.0 OK 1242683885 y37sm613282mug.19\r\n"
"221 2.0.0 closing connection y37sm613282mug.19\r\n"
ok


The only tricky bit is that for the AUTO LOGIN, the received text and the username and password you send is base-64 encoded. By default the connect is active=false, which means the responses are send to the creating process directly. Using passive mode requires explicit receiving of the response using ssl:recv/2

You'll have to handle errors better if you use this in production, but the basic protocol is pretty straightforward...

Friday, 2 January 2009

Is a Square a Rectangle?

I pre-ordered the Clojure book today, so I downloaded Clojure and had a peek at something that I know is interesting - "multimethods". I came across the following example statement:

(derive ::square ::rect)

i.e. a Square IS-A Rectangle in the Object-Oriented sense.

This rang a bell for me. Not a nice melodious one, but one that's a little off-key. A good teacher asked some colleagues and me the question "Is a Square a Rectangle?" a few years ago, and the answer wasn't all that clear ti us. But he showed us a realitively easy way to determine to some extent the validity of any IS-A relationship.

I will try and show you this principle by way of actually designing this hierarchy. First we design a simple Rectangle in Ruby:


class RectangleTest < Test::Unit::TestCase
def test_width_height

# Given
r = Rectangle.new(1,1)

# Then
assert_equal 1, r.width
assert_equal 1, r.height

# When
r.width = 5

# Then
assert_equal 5, r.width
assert_equal 1, r.height

# When
r.height = 2

# Then
assert_equal 5, r.width
assert_equal 2, r.height

end
end


If you think it's strange that I use a test as a design, I suggest you read this. I'm not even going to show the implementation since it's so simple. Next we will design our Square class:


class SquareTest < Test::Unit::TestCase
def test_width_height

# Given
s = Square.new(3)

# Then
assert_equal 3, s.width
assert_equal 3, s.height

# When
s.width = 5

# Then
assert_equal 5, s.width
assert_equal 5, s.height

# When
s.height = 2

# Then
assert_equal 2, s.width
assert_equal 2, s.height

end
end


And you can see that we make sure that when either the width or height has been set, we get the same value for both the width and height respectively. The same goes for the single-parameter constructor. Again, the implementation is trivial. 

Now, you might think that this design is OK, but it's not. We have to ensure that our inheritance doesn't break the Liskov Substritution Principle.

From Wikipedia:
"Liskov's notion of 'subtype' is based on the notion of substitutability; that is, if S is a subtype of T, then objects of type T in a program may be replaced with objects of type S without altering any of the desirable properties of that program (e.g., correctness)."

This means that for our designs, the unit test for the Rectangle should pass when I substitute the Rectangle instance with a Square instance. Will it? No it won't. Because in the rectangle test, we ensure that when the width is set, the height is not affected, and vice versa. Using a Square instance here will break the test.

Another way of looking at it is to think of the contract of the setters, and the pre and post conditions of those contracts. According to the Liskov princple, (again, from Wikipedia):
  • Preconditions cannot be strengthened in a subclass.
  • Postconditions cannot be weakened in a subclass.
If we design the Square to force the width to be set when setting the height (and vice versa), we are weakening the postconditions of the Rectangle methods. I.e. the post condition that states that either the width or height should not be altered when setting the height or width respectively has been weakened.

In conclusion, here we have a Square that is NOT a Rectangle. The lesson to be learnt is that (unfortunately for us), IS-A in the real world or other domains does not necessarily imply IS-A in the OO world! 

P.S. If you remove the setters from the Rectangle, and only allow the fields to be set during construction, there is no violation of the Liskov Substitution Principle :)