I always found Haskell quite nice, but never tried developing web apps with it. After using Yesod, I’m slowly starting to think that Haskell might be much better for it than many of the commonly used languages like PHP, Python, Ruby or JavaScript.

Here’s why:

Type-safety

As you may know, Haskell is statically typed. This may sound like an inconvenience, but Haskell’s type system is very flexible and it has very powerful type-inference – meaning you don’t need to specify types manually most of the time – so in actual usage, it’s not bad at all.

Yesod expands type-safety to various bits of web programming:

• Type-safe URLs
• Type-safe DB queries

In short, these prevent a ton of errors.

Type-safe URLs

Yesod has a routing mechanism similar to most web frameworks these days.

You define routes using a simple DSL, for example /users/#UserId UserR GET POST. This defines a route which works with GET and POST methods.

Notice the #UserId part. This is a parameter. However, it’s not just a parameter definition – #UserId is an actual type.

In your handler function, the parameter will always be a UserId. You could also use #Text, #Int or whatever. The value you get will be cast into the type you want, and if not possible, an error will be presented instead.

Type-safety in URLs also applies directly to generating URLs in code. If you are generating a URL for a page which requires a UserId, it’s again checked that you have the correct type of parameter for it. Same goes if you change your routes: a very typical scenario is adding parameters to some route, and then having to find all places where it’s used. Because URL generation uses types in Yesod, you will automatically get errors from the compiler from every place where your URL generation code is incorrect.

Type-safe DB queries

Wouldn’t it be nice if you would get an error if you mistyped something in an SQL query? Or a MongoDB query?

With Yesod, you do. Your application will not compile if your DB queries are incorrect.

In addition to this, Yesod has a persistence layer which is sort of like a light weight data mapper.

The mapper does what you’d expect: Performing queries returns you nice “objects” of data. (You actually define a data type for it, but it’s kind of like a simple object if you’re not familiar with Haskell)

Let’s say that you have a User model and need to query for all users created before a certain date. The type-safety extends to this as well: Assuming you’ve defined your user’s create time field as a UTCTime, you can only query this field with variables of the type UTCTime, again saving you from possible errors.

With model IDs it works even better: For example with the UserId mentioned earlier, you can simply pass it to the Yesod persistence layer, and it will always get you a User. You can’t query for something that isn’t a User with a UserId.

Compile time errors

As Haskell is not a dynamic language, you must compile the application in order to run it. Putting this together with Haskell’s type system again eliminates a lot of bugs, as many errors you could accidentally make will prevent the code from compiling at all.

This all may sound like a hassle when you can simply reload or restart with dynamic languages, but with Yesod it’s just a single command to compile, or if you’re using Yesod’s development server, it will compile it automatically whenever your files change.

Compile-time checked configuration and templating

Some common parts in Yesod use specialized DSLs. These are done using Template Haskell, Haskell’s metaprogramming facility. Essentially what this allows, is compile-time checking for correctness.

As mentioned, Yesod’s routing configuration uses a DSL. Yesod also has DSLs for models, HTML, JavaScript and CSS.

Yesod’s JavaScript and CSS parsers are pretty straightforward: You just get variable interpolation and some other such niceties. However, the HTML DSL, called Hamlet, is a full language of its own. It’s not wildly different from your standard HTML, so it’s easy to learn, but it’s also checked by the compiler for syntax errors and other mistakes – just as all other Haskell code – which is great for preventing errors again.

It’s Haskell

Have I mentioned that Haskell is great and fun?

Conclusion

I think Yesod and Haskell brings many great ideas to the table for web development. These are just the few things I’ve found while working with it, and there’s still much more to it.

Just type-safety in URLs and compile-time checking for all this is a massive time saver when you have a mistake somewhere. Especially with URL generation, I have forgotten to change some code somewhere countless times and then only later stumbled upon it. With Yesod, I never would’ve had the problem in the first place.

Learning Haskell may be a large effort if you haven’t used functional languages, but this is just another example showing how much it’s worth.

Except there are various things where PHP is better than Node…

Be sure to read the comments too for some discussion

Easier to find hosting

Majority of the web’s hosting providers are able to provide PHP hosting.

For Node, you need a much more specialized hosting provider. Typically you need shell access to set up your application, and most web hosting services do not include anything like this, or if they do, the other packages are much cheaper.

It’s easier to get started with PHP

With PHP, you can simply install WAMP, LAMP or MAMP and off you go. For deploying code into a web host, you just drop your files there and you’re done.

Although Node itself is not hard to install, it does require more specialized knowledge to set it up nicely. In order to install it on a server, you generally need some level of Linux sysadmin knowledge in order to set up the necessary packages, and tools such as process monitoring to keep your node instance running in case of a crash.

If your PHP code breaks, it doesn’t bring your whole server down

As PHP code runs each in its own process and the web server sits in front and directs this, if one of the requests causes an error it will only affect that specific request.

In a Node environment, you typically have a single process serve all the requests. If one request causes an uncaught error, the entire server comes down.

PHP processes are short lived

In PHP each process is only alive for the duration of the request. This means you don’t need to worry as much about resource allocation and memory.

In Node as the process is running for a long time, you need to be careful to manage memory properly. For example, it’s easy to accidentally take tons of memory if you forget to remove items from a global array. It’s also easy to accidentally core which leaks memory. (which in turn can bring the whole process down again)

Bigger standard library

PHP’s standard library is much bigger than Node’s.

In closing

This is not to say that PHP is better in every way than Node. It’s simply to point out that while Node is very good at certain things (real time processing of messages for example), it is not a silver bullet for all problems.

Feel free to leave your thoughts in the comments, or if you can think of more ways where PHP is better than node.

Of course I had the option of putting it in a non-standard port like 8080, but it wouldn’t be nice if the URL would be somedomain.com:8080. The second option was ordering an extra IP address from Linode. That would’ve cost me a bit each month, so I didn’t really want to do that either.

The third option turned out to be pretty easy: Use Varnish.

Here’s how I did it…

Step 1: Put your servers on non-80 ports

First, we need to put Apache on some other port than 80. Simply change the Listen option in Apache’s config.

Let’s say Apache goes to port 6001 for this example.

The node application will also need to run on some other port than 80. Let’s put it to port 6002.

Step 2: Configure Varnish on port 80

First, find your Varnish config file. On Ubuntu, you can probably find it at /etc/default/varnish

Find the DAEMON_OPTS

Make sure the DAEMON_OPTS uses port 80 in the -a flag as follows:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

Or, if you use the advanced configuration, set VARNISH_LISTEN_PORT to 80.

Step 3: Set up Varnish proxying to Apache and Node

Open your varnish VCL file, most likely it is /etc/varnish/default.vcl

You will need to set up backends for all your servers here. In my case, I will set up backends “apache” and “node”, as follows:

backend apache {
    .host = "127.0.0.1";
    .port = "6001";
}

backend node {
    .host = "127.0.0.1";
    .port = "6002";
}

Next, we need to define the vcl_recv sub to route between these backends. This goes after the backend definitions in the VCL file:

sub vcl_recv {
    if(req.http.host == "apachesite.com") {
        set req.backend = apache;
    }

    if(req.http.host == "nodesite.com") {
        set req.backend = node;
    }
}

All we need to do is check what the host in the request is: If the domain used is apachesite.com, the request will be proxied to Apache, and for nodesite.com, to Node.

If you want to use a single domain, you could use subdomains to check which backend to use as well. In my case, I have an additional subdomain which I use to access my development node instance.

Step 4

There is no Step 4.

Conclusion

By using Varnish on port 80, we can pretty easily have any number of other servers serving content so that it seems that all of it is on port 80 when it really isn’t.

This of course is not the only thing Varnish can do. You can set up caching and all sorts of stuff to it, but for that you probably should look at the Varnish manual

In addition to your usual web app stuff, Web Commander talks to Battlefield 3 game servers in real time and has many features related to that. It also displays data from the servers in real time. Because of these, there is a fair bit of complexity to it.

Main components used

As said, the application itself runs on nodejs. The data is stored in MongoDB.

In case of the node process crashing, forever is used to automatically restart it.

In front of the nodejs server is a Varnish HTTP accelerator. This was mainly used due to the fact I wanted to run the node app on port 80, where I was also running Apache.

The setup with Varnish is pretty simple: Varnish runs on port 80, and looks at the Host header of each HTTP request. It then routes to Apache, the node production instance or the node testing instance depending on the domain name used. The actual servers sit on other ports, but the user can’t tell the difference.

Overview of the application

The application is structured between the web interface and the part which talks to the Battlefield 3 servers. Database access is relatively limited: Only the web interface and the StatService (more on that soon) talk to it.

Let’s talk about the more interesting part first.

Talking to the Battlefield 3 server

The objects which handle actions with the gameserver are the Connection and Server objects. Connection implements the low-level protocol used to talk to the Battlefield 3 server. It’s a binary-protocol, so it’s implemented using the binary writing/reading methods in Buffer. It implements methods for sending commands to the server and listening to the low-level events from the server.

The Server object on the other hand implements a higher level interface on top of the Connection – methods such as kicking and banning players or editing the maplist are callable via a Server instance. In addition, it also handles all the other server-related stuff involved: It keeps track of the maplist, current map, the list of players on the server and other things – essentially it represents the state of the gameserver in the code. Whenever the state changes in a meaningful way (for example if a player leaves or such), it emits events about it.

Both the Connection and Server are primarily custom code written on top of node builtins, mainly utilizing Buffer and EventEmitter.

To keep track of the state, the Server keeps several lists it updates whenever things happen on the server. In some cases, it also has to query the gameserver from time to time (using setInterval) in order to check if the data has changed, as certain things will not emit any events (such as changing server settings like friendly fire). Other things, like the player list, use both a periodic check and events to stay synced: Whenever players join or leave, the list must be updated, but many player events don’t include data like the player’s total score in them, which in turn needs to be polled from the gameserver.

The job of the ServerManager is to keep track of the servers. It does tasks like connecting and disconnecting servers, testing connections to newly added servers, and it also tries to automatically reconnect any servers which get their connection to the gameserver dropped which happens pretty often and would otherwise require the user to manually reconnect.

For reconnecting, it uses a pretty simple approach of “Wait X seconds and try again.”. If it can’t get reconnected, it waits some more and tries again until it works (or reaches the current limit of 100 retries – The servers are sometimes pretty stubborn).

Plugin architecture

To make it easy to provide further features outside the basics of what is implemented by the game protocol itself, the Server object supports plugins. These are basically features the users can enable or disable on a per-server basis, such as automatic team balancing or map voting. Plugins themselves work simply by registering with the server, and listening events from them and sending commands back using the interface provided by the Server object.

Plugins are always server specific. A new instance of a plugin is always created for each server it’s running on. Sometimes plugins need shared functionality, so for that there are the two services shown in the diagram. These are exposed to the plugins via the constructor and the plugin can grab any service instances it needs from there.

In theory, 3rd parties could create new plugins which can then be used, but as the code is not open-source there is currently no way for anyone else besides myself to create plugins.

Plugins only utilize the interfaces provided by services and server.

Services

As one might guess, the purpose of the StatService is to persist server statistics to the database. There is a plugin which when installed on a server, calls methods on the StatService to record all kinds of statistics, such as kills, deaths, score, etc. for all players. The StatService is currently the only component in addition to the web interface which talks to MongoDB.

The IrcService is the more interesting of the current two services. There’s a plugin available which can send reports from your server to an IRC channel of your choice. Now, we could create a new IRC bot for each server, but IRC servers limit the amount of connections you can do from a single IP address. Thus, we need to share a single IRC bot for this.

The IRC service keeps track of the IRC bot. If a plugin requests a bot on a channel, the service will check if the bot is on the channel, and if necessary, it will connect to the IRC server and join the channel. It also keeps track of the bot so, that if multiple servers require the bot on the same channel, the bot will only leave the channel after it has been disabled for all the servers which requested it.

The IRC service uses node-irc to talk to the IRC server, but I will probably have to replace it with something else since it sometimes makes the entire application unstable if there’s a connection error. The StatService uses Mongoose for the MongoDB parts.

The web frontend

The web parts of Battlefield 3 Web Commander are built on top of Express and use ejs for templating.

It’s split into three main modules: The server, admin and stats modules.

You can probably guess the primary purposes of each module based on their names.

Server module

The server module contains the logic involved in displaying the server list and all server-specific pages – overview, scoreboard, bans, maps and settings.

Many of the server pages update in real time. Socket.IO made a perfect tool for this. Essentially, for each server, there’s a bunch of event listeners in the server module which listen to some events from the Server object, and then emit some more events using Socket.IO. Users who are connected will then receive those events via Socket.IO, which in turn updates the KnockoutJS powered View Models on the client-side.

As mentioned, KnockoutJS is also used on many of the server module’s pages. I found it perfect for easily creating templates on the client which could be rendered based on data, without having to mess around with adding or removing elements in the DOM. Simply update the Knockout view model and the library handles keeping the DOM in sync.

The server related pages mainly just grab data from the Server instance and displays it to the user. Hidden to most people visiting are the admin tools present on the pages: On all server pages, an admin can perform actions on the things displayed. Admins can kick, ban, kill and chat from the Overview, kick/ban/kill from Scoreboard, add/remove bans from Bans, edit the server list using a snappy drag/drop based interface from the Maps page and change server settings or use presets such as Hardcore or Infantry Only on their server from the Settings page.

Some of the admin functionality use normal forms – for example, adding a new map simply submits a form – but others use Socket.IO to send commands to make the interface nicer to use. Reordering maps or removing maps is a good example of this. Since you may often move maps around or delete multiple maps, the tools for that use Socket.IO to make it faster to work with.

The admin tools in the server module only perform actions which directly affect a specific server instance.

Stats module

The stats module is perhaps the simplest of all of the web modules: It simply queries the database and displays what was queried. There is no real-time updates on any of the pages (except if you hit reload), and there are no admin tools or such.

Admin module

The admin module contains the rest of the admin functionality available: It allows you to add and edit servers and users, and to view entries in the admin audit log and any reports sent by players.

The audit log is simply a list of actions taken by admins. Whenever an admin does something, their action is logged along with any relevant details. This makes it easy to see who did and what later if necessary.

The reporting functionality is provided as an option for each server. A player can send a report by using a command in chat, and Web Commander will save the current state of the server along with the most recent lines of chat into the DB so that admins can take a look at it later. This is so that if there’s a problem on the server and no admins, it can be later solved based on the report. The IRC bot plugin can also send messages to IRC whenever there’s a new report.

The server editing in the admin module is the only part of it which uses Socket.IO. When creating a new server, it was easier to display the progress of it in a meaningful and easy to understand way if the page wasn’t fully reloaded. The connection can fail in a variety of ways and it can take several seconds for it to be established (or fail), so it makes sense to display a dynamic progress indication instead of a full page load.

Another part of the server editing which uses Socket.IO is connecting and disconnecting servers. This was done due to the same reasons as when creating new servers.

The server editing parts of the admin module are related to configuring how Web Commander handles the server, such as the plugins you want and their settings, and not gameserver-specific things like max player number or friendly fire settings, which are configured from the server module’s Settings page.

The users part of it is quite self-explanatory: You can edit users and choose what access rights they have on each of the servers. Web Commander implements a simple access check solution which supports having a “super admin” with access to all servers under an account, “server admins” who have full access to a specific server, and finally a more gradual access where you can choose what specific tools a user can use (eg. if they can kick a player or edit the map list).

Otherwise it’s mostly your standard admin panel type of functionality with some KnockoutJS sprinkled in for ease of use.

In closing

Although I hadn’t done much node stuff before this project besides some simple experiments, I think the choice of tools for this was pretty much perfect.

There have been no major issues working with node, problems mainly rising from the poor documentation of how the Battlefield 3 server protocol works.

The plan with this currently is to go forward into a SaaS model with paid subscriptions. This will bring some additional complexity to this in the form of having to handle payments and tracking if anyone has not paid, and disabling their servers/account if so.

We’ll see how that goes. Hopefully later this month.

What do you think? Is there anything else you’d like to hear about this? I can do another post with more details on any specific parts.

However, sometimes you need to customize formatting of values, or add other custom logic to it. Using a filter for this purpose is quite convenient, but Dojo’s implementation is completely undocumented as to how you would add your own.

Turns out it’s actually quite easy.

Step 1: Create a filter

All DTL filters in dojo follow a convention like below:

dojo.provide('my.dtl.filter.examples');
 
dojo.mixin(my.dtl.filter.examples, {
    exampleFilter: function(value) {
        //do something with value and return the result
        return value + 1;
    }
});

Doing this, we have a filter called exampleFilter, which adds 1 to a value passed to it, for example..

    <p>{{ someVal|exampleFilter }}</p>

The output of the template above would be the value of someVal with 1 added to it.

Step 2: Register your filter

To register our custom filter(s) with dojo…

dojox.dtl.register.filters('my.dtl.filter', { examples: ['exampleFilter'] });

In other words, you pass the base namespace as the first parameter, and then an object containing the filters.

In the object, the key should be the name of the object containing the filters, like ‘examples’ in our case, and the value should be an array with names of filter functions contained in the object in it.

Step 3: Profit

You’re done. Enjoy your shiny new custom dojox.dtl filter!

Adding custom tags works in a similar fashion, however implementing them is a bit more complicated. Refer to the built in tags to find out how to do it. To register tags, simply use dojox.dtl.register.tags instead of dojox.dtl.register.filters

Then they are disappointed when it isn’t, acting as if it’s a really bad thing.

But why do things need to be integrated to begin with?

Background

A lot of people asking for these integrations seem to be from a Ruby on Rails background. It’s understandable considering there are lots of integrations for all sorts of libraries like unit testing tools and such for RoR.

However, I think it’s actually not such a great idea to integrate everything.

A very good example about this was quite recently when a person was asking if there’s an integration of any browser testing tools for Zend Framework. Something like Selenium or such.

As far as I know, there aren’t any for ZF, but why do you even need one?

This person was a Ruby on Rails developer, and had already learned a browser testing library, which was integrated into RoR itself. Infact, it was integrated so deeply he couldn’t use it for anything but RoR itself.

What if there is no integration?

Now, what if there was no such integration at all for Rails?

He probably would have learned to use Selenium. After learning to use Selenium, he would have been able to use a tool which is completely technology agnostic. Selenium works with Rails, ZF, Django, anything you throw at it. It’s a much better skill to have than some Rails-specific library.

One could argue that integrated libraries give you a productivity boost. While it may be so, I think the main boost you get is the very first steps: It’s easier to get started, but after that the benefit fades. In the long run, it may even turn into a poor investment in general, as you could have learned a general purpose tool instead.

So next time you’re looking for something that integrates into the framework of your choice, consider the other options too. It may prove much more beneficial.

However, after thinking about it for a while, I decided to actually try CoffeeScript and see for myself. Now, after using it quite extensively on some projects, I feel I’m qualified to give my completely unbiased, 100% accurate and correct opinion on it.

How CoffeeScript compares to JavaScript

CoffeeScript is actually a quite neat language. It uses indentation instead of braces, making it kind of Python-like, and various other niceties, which remind some other languages like Ruby.

Using it feels like you need to write much less code to achieve the same results, especially when working with lots of callbacks or in functional style code (array map, filter, etc.).

I’m not going to dive deeply into CS syntax here, but consider the following which demonstrates this quite nicely:

itemIds = items.map (i) -> i.id

vs.

var itemIds = items.map(function(i) { return i.id; });

Although the CS example above contains some syntax which may look a bit odd at first, it’s much, much less “noisy”, letting you see what’s happening more easily with a glance.

In addition to just the syntax, it includes many useful language-level features for things like for-each loops, “splats” or binding variables from arrays or objects.

And, believe it or not, the compiled JS code it produces is actually quite tolerable and follows good conventions.

CoffeeScript does not change JavaScript into something else

Although CoffeeScript works in places to hide some of JavaScript’s perceived ugliness, it still is not a complete replacement.

It does not even attempt to entirely hide JavaScript from you: To be able to use CoffeeScript, you also need to know certain things from its JavaScript roots.

Two primary examples are closures and the binding of this

Binding of ‘this’

CoffeeScript provides a convenient way to bind anonymous functions into the current scope, using the => operator when declaring the function.

What this does is essentially the same as doing someFun.bind(this). (It does not compile to that exact code though).

However, without knowing how JavaScript’s scope binding works, you would not know when, why or how to use =>

Closures

Everyone who has used JavaScript has probably fallen victim of the closure in a loop problem.

Contrary to what you might expect, this same thing is present in CoffeeScript.

I spent a good 30 minutes debugging some CS code I wrote, only to realize that this was the cause. Imagine my headdesking.

Essentially the fact I was using CS made me completely oblivious to the cause of the problem. If it had been JavaScript, I would’ve spotted it much sooner.

So what is the point of CoffeeScript

To make writing JavaScript more fun!

Considering CoffeeScript lets me write more concise code, provides a bunch of nice features and is overall a quite well thought out language, it does indeed make writing JavaScript more fun to me.

Since it does not attempt to hide JavaScript from me behind some fancy schmancy syntax, it’s still ever as flexible as JavaScript is, letting you do confusing things with closures and mess with prototypes and whatever.

So it’s not is a replacement for knowing JavaScript. If you’re a beginner in JavaScript, I would suggest you learn to be good in JavaScript first, and then you can consider learning CoffeeScript if it suits your purposes.

Blog stats for 2011

Comparing to last year, the visitor stats remained pretty much the same, with 201 499 visits and 267 478 page views.

The top five posts were:

1. Handling errors in Zend Framework
2. Zend_Acl part 1
3. Zend_Acl part 3
4. Using canvas to do bitmap sprite animation in JavaScript
5. Zend_Form decorator tips

Zend Framework content is still the king here it appears. In fact, the Zend_Acl series and the error handling post have been in the top five for three years in the row now! Zend_Form decorator tips was also in the top 5 back in 2009, and while it was not in the top 5 in 2010, it was still in the top 10.

The completely new and surprising comer is the post on bitmap sprites. It was not in the top 10 in the previous years, not close to it at all. So why is it here this year? My guess is that because of the sudden HTML5 boom of late, people are looking more and more into using canvas for their games or animation purposes.

All of the top five posts were (again) older content, written in 2009 except the decorator one which is from 2008. The most popular post I’ve written in 2011 turned out to be my take on the discussion on fat controllers and models, which ranked on place #28.

Personal stuff for 2011

Last year I mentioned starting a new job, well, I’m still there and probably still going to stay for the foreseeable future.

In December I also launched my nodejs application, Battlefield 3 Web Commander. You can read a bit about it here. It’s probably the biggest personal project I’ve had for a while. We’ll see what comes of it in the long run

As for the blog, I’m hoping to have some more stuff to post here than last year, but we’ll see. I do have a few posts planned, but I’m not sure if I can top my best year of 2009 when I had time to post more than once a week.

Prerequisites: Some understanding of programming languages and related concepts.

1. Start by bashing some language. PHP is a pretty easy target and you can pull all sorts of issues with it with a few quick google searches. Security issues are generally pretty easy for this, or syntax or OOP, or lack/poor implementation of some programming paradigm.
2. At this point you can counter any arguments by pulling some functional or academic language and claiming if language you chose for #1 was not created by an idiot, it would not have any issues, like the language you chose for this point.
3. Namedrop Dijkstra or some other well known computer scientist.
4. Insult the intelligence of someone participating in the discussion. Claim it is not an insult.

At this point if the community does not have anyone around who actually knows anything about this stuff, you should be well off to pissing off many people!

Remember to be vague enough so that your actual lack of knowledge in above points is not obvious.

Loosely based on a common pattern I’ve seen used by trolls on several IRC channels. Enjoy!

Well, it’s because I’m lazy I’ve been working on something else:

Behold, Battlefield 3 Web Commander!

It’s essentially a Battlefield 3 server live tracking / admin tool service. It’s powered by nodejs and MongoDB, using libraries Express, Socket.io, Mongoose, Knockout, and a few others.

Do check it out, let me know what you think, and if you own a Battlefield 3 server, be sure to sign up for the beta