How to unit test NodeJS HTTP requests?

Tags:

I have a nodejs app where I wanted to unit test some HTTP requests. Node usually makes things simple, so I expected this to be like that too… but I soon discovered that this was not the case at all.

When unit testing, you don’t want HTTP requests to go out and affect the result. Instead you create a fake request – a test double – to replace the real one, but because node’s streams are hard to fake, this is where I hit the wall.

It took some work, but I found a way to make it simple! Let me show you how with some examples based on real live production code.

  1. Sending a GET request and testing response handling
  2. Sending a POST request and testing the parameter behavior
  3. Testing that failures are handled correctly

Example scenario

I based this and the tests on real world code and simplified them so they’re easy to follow. But the core techniques are the same as I use in real production code.

I’m going to call this file api.js

var http = require('http');
 
module.exports = {
	get: function(callback) {
		var req = http.request({
			hostname: 'jsonplaceholder.typicode.com',
			path: '/posts/1'
		}, function(response) {
			var data = '';
			response.on('data', function(chunk) {
				data += chunk;
			});
 
			response.on('end', function() {
				callback(null, JSON.parse(data));
			});
		});
 
		req.end();
	},
 
	post: function(data, callback) {
		var req = http.request({
			hostname: 'jsonplaceholder.typicode.com',
			path: '/posts',
			method: 'POST'
		}, function(response) {
			var data = '';
			response.on('data', function(chunk) {
				data += chunk;
			});
 
			response.on('end', function() {
				callback(null, JSON.parse(data));
			});
		});
 
		req.write(JSON.stringify(data));
 
		req.end();
	}
};

This should look familiar if you’ve used the http module. One function for fetching data, another for sending data. The code uses the JSONPlaceholder API, a simple REST API useful for this kind of prototyping and testing.

Necessary libraries

For testing we’ll only need two extra libraries: Mocha and Sinon, which are used as the test runner and mocking library respectively. A mocking library such as Sinon will allow us to create test-doubles easily, which can otherwise be time consuming.

npm install mocha sinon

Setting up the test suite

First, let’s create the test suite. We will use this to contain each individual test, and do some basic setup. I’m going to put this file into tests/apiSpec.js.

var assert = require('assert');
var sinon = require('sinon');
var PassThrough = require('stream').PassThrough;
var http = require('http');
 
var api = require('../api.js');
 
describe('api', function() {
	beforeEach(function() {
		this.request = sinon.stub(http, 'request');
	});
 
	afterEach(function() {
		http.request.restore();
	});
 
 
	//We will place our tests cases here
 
});

We will use the builtin assert module as a validator for our tests. sinon is our mocking library. PassThrough is a simple stream, which we can use as a test double for other streams. We will look at the use of http next.

The key things here are beforeEach and afterEach. beforeEach creates a stub to replace http.request, and afterEach restores the original functionality.

A stub is a test double that lets us define behavior and track usage – set return values, parameters, check the call count, etc.

But how does this help? We created the stub here, not in the api module! NodeJS caches required modules, so the changes here are reflected in other modules. In other words, api.js is forced to use our stub.

We save the stub into this.request, so we can reference it later.

Testing the GET request

Let’s start by adding a test that validates the get request handling. When the code runs, it calls http.request. Since we stubbed it, the idea is we’ll use the stub control what happens when http.request is used, so that we can recreate all the scenarios without a real HTTP request ever happening.

it('should convert get result to object', function(done) {
	var expected = { hello: 'world' };
	var response = new PassThrough();
	response.write(JSON.stringify(expected));
	response.end();
 
	var request = new PassThrough();
 
	this.request.callsArgWith(1, response)
	            .returns(request);
 
	api.get(function(err, result) {
		assert.deepEqual(result, expected);
		done();
	});
});

First, we define the expected data that we will use in our test and create a response stream using PassThrough. To copy how http.request works, we write a JSON version of our expected data into the response and end it.

We create a PassThrough stream for the request as well, but we only need it as a return value.

A successful http.request will pass the response to the callback. Looking at the earlier example, the callback is the second parameter, so we tell the stub to call it with the response as a parameter. The numbering starts from 0, so using 1 refers to the second parameter. A call to http.request always returns a request stream, so we tell it to return ours.

With the setup out of the way, we call api.get with a callback which verifies the behavior. The assert module is built-in to node, but if you prefer you can use your favorite assertion library, such as chai, instead.

We can run the test using mocha. You should find the test passes with a green light.

Testing the POST request

For the POST scenario, we want to ensure the parameters are passed correctly to the http request.

it('should send post params in request body', function() {
	var params = { foo: 'bar' };
	var expected = JSON.stringify(params);
 
	var request = new PassThrough();
	var write = sinon.spy(request, 'write');
 
	this.request.returns(request);
 
	api.post(params, function() { });
 
	assert(write.withArgs(expected).calledOnce);
});

When calling api.post, it should send parameters in the request body. That’s why in this test, we only need to verify request.write is called with the proper value.

Like previously, we define the expected data first. Doing this helps make the test easier to maintain, as we’re not using magic values all over the place.

We then define a request stream. To make sure the expected data is written into it, we need a way to check write was called. This calls for another test double, in this case a spy. Just like real secret agents, a spy will give us information about its target – in this case, the write function.

The http.request function only needs to return the request stream in this test.

We call api.post, passing in our parameters and an empty callback. Why an empty callback? Again, we only test what we need, and here we don’t need to test the callback.

Finally, we use assert to check the values from the spy: We check that the spy was called once with the correct parameters.

We can run the tests again using mocha, and they’ll pass.

Note: in a real app, you might want to have a test for the callback, but it should go into its own test case. A single test case should ideally have one assertion only. The test for a callback would resemble the test with a get request, so I won’t go into detail on it here.

Testing a failure scenario

And last but not least, we should include a test for a failure.

Most http request testing related libraries on npm had difficulties solving this, but using the PassThrough stream solves it cleanly as we’ll see.

it('should pass request error to callback', function(done) {
	var expected = 'some error';
	var request = new PassThrough();
 
	this.request.returns(request);
 
	api.get(function(err) {
		assert.equal(err, expected);
		done();
	});
 
	request.emit('error', expected);
});

This should be starting to look familiar. We define the expected data, create a request stream, tell our stub to return it and call the api. The callback passed to the api checks the error parameter is correct.

The new thing here is the last line. When an http request fails due to network errors, http parsing errors or such, it emits an error event. To simulate an error, we emit one from the test.

Run tests with mocha, and they’ll… fail? Huh?

This is why unit tests are important, and especially testing the failure cases! It turns out a bug snuck into the code.

No problem, let’s add handling for the error event.

get: function(callback) {
	var req = http.request({
		hostname: 'jsonplaceholder.typicode.com',
		path: '/posts/1'
	}, function(response) {
		var data = '';
		response.on('data', function(chunk) {
			data += chunk;
		});
 
		response.on('end', function() {
			callback(null, JSON.parse(data));
		});
	});
 
	req.on('error', function(err) {
		callback(err);
	});
 
	req.end();
},

The only thing added here is the three lines to handle the error event.

Run tests with mocha and now they will pass. Done!

Conclusion

Writing tests like these takes a bit of practice. You need to know how the stubbed and mocked functions work, but with experience it will become easy. The great thing is that you can apply these methods to test other things too – MySQL queries, Redis lookups, Ajax calls, TCP sockets… the list goes on and on.

I’ve made the example module and tests available for download from here for your convenience: api.js, apiSpec.js.

Any questions? Email me or leave a comment and I’ll help you out.