What is property based testing (and how to do it in JavaScript)?

Ever wonder… what if you didn’t have to write specific tests with hardcoded test data?

Or have you ever written a function which can take many different values as input? How can you test something like that easily with different values?

These are the kinds of important questions in life that keep me up at night! (ok maybe not)

But did you know both of these questions can be solved with something that’s called “property based testing”? That’s right – property based testing can help you test the untestable. And while at it, you can even use them to test your other functions with automatically generated data.

All of this means property based tests are great for finding those pesky corner cases that are easy to forget. This means less debugging weird bugs that you find in production – you know, the really hard to reproduce ones, which occur for only one user in some weirdly specific combination of settings.

Sounds pretty cool huh? Let’s find out some more!

What exactly is property based testing?

Often when you have some function, their behavior has certain properties, which are kind of like rules the function’s output always follows.

For example, what if we have a function which calculates some basic geometric properties, such as the area of a rectangle based on its width and height?

Traditionally, we might approach testing this kind of a function by hardcoding a number of width/height pairs and the expected result values. But there’s always a possibility our values wouldn’t include all possible cases. For example, what if one of the values is a float? Or one of the values is negative? Or both?

Computers are much better at generating lots of different values. What if we just told it what kinds of inputs our function accepts and give it some way of verifying the output?

Let’s think of some properties for the rectangle area function:

1. Given any two width and height values, the result is always a multiple of the two
2. Order doesn’t matter. A rectangle which is 50×100 has the same area as one which is 100×50
3. If we divide the area by width, we should get the height

Here we have three examples of how we can verify the test result for any given input without having to know the inputs up front. All of these should always hold true for any given valid input.

Property based testing is more common in the functional programming world. For example, the library we’ll be using later is based on a similar library in Haskell. In languages like it, it’s even possible for property based test tools to automatically figure out what kind of data is valid input for your function. Unfortunately this isn’t something JavaScript can do, but that doesn’t mean property based testing can’t be useful for us JavaScript developers too.

Before we go deeper into the topic, let me give you a “plain javascript” style example of what we want to do.

//this function is used to test the "property"
function areaIsMultiple(width, height, result) {
  return result === width * height;
}
 
//we can have inputs generated in code...
var width = Math.random();
var height = Math.random();
 
//and run a test:
var result = rectangleArea(width, height);
 
console.log(areaIsMultiple(width, height, result));

This is just a pseudo-code style sample to give you an idea of how this type of test would work, but it looks pretty easy, right?

The basic idea of property based testing is right there:

• Tell the test what values the function accepts – for example, any number
• Have some way of verifying the result for given inputs – for example, by multiplying the inputs and comparing to the result

Writing property-based tests with node-quickcheck

Now that we’ve got an idea of how property based testing works, let’s take a look at how to put it into use with node-quickcheck.

In the earlier code sample the test wasn’t very useful. We would need to run it a hundred times to even get enough random numbers. That’s why we’ve got tools like quickcheck, which we can use to run a hundred tests for us completely automatically. Quickcheck will even help us generate values and verify the test results more easily.

Let’s start by writing the example test into something that actually works. Below, you can see a CodePen with our example modified for node-quickcheck.

First, I’ve defined the rectangle area function. Yeah, it’s really basic – you definitely can test much more complex functionality with quickcheck, but let’s keep things simple for now.

You can ignore the mocha.setup and mocha.run lines around the code – they’re simply used to enable us to run the Mocha test suite. You can also write this as a Node.js based test if you so feel like, and run it using mocha from the terminal.

Then, we’ve got a test. It should calculate correct result for any given width and height.

Within the test we first have a function called areaIsMultiple. This function takes two parameters and returns true when rectangleArea‘s result is correct for those two parameters.

And now comes the tricky part. We use qc.forAll to run the actual test. It’s kind of like saying “X should be true for all cases of Y” – or in this instance, we’re saying “area is multiple should be true for all cases when given two arbitrary doubles as parameters”.

The first parameter we give to qc.forAll is the verification function – in this case, areaIsMultiple. All other parameters should be functions which return some type of values to test with. In this case, we pass qc.arbDouble two times. This is simply a function which generates an arbitrary double – as in, any valid JavaScript number.

Finally, we use should(result).be.true() to validate that our forAll check was successful.

Most of the work happens in qc.forAll:

• forAll takes the first parameter, and calls it using the remaining parameters
• areaIsMultiple(qc.arbDouble(), qc.arbDouble())
• It runs this 100 times
• If any of the 100 runs fails, it gives you an error with the values that failed
• Otherwise it returns true to indicate success

You can try and see what happens if you change areaIsMultiple to do something else. You’ll get an error which has an array such as [ 123, 653 ]. This means that the call areaIsMultiple(123, 653) returned false.

You might have some questions at this point. Let me try to address the most common ones:

• But tests shouldn’t have randomness in them! – Yes, normal unit tests shouldn’t have that. However, in property based testing it’s normal, and tools like quickcheck help eliminate the problem (because it tells us the values that failed)
• Isn’t running the test 100 times going to slow them down? – Possibly. However, in most cases with property based tests, you test functions where it won’t be a problem. Typically, tests that are complex are slow. Using many other functions, stubs and mocks, etc. also slow tests down. Typically you won’t be using those much in property based tests.

A more realistic example

The rectangle area calculation is fairly trivial. Why don’t we look at something more interesting – and more relevant to web development.

You might’ve heard of a jQuery plugin called Masonry. It essentially lays out different sized boxes into a layout which resembles a set of bricks laid out by a mason.

What kind of properties would this kind of layout algorithm have?

• None of the contents should overflow horizontally from the container
• Contents should not overlap with each other

These are just two ideas, but we can use these to explode how you could approach testing this with property based tools.

Let’s say we have a function calculateMasonryLayout, which takes an array of items as its parameter, and returns an array of coordinates and dimensions of how the contents should be laid out.

var items = [
  { width: 100, height: 200 },
  { width: 50, height: 50 }
];
 
var containerWidth = 100;
 
var layout = calculateMasonryLayout(items, containerWidth);
 
console.log(layout);
// [
//   { x: 0, y: 0, width: 100, height: 200 },
//   { x: 0, y: 200, width: 50, height: 50 }
// ]

Given a list of items with a width and a height, we can pass it into the function and it’ll give us a new list with x and y coordinates in exchange.

There’s a lot of moving parts in this. Let’s see if we can fit this into node-quickcheck. We had two properties for this – let’s implement a test for none of the contents should overlap with each other, since it’s a lot more interesting than the one about container overflow.

The first thing we need is a way to generate valid parameters for the function.

The first parameter is an array of items. All of the items have a width and height, which can be some non-negative whole number – in other words, an unsigned int. The array can be of any length, it doesn’t matter for the layout algorithm.

The second parameter – container width – can also be some non-negative integer number.

How could these values be generated in code?

function arbUnsignedInt() {
  return Math.floor(Math.random() * 1000);
}
 
function itemArray() {
  var len = Math.random() * 50;
  var arr = [];
  for(var i = 0; i < len; i++) {
    arr.push({
      width: arbUnsignedInt(),
      height: arbUnsignedInt()
    });
  }
 
  return arr;
}

Here we have two functions to generate values for us. First, arbUnsignedInt generates an unsigned int. Second, itemArray generates a random length array of items with random dimensions.

We can use these functions to produce valid inputs for the calculateMasonryLayout function.

Next, we need a function that can validate the result of the test. Since each of the result items define the coordinates and dimensions of a rectangle, we need some way of checking that none of the rectangles overlap with each other.

function noIntersection(a, b) {
  return a.x + a.width < b.x || b.x + b.width < a.x || a.y + a.height < b.y or b.y + b.height < a.y;
}
 
function layoutHasNoOverlap(items, containerWidth) {
  var layout = calculateMasonryLayout(items, containerWidth);
 
  return layout.every(function(item, index) {
    for(var i = 0; i < layout.length; i++) {
      if(index !== i && !noIntersection(item, layout[i])) {
        return false;
      }
    }
 
    return true;
  });
}

The noIntersection function is used to determine whether two rectangles intersect. We use it in layoutHasNoOverlap as a helper to determine just that.

layoutHasNoOverlap runs the masonry layout algorithm and then compares the result, checking for rectangle intersection. The way I’m doing the comparison here might not be the most efficient way of doing it because it does a lot of looping, but it should suffice for this example.

All we need to do now is plug these functions into quickcheck:

describe('calculateMasonryLayout', function() {
  it('should produce results which do not overlap with each other', function() {
    should(qc.forAll(layoutHasNoOverlap, itemArray, arbUnsignedInt)).be.true();
  });
});

As you can see, testing even more complicated logic using property based testing is possible. Unlike some other types of testing, it does require us to do some thinking up-front:

• What is the range of valid inputs for this function?
• How to validate the result?

Solving these questions wasn’t too tricky. We just wrote some reasonably straightforward functions, which then plug into node-quickcheck.

Conclusion and next steps

Property based testing can be a useful tool in the testing arsenal. It isn’t suitable for testing all kinds of functions, but it’s great when you can verify a function’s result using some kind of calculation or condition. If you want to sound very fancy, you can say property based testing is great for testing function invariants.

Just remember the basic steps:

1. Figure out what is the range of possible valid inputs for the function being tested
2. Write a function to generate those inputs, if needed
3. Write a function that can determine if the output is acceptable

The biggest challenge when it comes to this type of testing is thinking of good properties to test. There are many possible candidates – I listed some examples in the article for the functions we looked at, but you could probably think of some others as well.

What’s next?

• You can learn more about node-quickcheck from the project’s GitHub page. Admittedly their documentation isn’t that great, but the library is fairly simple to use.
• If you’re interested in a much more advanced property based testing library, you’ll want to check out JSVerify