Some time ago I stumbled upon some crazy stuff…
Specifically, I found out that SQLite has 787 times more tests than they have actual code!
It’s no joke, it’s documented right on their website. While they have about 116 300 lines of source code, they have 91 577 300 lines of test code.
That sounds completely insane.
I bet you’ve sometimes wondered what is the right amount of tests to write. In some cases, it’s easy to end up with more tests than code.
But 787 times more tests than code? What kind of crazy nonsense is that?
If you think it sounds crazy, you would be absolutely right. For almost any kind of project, that number is crazy.
When thinking of how many tests is enough, we need to think of what the goals are – both for the tests and our actual project.
SQLite is a database. While as web developers we might be more familiar with MySQL or MongoDB, SQLite is also heavily relied upon in many places. For example, your phone most likely uses it. Maybe even your car!
You can probably imagine reliability and performance are critical for databases. Much more so than most applications. If you look at what kind of tests SQLite has, you’ll see things like “Out-of-memory tests”, “Crash and power loss tests”, “Undefined behavior checks” and so on.
When we build JavaScript apps – or almost any kinds of web apps really – we don’t need any of that.
What possible benefit would we get from having out of memory tests for an app which can safely crash and restart and there’s no harm done?
Even in the world of databases SQLite is a bit of an outlier. PostgreSQL’s tests cases cover only about 70% of all of its code. This means they probably have more tests than code, but definitely not as much as SQLite.
—–
So how can you figure out what the right amount of tests for your project is?
Let’s talk about unit tests first.
The best thing to do of course would be to use test-driven development. This way, you naturally end up with a good amount of tests which should cover most of your code too.
But there are places where TDD is hard. If you’re writing code which directly deals with modifying the page – for example via jQuery or even React – it can be difficult to test that.
Is it worth the effort to bend over backwards, to build clunky tests for something that just is really hard to test?
It would be good because hard to test code is exactly where you get bugs easily… but if you spend hours trying to figure out, it might not really be worth it.
Here’s two things:
- Consider refactoring. Check here for a pattern which can help
- Just leave it. If it turns out to have a bug later, spend the time then as you have a justified reason.
But what about if you’re not using TDD? Perhaps you have an existing codebase which doesn’t have tests.
In codebases like that….
- Write regression tests for any bugs you fix
- Write tests for any new code
- If you write new code that uses existing functions, then add tests for those existing functions
1 and 2 here give you the best bang for your buck. #1, you definitely want to know if a bug you fixed is fixed for real. #2, adding tests for new code is often fairly easy, so why not?
#3, often old codebases require some refactoring from time to time – especially when you try to test them. By adding tests to functions you use, you start to add a system in place that ensures the new code you added works correctly, and can help you in the future if you refactor the code.
Now, with the unit level covered… should we add any other types of tests?
The best way to go here is based on use and problems. If you have some particularly complex piece of code, or some area where bugs occur often, then you can consider adding integration or functional tests for those areas.
Other than that, unit tests are often more than enough for almost all kinds of web apps!
As a last pointer… don’t worry about it!
I’ve seen many people ask “well, I have this function and these tests for it… but my tests are longer than my function. Is this ok?”
Yes! Absolutely! :)
And honestly, is having more tests than code really going to be a problem? I’ve never seen that happen.