How Good Is Your Test Suite?
If you go a long way back with PHP, you're probably saying that back in the day, we did not write tests along with our code. While that might have been acceptable in the past, it simply wouldn't cut it today.
PHP has evolved a lot over the last few years — more and more complex systems are being built with it and shipping without tests is considered bad practice.
Having a test suite allows us to easily verify that the code is working, prevent unintentional future changes because new features were added or existing code was refactored, show us how code works and how to interact with it, and serves as a mentor in good design practices.
It all sounds perfect, but at one point we should ask — if good tests assure our code quality, how can we check if our tests are good enough? Code coverage gives us a number we can display and see if it goes up or down, but does it really tell us how good our tests are?
Without being too philosophical, there are actually two things that you can start doing right away that will help you improve your test suite: mutation testing and property based testing.
What is mutation testing?
Mutation testing is a technique based on producing errors in your codebase and then running a test suite to see if those errors are being caught.
Traditionally, you want your tests to pass. But, while running mutated code, failing tests is a desirable outcome. It shows that your tests are capable of catching the change. Passing tests after the code change shows that tests are inefficient in catching the change.
Every code change is called a mutation and it results in a mutant — changed instance of code on which we run our test suite. If tests fail, mutant is marked as killed. If not, mutant is considered escaped.
A tool that is gaining a lot of traction these days is called Infection. It is based on the Abstract Syntax Tree parser that allows quick and easy code changes, resulting in faster execution and simpler mutators.
Consider having this method in your code.
Infection will apply three different mutators on it.
Function signature mutator will reduce method visibility from public to protected. By doing that, it will verify whether visibility for this part of the API is larger than necessary.
Negated conditionals mutator is pretty straightforward. It will change identical operator to non-identical.
Return values mutator is also very simple, it will change return value from true to false.
Mutation testing also brings a new metric that we can use, along with code coverage, that will paint a much better picture of how good our code and test suite is.
Infection calculates the Mutation Score Index as a percentage of tests that fail after mutation (remember, failing tests is good) and the total number of mutants.
The downside of mutation testing is speed. Mutations have to be generated and, on top of that, tests need to be run multiple times.
While Infection supports threaded execution, it can still take a while on bigger test suites. In your daily development, you will want to filter out parts of the code you are not working on.
All in all, there is really no excuse for not including Infection in your project.
What is property based testing?
When writing unit tests, one can ask themselves, how many tests are enough? One? Forty-two? Is there an edge case I may have missed?
The idea behind property based testing is to write specifications for input and output and then let the testing framework generate thousands of assertions to verify it.
By running your test with thousands of different inputs, you are more likely to find cases where it fails. Also, inputs get to be very sneaky.
You are not just testing with ideal inputs, you are testing with: zero values, nulls, negative values, empty arrays, big arrays, max integers, bad strings, special characters — you name it.
While it sounds ideal on paper, in reality, it takes quite a bit of practice to wrap your head around the concept and to start thinking way harder about the problem you are solving.
If you want to start doing property based testing in PHP, your safest bet is to include Eris in your project. Eris is a port of Haskell’s QuickCheck, the grandfather of property-based testing libraries. It integrates with PHPUnit and it will generate assertions based on your specifications.
Take a look at this simple example, where we are trying to assert that our LegalAge checker is only letting people aged 21 and over in.
As you can see, it is very easy to specify two tests, one that will check if we are denying access to people aged 0 to 20, and another where we allow access to people aged 21 and over. Running these tests can potentially discover a bug in our code.
PHPUnit 6.5.5 by Sebastian Bergmann and contributors.
Runtime: PHP 7.2.0
.F 2 / 2 (100%)
Reproduce with:
ERIS_SEED=1517309391159471 vendor/bin/phpunit --filter 'LegalAgeTest::testCheckGranted'
Time: 34 ms, Memory: 4.00MB
There was 1 failure:
1) LegalAgeTest::testCheckGranted
When age: 21
Failed asserting that false is true.
FAILURES!
Tests: 2, Assertions: 126, Failures: 1.
In this case, you can see that Eris generated a hundred assertions for first test (one hundred being default value) and they all passed. When executing the second one, the assertion failed, indicating that we have a bug in our code.
One thing to notice here is the line:
ERIS_SEED=1517309391159471 vendor/bin/phpunit --filter 'LegalAgeTest::testCheckGranted'
That will tell us the value to use for seeding random generators in order to produce same results. It is easy to forget that Eris works with random input and what is a failing test one run might not be in the next run.
Because of the randomness, it is best to use Eris alongside your example based tests. If you encounter a generated failing test, take those inputs, write them down in a separate test, and fix your code.
It will be much easier to understand later on what happened by reading an example test instead of property based specifications. Also, you can’t expect to solve all of your testing problems with property based tests — they are very good for math problems, very impractical for testing complex business rules.
If you are coming from a different language, do not despair. All concepts shown here are applicable to any other mutation testing or property-based testing library and all languages have their equivalent.
