All sensible programmers agree that Unit Testing is an essential part of the development process but there is a great range in opinions as to how this is most effectively done. In this essay I describe a disciplined but not over-zealous Unit Testing policy for software written in Java. GrandTestAuto is designed to implement this policy.
Here are some characteristics of an effective testing policy:
The sections below we expand on these points. Later on, I'll explain how the design of GrandTestAuto addresses them.
Any testing process where somebody must examine the output of a program is going to be slow, tedious and error prone. Any test that requires a user to provide input will have the same problems. For these reasons, all software testing should be as automated as possible, and unit tests certainly should just run and then provide a true/false result as to the validity of the software being tested. This point is well explained in the documentation for JUnit.
Apart from the compiler, Unit Tests are the quickest and cheapest way of discovering programming errors. For this reason, Unit Testing should be the step in the build process immediately following compilation. There is another sense in which Unit Testing should be done early: Unit Tests for classes should be written before or with the classes they are testing. In the eXtreme Programming (XP) methodology, unit tests are written first. Various XP resources sites argue the benefits of frequent and early unit testing.
In order to be able to release a jar file for an application that is free of test code, there must be an easy way to separate application code from test code. This could be achieved by having the test code in separate packages (which has problems) or by a naming pattern for test classes which are mixed in with application classes (either as classes in the same package, or as inner classes).
If test classes are in separate packages, then the classes being tested must be public in order to be accessible to the test classes. This is at odds with the software engineering principle that the accessibility of classes and methods should be as restricted as possible. The tension here between over-exposing and under-testing is a result of by the crude accessibility options for classes in Java.
Any lists of unit tests to be run will be a maintenance problem, and the cost of missing a test can be very high. For this reason, the test software should eliminate all such lists (and all other 'accountancy' associated with tests).
Any class important enough to be made public should be unit tested. Typical objections to the unit testing of a class are:
Even the best developers make mistakes, and truly simple classes are simple to test. So no class is too simple to be worth testing. Moreover, simple classes often evolve into more complex entities and unless there are unit tests in place, these changes will introduce errors.
As to classes that are too complex to be unit tested, this usually means that the class in question does not represent a single well-defined abstraction and so could do with being re-designed. One of the benefits of unit testing is the increased discipline that it brings to design and implementation above the mere fact of detecting errors.
As to the third case, the class should not be public and by GTA's reckoning, need not be tested.
Every public or protected method of a testable class should be tested. There is a school of thought that truly simple methods, like getters and setters, do not need tests. This argument misses the point that unit tests are not just testing the current version of the code, they also guard against errors introduced as software changes. For example, if the implementation of a method changes from being a getter to a calculation, and there is no existing unit test for the method, it is quite likely that the method will remain untested and that bugs will be introduced. In any case, truly trivial methods are very easily tested.
A class constructor is a method for creating an object that is in a known and well-defined state. As such, all accessible constructors should be tested.
GrandTestAuto is designed to implement the unit-testing policy described above but to otherwise make very few demands on the design of unit tests.
GTA searches a classes directory for packages and then looks for test sub-packages for each of these. If a package is not accompanied by a test package, then the overall unit testing result will be 'false'.
For each package, GTA employs Java's reflection API to search for the public non-abstract classes. A test class is sought for each of these. If no test class is found, the overall testing result will be 'false'.
For each class to be tested GTA again employs reflection to find the testable methods (including constructors). These are the methods that are public or non-final and protected. Test methods are sought in the test class for each testable method. If a method is not tested, the overall testing result is 'false'. The test methods are always public, return a boolean and take no parameters.
In the reflective search for test classes and test methods within them, naming patterns are used. The naming pattern for test classes is very simple: for class X the test class should be called XTest. The naming pattern for methods is more complicated. It is decribed in the tutorial for GTA (and also in the javadoc).
The use of reflection means that the unit-tests require no hard-coded lists of sub-tests. Moreover, it ensures that as classes change, the unit tests track them. This imposes a discipline on developers with software engineering benefits beyond the simple fact of thorough testing. These benefits are examined in a later section.
The testing of only public classes and accessible classes within them raises some problems. The major one is that you might have a class or method that you want to unit test but otherwise would not want it made accessible beyond its defining package. Having too many public classes and methods becomes a terrible maintenance issue in a large project. In my experience, the best solution is:
When one is working on a class and knows that for each accessible method a unit-test must be written, one tends to write as few accessible methods as one can get away with. This leads to enormous benefits should the class ever need to be changed. In a large project, changing a class that has lots of public methods is a nightmare of searching for method invocations and assessing how the changes will affect each.