Another Good Review for Swing Extreme Testing

Our book Swing Extreme Testing has received another good review, this time from Alex Ruiz. Although the review is positive overall ("The book is well-written, interesting and useful."), there are a few criticisms of the book that we think are unwarranted, which we respond to here. Also, the review does not make any mention of a number of important topics covered in our book, that are not covered in any other books, and that will be of great value to Java developers.

We routinely run thousands of tests with no such issues arising. Perhaps the FEST approach differs in some subtle but problematic way.

We are totally in favour of this approach, as far as it can be practically applied. For example, on page 48 we write "... we prefer to write the test for a method before the method itself...". However, we also acknowledge that if we follow the 'test first' dictum slavishly, then at some point we have a test that does not compile, and certainly cannot be run.

This is the way we have arranged our source and test code, but in the book we acknowledge that a lot of people will prefer the 'parallel hierarchies' approach. This doesn't seem like a hugely important issue to us.

On the contrary, our policy is not to have explicit tests for package-private and private classes and methods. Our policy is to test all public and protected classes and all of their public and protected methods and constructors. This is already a huge amount of testing and it is certainly sufficient to ensure complete code coverage, as we demonstrate in the book. On the rare occasions we feel we need to test a package-private class, we either use reflection or take the pragmatic step of making it more accessible.

We know that exposing code to make it more testable is deeply offensive to some, and it is always a last resort for us. But, as we said, we almost never need to do it. Of course, the real problem here is that the visibility options available in Java are very coarsely defined, which JSR 294 (Improved Modularity Support in the JavaTM Programming Language) may alleviate.

There are several advantages of GTA.

First, it runs unit, function and load tests, that is the three levels of testing we use in our day-to-day development of LabWizard. At the unit test level, it ensures test coverage, as explained elsewhere on this site and in the book. Our book is not just about unit testing, and we need a tool that can cope well higher levels of tests.

Second, it is extremely flexible and with just a few keystrokes we can run either all of our tests, or just those for some packages, or just those for some classes within a package, or some methods within a class. We have so many tests and run them so frequently that this kind of flexibility is really essential.

Third, with GTA we can distribute the tests across a number of machines.

Fourth, GTA is less intrusive than other tools that we've seen.

Finally, why should we always use the same old tools? Who is sorry that people got sick of make and developed ant?

We do agree that the lack of IDE support for GTA is unfortunate, but this does not really bother us, as we run our tests from the command line or using CruiseControl.

Of the twenty chapters in the book, only one is directly concerned with GTA. It seems odd to criticise the use of this tool and not mention so many of the other chapters (see below).

Assert statements are useful, but are in no way a replacement for a full-powered Design-by-Contract tool. Anyone who thinks otherwise is kidding themselves. In the early days of our project we made extensive use of a tool called iContract, that allowed us to insert preconditions, postconditions and class invariants as executable JavaDoc comments. (We give an example in the book of the kind of really powerful contracts we were able to make using iContract.) This tool really was valuable, but is no longer supported. In the best of all possible worlds, we would still be using something like iContract.

In the absence of true Design-by-Contract support in Java, we agree that the assert keyword is better than nothing.

Actually, since our book was published, we've started making a lot of use of the @NotNull and @Nullable annotations provided by JetBrains. These annotations show up in code-completion and parameter help dialogs displayed by IntelliJ, which makes them even more visible than JavaDoc comments. Maybe these annotations will develop further into a proper DBC tool. We live in hope.

Our LabWizard software can be run using either a relational database (MS Access) or a BTree database (InfinityDB) to persist the KnowledgeBases. All database code is confined to a single package that only has a few really big classes in it. There is no reference to databases in any of the other hundred-odd packages comprising the software. We think that this is an outstanding example of the underlying storage technology being hidden from the rest of the application.

The criticisms we've responded to above are in fact pretty slight. Is the arrangement of test and source code really worth getting excited about? No. Our main source of disappointment with the review is that it did not mention that our book deals with:

• Application-level testing. This is the tracing of requirements to test specifications to actual Java classes that prove that our application meets its requirements. While there are a lot of books on unit-level testing, not much has been written about automated tests of an entire application. When we write serious code for serious paying customers, we need traceability and automated testing of the requirements that our customers are paying us to implement.
• Load testing. We've got a short but sweet chapter on this, relating heavily to how we use load tests in our day-to-day development processes.
• Internationisation: We introduce a design pattern that allows unit-testing of properties-files. These unit tests ensure that all problems with missing resources are found in the unit tests, that mnemonics are correctly defined, and so on.
• The Help System: We provide a tool that guarantees that all JavaHelp files are indexed and available and that they contain no missing images or broken links. Further, we show how to test that the correct help file is displayed when context-sensitive help is launched.
• Testing email, spreadsheets, PDF, log files and web services.
• How to track down intermittent problems with tests.
• Lots of other things... have a closer look!

Overall, we're grateful that Alex Ruiz has a lot of positive things to say about our approach to the testing of Swing user interfaces. However, we're disappointed that he spends so much of his review in what seem to us to be petty criticisms (the layout of test code, not using JUnit, not adhering too slavishly to a test-first approach etc.) and doesn't make mention of a lot of the other material in the book that we think a lot of readers would be very interested in.