Another Good Review for Swing Extreme Testing
Our book Swing Extreme Testing
another good review
this time from Alex Ruiz.
Although the review is positive overall ("The book is well-written, interesting and useful."
there are a few criticisms of the book that we think are unwarranted, which we respond to here.
Also, the review does not make any mention of a number of important topics covered in our book,
that are not covered in any other books, and that will be of great value to Java developers.
What He's Not Sure About
"Although the book points out the importance of accessing UI
components in the EDT, I'm not sure their implementation, which
involves calling the method invokeAndWait
,) is completely correct.
We tried a similar approach in FEST
and we ended up with horrible OutOfMemoryError
We routinely run thousands of tests with no such issues arising.
Perhaps the FEST approach differs in some subtle but problematic way.
The Test-First Approach
"The authors suggest that is impossible to create software using a test-first approach...."
We are totally in favour of this approach, as far as it can be practically applied.
For example, on page 48 we write "... we prefer to write the test for a
method before the method itself...". However, we also acknowledge that if we follow the 'test first' dictum
slavishly, then at some point we have a test that does not compile, and certainly cannot be run.
The Arrangement of Source and Test Code
"The authors propose to have test classes in the same root folder as the source code,
with the same package name with ".test" added at the end..."
This is the way we have arranged our source and test code, but in the book we acknowledge that a lot
of people will prefer the 'parallel hierarchies' approach. This doesn't seem like a hugely important issue to us.
Testing Package-Protected Code
"Because of the point mentioned above, the authors suggest using reflection to test package-protected classes."
On the contrary, our policy is not to have explicit tests for package-private and private classes and methods.
Our policy is to test all public and protected classes and all of their public and protected methods
and constructors. This is already a huge amount of testing and it is certainly sufficient to ensure
complete code coverage, as we demonstrate in the book. On the rare occasions we feel we need to test a
package-private class, we either use reflection or take the pragmatic step of making it more accessible.
We know that exposing code to make it more testable is deeply offensive to some, and it is always a last
resort for us. But, as we said, we almost never need to do it. Of course, the real problem here is that the
visibility options available in Java are very coarsely defined, which JSR 294
(Improved Modularity Support in the JavaTM Programming Language) may alleviate.
We Use Our Own Testing Tool
"Instead of using one of the well-known testing frameworks (TestNG or JUnit,) the authors created their
own testing framework: Grand Test Auto. I don't understand why."
There are several advantages of GTA.
First, it runs unit, function and load tests, that is the three levels of testing we use in our day-to-day
development of LabWizard. At the unit test level,
it ensures test coverage, as explained elsewhere on this site and in the book. Our book is not just about
unit testing, and we need a tool that can cope well higher levels of tests.
Second, it is extremely flexible and with just a few keystrokes we can run either all of our tests, or just
those for some packages, or just those for some classes within a package, or some methods within a class.
We have so many tests and run them so frequently that this kind of flexibility is really essential.
Third, with GTA we can distribute the tests across a number of machines.
Fourth, GTA is less intrusive than other tools that we've seen.
Finally, why should we always use the same old tools? Who is sorry that people got sick of make and developed ant?
We do agree that the lack of IDE support for GTA is unfortunate, but this does not really bother
us, as we run our tests from the command line or using CruiseControl.
Of the twenty chapters in the book, only one is directly concerned with GTA. It seems odd to criticise the
use of this tool and not mention so many of the other chapters (see below).
Design By Contract
"The authors promote 'design by contract' (DbC) using Javadoc comments, because, according to them,
using JDK 1.4 assert requires discipline to "check that comments accurately reflect the checks that are being done."
I completely disagree. Javadoc comments for DbC do not have IDE support and are not type safe,
therefore they can potentially be a maintenance nightmare. In addition, I haven't found a DbC tool
that allows me to have the checks and the source code in the same file while keeping type safety."
Assert statements are useful, but are in no way a replacement for a full-powered Design-by-Contract tool.
Anyone who thinks otherwise is kidding themselves. In the early days of our project we made extensive use
of a tool called iContract, that allowed us to insert preconditions, postconditions and
class invariants as executable JavaDoc comments. (We give an example in
the book of the kind of really powerful contracts we were able to make using iContract.)
This tool really was valuable, but is no longer supported. In the best of all possible worlds,
we would still be using something like iContract.
In the absence of true Design-by-Contract support in Java, we agree that the assert keyword
is better than nothing.
Actually, since our book was published, we've started making a lot of use of the @NotNull
and @Nullable annotations provided by JetBrains.
These annotations show up in code-completion and parameter help
dialogs displayed by IntelliJ, which makes them even more visible than JavaDoc comments.
Maybe these annotations will develop further into a proper DBC tool. We live in hope.
"What I disagree with is the authors' vision of a unified way to access data, regardless of the underlying storage technology.
I have found that the design of the data access layer depends on many factors, such as the database itself and how it is
Our LabWizard software can be run using either a relational database (MS Access) or a BTree database (InfinityDB)
to persist the KnowledgeBases. All database code is confined to a single package that only has a few really big
classes in it. There is no reference to databases in any of the other hundred-odd packages comprising the software.
We think that this is an outstanding example of the underlying storage technology being hidden from the rest
of the application.
What is not mentioned
The criticisms we've responded to above are in fact pretty slight. Is the arrangement of test and source
code really worth getting excited about? No. Our main source of disappointment with the review
is that it did not mention that our book deals with:
- Application-level testing. This is the tracing of requirements to test specifications to actual
Java classes that prove that our application meets its requirements. While there are a lot of books on
unit-level testing, not much has been written about automated tests of an entire application.
When we write serious code for serious paying customers, we need traceability
and automated testing of the requirements that our customers are paying us to implement.
- Load testing. We've got a short but sweet chapter on this, relating heavily to how we
use load tests in our day-to-day development processes.
- Internationisation: We introduce a design pattern that allows unit-testing of properties-files.
These unit tests ensure that all problems with missing resources are found in the unit tests,
that mnemonics are correctly defined, and so on.
- The Help System: We provide a tool that guarantees that all JavaHelp files are indexed and available
and that they contain no missing images or broken links. Further, we show how to test that the correct
help file is displayed when context-sensitive help is launched.
- Testing email, spreadsheets, PDF, log files and web services.
- How to track down intermittent problems with tests.
- Lots of other things...
have a closer look!
Overall, we're grateful that Alex Ruiz has a lot of positive
things to say about our approach to the testing of Swing user interfaces.
However, we're disappointed that he spends so much of his review in what seem to
us to be petty criticisms (the layout of test code, not using JUnit, not adhering too slavishly
to a test-first approach etc.) and doesn't make mention of a lot of the other material in the book
that we think a lot of readers would be very interested in.