JHammer: Psuedorandom Unit Testing

I started a new open source project, yesterday, called JHammer.  From the JHammer website:

jhammer is a simple extension to JUnit that provides a framework for writing repeatable random unit tests. The framework aspires to help developers uncover difficult “show-stopping” bugs earlier in the development process through the use of automated psuedorandom unit tests.

The basic idea behind the framework is to provide an easy way for software testers to generate interesting stimuli for their unit tests, and do as much as possible to guarantee that those tests are repeatable.

The main advantage to pseudorandom testing is that the test writer writes one test and one set of checks/assertions to ensure correctness).  Every time the test is run it generates new (typically unique) random stimulus to exercises the unit under test (UUT).   The important part is that the psuedorandom test generation is able to create test cases that a human test writer would have a hard time designing.

For example, I had a simple open-addressed hash table (see Wikipedia for a good explanation) implementation written in C.  Like any decent programmer, I wrote  simple unit tests and they all passed.  Just for kicks I wrote a (quick and primitive) random test generator and started running test iterations.  The random tests immediately (within 100 iterations) found a bug in my code! Basically I had a copy-paste error that calculated the wrong array offsets when resizing the array).

So I fixed that bug (I probably would have stumbled upon it, eventually) and happily completed 500 or so random test iterations, multiple times.   I then “upped the anti” and started running hundreds of thousands of test iterations (and randomized the hash table size).  The random tests hit a non-trivial bug that required the following sequence of events to hit:

1. Element A added to hash table
2. Element B added to hash table ‘close’ to element A
3. Element B removed from hash table.
4. Hash table resized.
5. Element A added back into hash table.

What happened was that due to an errant bit flip in the implementation, the table now had two element ‘A’s. Because elements in the table must be unique, my program was no longer correct!  Here was where I realized the true power of psuedorandom testing – I would not have thought to test that particular sequence of events, yet that bug would have had catastrophic results if it washit in the field.

As sort of a pet project I decided to make a similar test framework for Java, and called it JHammer.  Writing unit tests is with JHammer is almost identical to writing JUnit tests , so there isn’t much of a learning curve.  Even so, there are a few things to keep in mind to ensure a repeatable test.  I wrote up a (rather lengthy) article at http://jhammer.tigris.org/wiki/GettingStartedWithJHammer that outlines how to use the tool.

There is a very preliminary preview build available in the Documents and Files section – click on previews and get the most recent preview build (Preview 6 as of this writing).  I would greatly appreciate more feedback on the project before I start adding more features.