art with code

2009-01-04

Current testing stats

1500 lines of new tests written, 20 bugs found, codebase sans tests is 1700 lines (2200 with the aliases.) The first 800 lines of tests uncovered 6 bugs, the next 700 found 14.[1] Apparently practice does make you better at writing tests.

I'm still around a thousand test lines short of done, so at the current rate of one bug per 50 lines of tests I might expect to find 20 more bugs. Which'd put prelude.ml's pre-testing bugs/kSLOC at 24 bugs per thousand lines of source code (bug every 42 lines.) Prelude.ml contains around 700 top-level function definitions, giving a ratio of one bug every 17.5 functions.

Furthermore, I might expect the current set of tests to contain 36 bugs (1500 * 24/1000) and the finished tests to contain 60 bugs. A bug in a test would be omitting some relevant test case or expecting wrong behaviour.

A few questions arise: How to find more defects with less effort? How to assert the correctness of the tests? How to write code with fewer defects?


[1] My counting methodology was: Misbehaviour is a bug. Changing code and tests to change expected behaviour counts as a bug. Solving multiple bugs with a single search&replace counts as one bug. Changing performance characteristics of a function (e.g. making a function tail recursive) counts as a bug.

2 comments:

Anonymous said...

Mutation testing can be useful.

Jester for Java: http://jester.sourceforge.net/
Heckle for Ruby: http://ruby.sadi.st/Heckle.html

Ilmari Heikkinen said...

Thank you for the links, mutation testing sounds like a good idea. I couldn't find an OCaml version but it should be doable to write one with CamlP4.

A fun (if cpu-burning) idea might be to use a code mutator to drive a test generator to search for a minimal specification of a given function. Come think of it, the mutator is a test generator targeting the tests.

Blog Archive