As a follow-up to a previous post which cited a Perspective in Science, I thought that I would mention some recent developments in Software Testing. Science published a letter which suggested that pre-publication peer review of code would put too much strain on the review system. I am sure that anyone who acts as a referee for journals would agree that it can consume a significant amount of time; working through someone else’s code would certainly add to the load. Moreover, I doubt that it would be practical to try and read the entire source code for large atomistic codes such as VASP, CASTEP, abinit, PWSCF, NAMD, AMBER or CHARMM. The authors of the perspective responded, acknowledging that it would involve more work, and suggesting that availability of the code to the reviewer, and external code review could be considered. (See also this other article from Science on why computer codes should not be treated as black boxes.)

The Mozilla Foundation, responsible for the Firefox browser among other things, have recently announced that they will investigate the feasibility of debugging scientific code (see this article in Nature and the Mozilla ScienceLab). There have been different reactions to this idea, with the most negative being that scientists will be discouraged from posting or submitting their code if they think that it will be scrutinised.

I think that code review is an excellent idea, and that scientific coders need to become more literate in computer science. The Software Carpentry organization is an excellent example of how to go about this, as is the Software Sustainability Institute. There is a growing trend in the UK to acknowledge the importance of support for scientific computing, and as computational scientists, we need to support this. If our codes are not open source, we should certainly publish details of the tests we’ve done to ensure their reliability, as well as details of the development approach and version control used. This will move us towards confidence in the results and the science generated with the codes.

Update There’s a nice piece giving a basic introduction to unit testing here. Code testing is becoming something of a hot topic, as it should be.