A Software Testing Body of Knowledge?

This is an article from Stuart Reid which appeared in the 3rd Edition of the EuroSTAR Newsletter entitled - A Software Testing Body of Knowledge?

A Software Testing Body of Knowledge?

So, what is a Body of Knowledge or BOK?
A BOK describes the generally accepted knowledge for a particular discipline; it is a formal inventory of the intellectual content of the field. A BOK is thus one way of defining a profession. For a BOK to be accepted there should be widespread consensus within the community that the knowledge and practices within the BOK are both valuable and useful, and applicable to most projects most of the time. The BOK provides the basis for the regulation of the profession; it also defines its boundaries.


Example BOKS in the IT area cover disciplines such as Project Management (APM and PMI variants) and Configuration Management (CMBOK). There is also the IEEE Software Engineering BOK (SWEBOK), which includes a chapter on software testing. The SWEBOK is being advanced to ISO status, but has been dogged by disagreements and, so far, has not been widely accepted by the community.

Who uses a BOK?

Unsurprisingly, a BOK has various stakeholders. New entrants to a field can use it to identify what they need to know, while practitioners can use it as an essential source of information on topics that they only need to reference infrequently. Certification (and licensing) bodies and academics may use it in the form of a syllabus as the basis for qualifications, which, in turn, will mean that training providers and students are also users. Does a Software Testing BOK already exist? Although the authors may disagree, it seems clear that the discipline already includes a number of ‘pseudo’ BOKS. By this I mean that there are several well-used software testing resources, but not one that covers the complete discipline and there is also not one in which there is general consensus. Examples of these ‘pseudo’ BOKs are:

* qualification syllabi created by certification bodies such as ISEB/ISTQB;
* approaches to testing such as TMap®;
* test process improvement models such as TPI® and TMMi™;
* well-regarded text books such as Glen Myer’s original edition of The Arts of Software Testing;
* standards on software testing, such as IEEE 829 and BS 7925; and
* the software testing chapter of the SWEBOK.

Although providing various levels of coverage of the field of software testing, not one of these ‘pseudo’ BOKs on it own satisfies the criteria of becoming the single BOK for the industry. This is because none of them provides broad enough coverage of the discipline of software testing. Neither do any of them appear to command the respect and trust of a large enough proportion of the software testing community to be considered as representing a true consensus.

Is the discipline of software testing ready for a BOK?
Implicitly many contributors to the ‘pseudo’ BOKs appear to believe so; however there is also a strongly-held opposing point of view. Let’s consider the opponents’ view first. Some consider that a BOK acts as a barrier in a number of ways. They feel that BOKs are, by nature, inert and rarely evolve, restricting new thinking and debate on currently accepted ‘truths’. They also point to the continuous stream of project failures and the apparent lack of ‘engineering’ in software testing where scientific theories are not backed-up by solid empirical data. Both points are presented as evidence of the field’s immaturity.

Another argument presented against a software testing BOK is that the discipline is too diffused and changes from domain to domain. Detractors question whether there are enough generally good practices in software testing that apply to most projects and suggest that many good practices are only applicable to specific application domains. For instance, they say that the generally useful practices applied to testing safety-critical system may not be appropriate for the testing of low integrity commercial applications.

The supporters of a software testing BOK point to the benefit of certification in providing a means of regulating the industry and defining training for new entrants. They argue that certification also lends software testing credibility with both customers and developers, while the availability of a single consensus BOK would encourage academics (even those with little interest in, or knowledge of testing) to adopt it. Another suggested advantage of a BOK is that it provides guidance to practitioners on how to improve their current practices. Many of those who feel that software testing should be considered a legitimate engineering discipline see a BOK as a necessary stepping stone to a profession of software testing.

Should a software testing BOK be created?

If the industry decides that a BOK is needed for software testing then it is most important (and probably very difficult) to ensure that consensus is reached. Any initiative must be an inclusive, multi-national effort and care must be taken to ensure that the stakeholders in the previously-mentioned ‘pseudo’ BOKs are invited to join the development process. Ownership of a new BOK could be difficult to manage, and although it is often argued that anything provided for free may be considered worthless by the recipient, I believe that any newly-created software testing BOK should be made freely available to the whole community.

Developers of a BOK must ensure that it does not include practices that are new and unproven with no evidence of their efficacy. A BOK should embody achievable good practice and not simply be a reiteration of academic texts, which may have little connection with the real world. The speed of evolution of the software testing discipline means that its BOK must carry with it the requirement for its continual review and revision. Although a difficult task, I believe that simply by attempting to build a BOK the software testing industry will continue to expand its knowledge of the discipline and so add value to the testing community.

EuroSTAR 2006 Workshop

The topic of a software testing BOK will be covered by an advanced workshop at the EuroSTAR conference in December. The aim is to open up debate on whether the industry should support its creation (with all the attendant questions) or wait until we have more obviously reached maturity. If you feel you would like to contribute to the discussion then please make a note to attend in your diary.

Biography

Stuart Reid has spent the last 17 years involved in software testing, having previously worked on high-integrity systems. He is Chair of the BCS SIGiST and its Standards Working Party and was Chair of the ISEB Software Testing Board and founder of the ISTQB.