Project 40: Testing methodology

Pasadena 1984:
==============

a) DETEST for ode (Hull)

Complete rewrite and careful documentation of DETEST (for testing 
both stiff and non-stiff ODE solvers) was completed at Toronto 
in October 1983, by W.H. Enright and J.D. Pryce. Technical report 
and/or tape available from University of Toronto or Bristol.

Hull had planned to publish note on this in SIGNUM; and then 
write to other test centres to encourage analogous contributions, 
but haven't done so as yet.

(Experience so for has been that few others have used DETEST, but 
mostly the interest has been in our test problems).

b) Testing optimization software (Wright)

Wright reports briefly on an effort by the Algorithms Group, 
Systems Optimization Laboratory, Stanford to complete two tasks 

(i) development of a large set of useful constrained optimization 
test problems in order to evaluate reliability and efficiency
of optimization software; and 

(ii) definition of a set of quantifiable characteristics that affect
problem difficulty, in order to analyze and classify existing problems
and to generate new problems with specified features.

Vouk feels a need for defining some numbers attached to the software 
which reflect its reliability. Cody asks what to test and why: 
the purpose of a test is to see if a program performs what it is 
designed to do and how well it performs it. Then arises the question 
of the presentation of results (Chernoff face).

Ford distinguishes between algorithmic and software testing, then 
on the 2nd version of the program, between implementation and 
installation testing.

Felman, Vouk, Reid, Hanson and Rice discuss the difficulty to 
create a battery of test cases which represent the real world. 
Huddleston reports on a testing methodology used on large differential 
equations codes. Reid suggests that a paper should be written by
Hull, Wright, Cody, Vouk, with the assistance of a subgroup consisting 
of Aird, Chatelin, Ford, Gentleman, Lawson, Reid, Rice.

Como 1987:
==========

Wright: As the first step toward producing a concrete result after
several years of preliminary discussion, we intend to compile an
annotated bibliography on testing methodology. The bibliography is
likely to be rather diverse, since the subgroup members have expertise
in different areas of numerical analysis. Once the bibliography is
complete, the next phase of the project is to outline and write a
technical paper. Ideally, the paper should analyze only themes common
to all areas of testing methodology, with a particular emphasis on
testing of numerical software, and also identify and explain (if
possible) differences arising from particular needs of certain areas.

During the discussion it was pointed out that in the PC and
microcomputer world compilers and operating systems are a problem, and
checking of these system "parts" is also needed (Aird, Gentleman).
Compiler validation suites and, in general, environment testing and
updating are a problem. The Ada validation suite, for example,
currently contains very little numerical testing, although
improvements are in sight. (Gentleman, Ford, Feldman, Rice, Vouk,
Dekker, Smith, Delves)

Stanford 1988:
==============

Document:	IFIP/WG 2.5 (Stanford-20) 1520, 1 page.

M. Wright reported on the activities of this subgroup. Vouk has
compiled a preliminary set of references on testing methodologies in
the field of numerical software. Additional work needs to be done in
collecting information of publications dealing with test suites for
numerical software.

Jerusalem 1990:
===============

Vouk suggested that the project be closed. This was unanimously
accepted.