Project 72: Testing Numerical Software Purdue 1999: ============ Gaffney presented a proposal for refereeing and publishing numerical software. He advocated that real users do the testing and evaluation of submitted packages and the refereeing would be monitored and reported by newsgroups. The model would be to `publish' and then `referee'. There would have to be a mechanism where a. newsgroup would collect referees' reports (over time) and eventually accept/endorse the package. Testing Hamiltonian Solvers --------------------------- Gladwell discussed some of the special difficulties that arise when one attempts to evaluate and compare numerical methods designed for long time integrations of special second order initial value problems. Ottawa 2000: ============ Although there was no report on this project there was some discussion of this project. Boisvert indicated that ACM TOMS would be willing to listen and cooperate in assessing its Algorithms. Others expressed some concerns with the `publish then referee' plan and noted that some new metric for assessing `quality' may be required. Amsterdam 2001: =============== Although there was no report on this project there was a lively discussion on the need and/or support for results in this area. Ford pointed out that there is a renewed interest arising from recent court cases and there is now likely to be more interest in certification of numerical software. Pool pointed out the need for more software testing in large scale projects such as the ASCI project. Gaffney promised to prepare a report for our next meeting. Portland 2002: ============== Although there was no report on this project there was a lively discussion of the related legal difficulties that are arising. Testing and reporting comparisons can result in litigation. For example MS licences forbid the publishing of `benchmarking' results. Nevertheless it is important to continue testing and the investigation of where existing software is likely to be unreliable. Strobl 2003: ============ Ford pointed out the new incentives for measurement and analysis of performance that has resulted from recent litigation which recognized inadequate testing of software. This is an active area. Washington 2004: ================ This is an active area and Ford noted the need for testing new versions of software and programs. He noted, in particular, the work in the area of meteorology where one had to deal with uncertainty in the measurements. Prescott 2006: ============== This is an active area with several members of the group engaged in related research. The talks by Gaffney on "Virtual Manufacturing - The Vision for Virtual Paint Operations" and by Smith on "A Test Harness TH for Numerical Applications and Libraries" at the working conference and the one by Enright on "An investigation of the Reliability/Efficiency Trade-off in ODE Solvers" at the working group meeting are examples of reports of activity in this area. Uppsala 2007: ============= This is an active area with several members of the group engaged in related research. Several members requested that their names be added to the list of participants in this project. Some discussion continued on the suitability of this topic to be the theme of the next Working Conference. Toronto 2008: ============= This is an active area with several members of the group engaged in related research. Enright's workshop talk "Developing easy to use ODE Software" is a report on one component of this activity. It was pointed out that this will likely be an important component of WoCo 10. Raleigh 2009: ============= Wilfried Gansterer's presentation "Case Studies in Trading Accuracy for Performance in Numerical Software" and Brian Smith's "3D Magnetic Field Modeling Using The Boundary Element Method" at the "IFIP Working Group 2.5 Symposium 2009" are reports on this project. This is an active area with several members of the group engaged in related research. Enright agreed to provide an updated paragraph. Leuven 2010: =========== The talk by Tim Hopkins "The collected algorithms of the ACM - past, present and future" at ICCAM 2010 is a contribution on this topic. Pat Gaffney and others will talk on this subject at WoCo10 in Boulder, 2011. Boulder 2011: ============= This is an active area with several members of the group engaged in related research. The talks by Smith and Enright at the associated WoCo can be considered reports of different aspects of this project. Santander 2012: =============== The papers by Dan Lozier "DLMF Tables: A New Source of Data for Mathematical Software Developers" and Annie Cuyt "Validated Evaluation of Special Mathematical Functions" at the workshop can be considered reports of activities related to this project. Shanghai 2013: ============== Report by Jose Javier Current activity involves, among other aspects, the numerical testing of software for evaluating mathematical functions. The forthcoming proceedings of the Santander conference (eds. A. Gil, J.M. Muller and J. Segura) will contain several papers related to the subject, including the accepted paper "Validated Evaluation of Special Mathematical Functions" by F. Backeljauw, S. Becuwe, A. Cuyt, J. Van Deun and D. W. Lozier. The talk "Numerical methods for special functions" presented by N. M. Temme at the SIAM Annual meeting 2013 included discussions on the testing of numerical software for special functions, particularly for some probability distribution functions (non-central chi-squared distributions). This study revealed some erroneous computations in commercial software; this discussion is also contained in a paper by A. Gil, J. Segura and N.M. Temme (submitted to ACM TOMS). At the same SIAM minisymposium, Dan Lozier (NIST) presented the latest developments in the design of an on-demand data service for special functions. The presentation by Ronald Boisvert "A Survey of Data Intensive Problems at NIST" is available as http://conf.shu.edu.cn/disd2013/ppt/8-2/1-Boisvert.pdf Vienna 2014: ============ The presentations of Gil ("Algorithms and Software for Computing Associated Legendre Functions") and Segura ("Algorithms for the computation and inversion of cumulative distribution functions") at the Workshop are examples of recent activities of this project. Segura will provide an updated report. Halifax 2015: ============= Report by Segura and Boisvert: The project is active with several members of the group involved in related research. Some of the activities involve the numerical testing of software for function evaluation. As an example of this activity, we can mention the presentation "Computation and Inversion of Certain Cumulative Distribution Functions" delivered by Amparo Gil at the 13th International Symposium on Orthogonal Polynomials, Special Functions and their applications (June 1-5, 2015, NIST, Gaithersburg, Maryland). Comparison of recent algorithms against commercial software and the R programming language were reported, in some cases showing inaccurate results. The NIST Digital Library of Mathematical Functions (DLMF) project, in association with the University of Antwerp (Annie Cuyt's group) have released a prototype web service to provide tables of special functions on demand with guaranteed error bounds. See http://dlmftables.uantwerpen.be/. A small collection of real-valued functions from each of the following function classes are included in the prototype: elementary, error functions, Dawson's and Fresnel integrals, incomplete gamma, Bessel, confluent hypergeometric, hypergeometric, and generalized hypergeometric. Additional functions will be added over time. 2020 (online meeting): ====================== Report by Segura: The project is active. Current activities include, among others, the numerical testing of software for special functions. Software for the numerical computation and inversion of cumulative distributions functions has been considered and tested against free and commercial software (R and Matlab), in particular software for the binomial and negative binomial distributions (see, A. Gil, J. Segura, N.M. Temme, ETNA 52 (2020) 270-280) and the non-central beta distribution (AG, JS, N. M. Temme, contribution to MCQMC2020, Oxford, August 2020). In addition, software for the fast numerical computation of Gaussian quadratures (in particular Gauss-Jacobi quadratures) has been recently developed and tested against the CHEBFUN algorithms (see AG, JS, N.M. Temme, Fast and reliable high accuracy computation of Gauss-Jacobi quadrature, submitted.) 2022 (online meeting): ====================== The project is active with several members of the group involved in related research. Current activities include, among others, the numerical testing of software for special functions. In particular, software for the numerical computation of the relativistic Fermi-Dirac integral and its derivative (functions widely used in stellar astrophysics) has been recently published (A. Gil, A. Odrzywołek, J. Segura, N. M.Temme, Comput. Phys. Commun., 2022). The software, implemented in Matlab, has been tested against numerical quadrature in extended precision platforms. In addition, software for the numerical computation of the central beta distribution has been developed and tested against Matlab (V. Erogova, A. Gil, J. Segura, N. M. Temme, submitted).