US Election Assistance Commission - Voluntary Voting System Guidelines Vote
EAC Home
Introduction
View Guidelines
View Comments
Glossary

View Comments

Section CommentsGeneral CommentsGlossary Comments
 
Name :   Cem Kaner
Organization :   N/A
Post Date :   9/30/2005

General Comments
Comment :  In modern programming, the 100-trillion-test example would be a single subroutine (a function or a
method) of a larger program. A voting system might have thousands of subroutines of this complexity.
Thus, assessments of any testing strategy must be considerations of the adequacy of the testing effort
and not of the completeness. The testing effort is guaranteed to be incomplete, but it might be adequate.
Independent testing under government regulations or the most widely used industry standards involves
creation of extensive documentation and the maintenance of extensive records that allow a manager,
client or auditor to determine what testing has been done, what was tested, when the testing was done,
and what the results were. Such records can be of great value, for many purposes. I am not suggesting
that the Guidelines relax or revise such requirements.
However, recordkeeping comes at a price. If you start with an infinite task, you cannot have enough
time or money to complete it. Given a finite budget, you can only do so much. The more you spend on
formal planning (and on written description of your plans), on high level and detailed design (and
written description of the designs and the many drafts of the designs), on the written description of the
individual tests and the tracking of the executions of these tests, and on the many types of reports of
testing progress and results, the less you have left for actually running tests and evaluating their results.
If each test has a high cost for planning, design, description and tracking, there is a significant impact
on the total number of tests that can be run within the budget.