Colorado was the recipient of a grant from the Election Assistance Commission (EAC) to implement Risk-Limiting Audits. These notes from Neal McBurnett describe the goals and status of the project, along with related followup activities.
The Colorado Department of State plans to pilot risk-limiting audits in five counties in a mix of areas – urban, suburban, and rural. Colorado will develop, test, and implement a risk-limiting audit system that helps meet legislation requiring the state’s jurisdictions to implement a risk-limiting audit system by the 2014 elections [Note: deadline later moved to 2017 elections]. The research team will review current election processes in target counties and solicit input from other stakeholders, including county election officials, voters and concerned citizen groups. The team will also review existing best practices in the state and the nation, selecting the most promising practices for testing the appropriate target counties. Due to the wide array of election methods, vendors, and types of voting equipment used in the state’s 64 counties, the recommendations developed by the research team will be applicable in many of the jurisdictions across the nation.
Paper ballots are critical, but we also need to look at a sample of them by hand to check up on the equipment and procedures - good audits are critical also. This has been recognized by the Presidential Commission on Election Administration. They say:
Recommendation: Audits of voting equipment must be conducted after each election, as part of a comprehensive audit program, and data concerning machine performance must be publicly disclosed in a common data format. ....
The Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected.
We had first-hand experience with the benefits of audits, and the failures of certification. We've done innovative audits starting in 2004 in Boulder County and did the first risk-limiting audit outside of California. In contrast, the state lost a lawsuit due to inadequate certification and flawed security. So we adopted a flexible framework for voting system approval, and a state-wide requirement for risk-limiting audits by 2017 (Colorado Revised Statute 1-7-515).
The CORLA project in Colorado has now given us experience with Evidence-Based Elections as described by Stark and Wagner.
On November 14, 2013, we did a risk-limiting ballot-level audit of three contests for the general election in Arapahoe County. We used the website Tools for Comparison Risk-Limiting Election Audits to audit cast vote records generated by hardware and software from Clear Ballot Group.
The full report for the EAC is available at Risk-Limiting Audit – Final Report (pdf)
In August of 2014, after the CORLA project was over, Arapahoe County experimented with a follow-on audit, this time of the 2014 primary election, using the open source OpenCount software. It got good news coverage:
There are some misconceptions in the articles though. The ballot comparison audit is the most efficient way to determine if the outcome was correct, but it requires more detailed reports (cast vote records) than most tabulation systems now provide. The ballot polling audit, demonstrated in Colorado for the first time in this 2014 audit, is much less efficient for close contests, but can be used with any kind of tally equipment. In other words, both achieve the same goal (verifying the outcome), but differ in their efficiency and applicability.
Four major vendors, including ES&S, Clear Ballot, Dominion, and Hart, now provide Cast Vote Records that can be used (despite some unfortunate formatting problems) to do ballot-level Risk-Limiting Audits, so Colorado has really helped move the world of elections forward.
In 2015, Colorado selected a voting system based on Dominion's proposal.
For more information, see:
See also the presentation Philip Stark made on ballot-level auditing supported by ballot scans done with open-source software. He delivered it at the Colorado Best Practices and Vision Commission on Dec 14, 2011. You can get his slides from
and listen to the audio there also. The commission likes the approach, as do many other lawmakers and election administrators, including the secretary of state. There are several options for open source software to do state-of-the-art, easily auditable scanning and tabulation, including the OpenCount system for Operator-Assisted Tabulation of Optical Scan Ballots. OpenCount has been used to audit several California elections.
An excellent, comprehensive introduction to risk-limiting audits is Risk-Limiting Post-Election Audits: Why and How., by Bretschneider, J., S. Flaherty, S. Goodman, M. Halvorson, R. Johnston, M. Lindeman, R.L. Rivest, P. Smith, and P.B. Stark, 2012.