The Colorado Risk-Limiting Audit Project (CORLA)

Colorado is rolling out Risk-Limiting Audits (RLA) so that we can transparently provide citizens with scientific evidence that the outcomes of our elections correctly reflect the votes that were cast.

After over a decade of gathering experience with election audits in Colorado, in 2017 the Secretary of State's office convened a working group including auditing experts from the Election Verification Network (EVN), experienced election administrators from five counties, and representatives from voting system vendors, establishing the Risk-Limiting Audit Representative Group. The group is helping the SoS establish official rules for how RLAs will be conducted, as required starting with the 2017 elections.

The kickoff presentation provides an official introduction. A good summary of procedures recommended by auditing experts is at Concise recommendations for Colorado Risk-Limiting Audit rule in 2017, but there is a wealth of other information at the Risk-Limiting Audit Representative Group site.

Solicitation for Open Source Risk-Limiting Audit Software

In a historic move, on May 5, 2017, the Colorado Department of State issued a solicitaion (using their fast-track "Documented Quote" procedure) for audit support software for the upcoming 2017 elections and beyond: DQ_RLA_0500517_FE_KRT_VAAA_2017-1420.pdf. An official copy of it, along with two attachments, can be found by logging in (e.g. for "Public Access") and searching for "Election" at the ColoradoVSS site.

They solicited bids which develop open source software, but regardless, they intend to disclose the source code in the interest of transparency.

The contract was awarded in June, 2017 to Free & Fair: Free & Fair to build risk-limiting audit system for State of Colorado – Free & Fair

The next major step is writing the official audit rules. A draft was published in June, and a rulemaking meeting happend July 11: Election Rulemaking Hearing 7/11/2017.

We hope the state continues to audit all the contests, as they have since 2005, rather than settling for the minimum of one statewide and one in each county which is listed in the DQ.

Background

Paper ballots are critical, but we also need to look at a sample of them by hand to check up on the equipment and procedures: good audits are critical also. This has been recognized by the Presidential Commission on Election Administration. They say:

Recommendation: Audits of voting equipment must be conducted after each election, as part of a comprehensive audit program, and data concerning machine performance must be publicly disclosed in a common data format. ....
The Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected.

Colorado had first-hand experience with the benefits of audits, and the failures of certification. We've done innovative audits starting in 2004 in Boulder County. The Conroy v. Dennis lawsuit in 2006 documented problems with relying on equipment certification for security. In 2008 we started doing robust audits in Boulder County. 2009 the state adopted a flexible framework for voting system approval, and a state-wide requirement for risk-limiting audits by 2017 via Colorado Revised Statute 1-7-515. In 2010 we the first risk-limiting audit outside of California.

An excellent, comprehensive introduction to risk-limiting audits is Risk-Limiting Post-Election Audits: Why and How., by Bretschneider, J., S. Flaherty, S. Goodman, M. Halvorson, R. Johnston, M. Lindeman, R.L. Rivest, P. Smith, and P.B. Stark, 2012.

Grant from Election Assistance Commission (EAC)

In 2011, Colorado was the recipient of a grant from the Election Assistance Commission (EAC) to implement Risk-Limiting Audits.

Arapahoe County Risk-Limiting Audit, 2013

The CORLA project in Colorado has now given us experience with Evidence-Based Elections as described by Stark and Wagner.

On November 14, 2013, we did a risk-limiting ballot-level audit of three contests for the general election in Arapahoe County. We used the website Tools for Comparison Risk-Limiting Election Audits to audit cast vote records generated by hardware and software from Clear Ballot Group.

The full report for the EAC is available at Risk-Limiting Audit – Final Report (pdf)

Arapahoe County Audit with OpenCount software, August 2014

In August of 2014, after the CORLA project was over, Arapahoe County experimented with a follow-on audit, this time of the 2014 primary election, using the open source OpenCount software. It got good news coverage:

There are some misconceptions in the articles though. The ballot comparison audit is the most efficient way to determine if the outcome was correct, but it requires more detailed reports (cast vote records) than most tabulation systems now provide. The ballot polling audit, demonstrated in Colorado for the first time in this 2014 audit, is much less efficient for close contests, but can be used with any kind of tally equipment. In other words, both achieve the same goal (verifying the outcome), but differ in their efficiency and applicability.

2015 ballot-level Mock Audits

Given the legislative mandate, and based on the positive experience with CORLA, as part of Colorado's Uniform Voting System project, a series of "Mock Risk-Limiting Audits" were performed in several counties using equipment from several different manufacturers. They included innovative demonstrations during the 2015 general election in:

Uniform Voting System

Four major vendors, including ES&S, Clear Ballot, Dominion, and Hart, now provide Cast Vote Records that can be used (despite some unfortunate formatting problems) to do ballot-level Risk-Limiting Audits, so Colorado has really helped move the world of elections forward.

In 2015, Colorado selected a voting system based on Dominion's proposal.

For more information, see:

Related information

Site run by Neal McBurnett