The Colorado Risk-Limiting Audit Project (CORLA)

Colorado is rolling out "Risk-Limiting Audits" (RLA) so that we can transparently provide citizens with scientific evidence that the outcomes of our elections correctly reflect the votes that were cast, or if necessary correct preliminary outcomes when they are wrong via a recount. RLAs are a method of vote tabulation auditing which should be widely adopted, along with audits of registration information, signature verification, ballot reconciliation and chain-of-custody, etc. All are part of the practice of Evidence-Based Elections as described by Stark and Wagner.

After over a decade of pioneering election audits in Colorado, in 2017 the Secretary of State's office convened a group to plan the rollout, building on a good summary of procedures recommended by auditing experts at Concise recommendations for Colorado Risk-Limiting Audit rule in 2017. The result was Rule 25. Post-election audit.

Information relating to the audit for Coordinated Election on November 7, 2017 is available in Colorado's Audit Center.

Open Source Risk-Limiting Audit Software

In a historic move, on May 5, 2017, the Colorado Department of State issued a solicitation (using their fast-track "Documented Quote" procedure) for audit support software for the upcoming 2017 elections and beyond: DQ_RLA_0500517_FE_KRT_VAAA_2017-1420.pdf. An official copy of it, along with two attachments, can be found by logging in (e.g. for "Public Access") and searching for "Election" at the ColoradoVSS site.

The contract was awarded in June, 2017 to Free & Fair, a company with extensive experience producing high assurance and open source software. See Free & Fair to build risk-limiting audit system for State of Colorado.

The open source software itself is available at ColoradoRLA: Software to facilitate risk-limiting audits at the state level, developed for the state of Colorado by Free & Fair, and can be customized for use in other states and counties.


Paper ballots are critical, but we also need to look at a sample of them by hand to check up on the equipment and procedures: good audits are critical also. This has been recognized by the Presidential Commission on Election Administration. They say:

Recommendation: Audits of voting equipment must be conducted after each election, as part of a comprehensive audit program, and data concerning machine performance must be publicly disclosed in a common data format. ....
The Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected.

Risk-limiting audits have also been endorsed by the League of Women Voters: Report of the Election Audits Task Force.

Already in 2009, Colorado had first-hand experience with the benefits of audits, and the failures of certification. We had done innovative audits starting in 2004 in Boulder County. We were pioneers in auditing at the ballot level. In 2005 Paul Walmsley wrote A Statistical Method for Auditing Vote-Counting, Whereby Counting a Relatively Small Number of Ballots Can Yield a High Degree of Statistical Confidence. In Eagle County, Harvie Branscomb led an audit that sampled individual VVPATs for audit, and has been a champion of election transparency for over a decade, identifying issues and solutions for many of the complications that arise from Colorado's high rate of drop-off and mail-in ballots, unsorted by precinct or style. See Preserving Anonymity of Cast Vote Records for a 2017 summary and recommendations.

The Conroy v. Dennis lawsuit in 2006 documented problems with relying on equipment certification for security. In 2008 we started doing robust audits in Boulder County.

In 2009 the state adopted a flexible framework for voting system approval, and a state-wide requirement for risk-limiting audits by 2014, later changed to 2017, via Colorado Revised Statute 1-7-515. In 2010 we conducted the first risk-limiting audit outside of California.

An excellent, comprehensive introduction to risk-limiting audits is Risk-Limiting Post-Election Audits: Why and How., by Bretschneider, J, S. Flaherty, S. Goodman, M. Halvorson, R. Johnston, M. Lindeman, R.L. Rivest, P. Smith, and P.B. Stark, 2012.

Grant from Election Assistance Commission (EAC)

In 2011, Colorado was the recipient of a grant from the Election Assistance Commission (EAC) to implement Risk-Limiting Audits.

Arapahoe County Risk-Limiting Audit, 2013

The CORLA project in Colorado provides experience with Evidence-Based Elections as described by Stark and Wagner.

On November 14, 2013, we did a risk-limiting ballot-level audit of three contests for the general election in Arapahoe County. We used the website Tools for Comparison Risk-Limiting Election Audits to audit cast vote records generated by hardware and software from Clear Ballot Group.

The full report for the EAC is available at Risk-Limiting Audit – Final Report (pdf)

Arapahoe County Audit with OpenCount software, August 2014

In August of 2014, after the CORLA project was over, Arapahoe County experimented with a follow-on audit, this time of the 2014 primary election, using the open source OpenCount software. It got good news coverage:

There are some misconceptions in the articles though. A ballot-level comparison audit is the most efficient way to determine if the outcome was correct, but it requires more detailed reports (cast vote records which can be matched with individual paper ballots) than most tabulation systems in current use provide. The ballot polling audit, demonstrated in Colorado for the first time in this 2014 audit, is much less efficient for close contests, and has a large variance in the number of ballots that need to be audited, but it can be used with any kind of tally equipment. In other words, both achieve the same goal (verifying the outcome), but differ in their efficiency and applicability.

2015 ballot-level Mock Audits

Given the legislative mandate, and based on the positive experience with CORLA, as part of Colorado's Uniform Voting System project, a series of "Mock Risk-Limiting Audits" were performed in 2015 in several counties using equipment from several different manufacturers. They included innovative demonstrations during the 2015 general election in:

Uniform Voting System

The most recent systems from four major vendors, including ES&S, Clear Ballot, Dominion, and Hart, now provide Cast Vote Records that can be used (despite some unfortunate formatting issues) to do ballot-level Risk-Limiting Audits, so Colorado has really helped move the world of elections forward.

In 2015, Colorado selected a voting system based on Dominion's proposal.

For more information, see:

2017 Rulemaking and Rollout

In 2017 the Secretary of State's office convened a working group including auditing experts from the Election Verification Network (EVN), experienced election administrators from five counties, and representatives from voting system vendors, establishing the Risk-Limiting Audit Representative Group. The group helped the SoS establish official rules for how RLAs will be conducted, as required starting with the 2017 elections.

The kickoff presentation provides an official introduction. A good summary of procedures recommended by auditing experts is at Concise recommendations for Colorado Risk-Limiting Audit rule in 2017, and there is a wealth of other information at the Risk-Limiting Audit Representative Group site.

The official audit rules were developed via rulemaking process with extensive input documented at Election Rulemaking Hearing 7/11/2017.

Auditing Multi-County Contests

So far the audits in Colorado will only limit the risk for contests at the county level, or entirely contained in a single county. The math and algorithms for efficiently coordinating coordinating audits of a single contest between different counties is still a topic of research. See for example Bayesian audit support program:

In 2018, Colorado will have a number of state-wide contests, and many big contests shared between counties, and this research will be distilled into practical procedures and code to support that.

Audits in Other States

In September of 2017, Rhode Island became the second state to require risk-limiting audits, for implementation by 2020, with possible pilots in 2018.

Dozens of other states require post-election tabulation audits, including pioneering efforts in California, and an innovative coordinated, margin-sensitive state-wide audit ("Voting System Check") of several contests in New Mexico, conducted by an independent auditor. See e.g. 2016 General Election Voting System Check.

Using Images of Ballots, for Quality Control, not Auditing

Most modern optical scan voting equipment scans the paper ballots, producing a digital image. It then interprets the image to produce a Cast Vote Record of how it thinks the ballot was marked. These digital images are often of low quality (200 DPI, one bit per pixel) and sometimes the scanners are configured to filter out certain colors. They can be useful for quality control of the scanning process, for example when ballots are marked in nonstandard or confusing ways. But it is sometimes hard to determine voter intent from the images.

The computerized image data can lead to differing interpretations than humans would see looking at the ballots, due to error, or due to intentional alteration (hacking) of the scanners and associated computers. Thus, while software, e.g. from OpenCount or Clear Ballot can be used to analyze ballot images to help weed out the poorly-marked ballots from the rest, and to suggest which paper ballots which are good candidates for targeted auditing, only manual interpretation of voter-verifiable paper ballots can be used to guard against hacking or cyber attacks on election systems, or to reliably detect all machine misinterpretations of voter intent.

This is discussed in more detail at Some Observations re: Maryland Election Procedures, 2016 and Risk-limiting audit procedures must examine the paper ballots - input for Maryland Election Procedures, 2016.

See also Machine Retabulation is not Auditing, and Retabulations, Machine-Assisted Audits, and Election Verification, both by Mark Lindeman, Ronald L. Rivest, and Philip B. Stark, March 2013.

Related sites

Related information

Site run by Neal McBurnett