In November of 2017, Colorado rolled out regular "Risk-Limiting Audits" (RLA). RLAs are designed to provide voters with scientific evidence that the outcomes of our elections correctly reflect the ballots that were cast (or if necessary, correct preliminary outcomes when they are wrong, via a recount). RLAs are a method of vote tabulation auditing which should be widely adopted, along with audits of registration information, signature verification, ballot reconciliation, chain-of-custody, etc. All are part of the practice of Evidence-Based Elections as described by Stark and Wagner.
How did we make that happen? This web site assembles the most complete set of materials for understanding what we've done, and what work remains.
This 3-minute video gives a visual overview of the process that Denver used in 2018.
Here is a more comprehensive 30-minute video of the 2017 Colorado audit.
After over a decade of pioneering election audits in Colorado (see below), in 2017 the Secretary of State's office convened a group to plan the rollout, building on a good summary of procedures recommended by auditing experts at Concise recommendations for Colorado Risk-Limiting Audit rule in 2017. The result was Rule 25. Post-election audit.
The rule was implemented for the first time in November 2017. See the blog post by Dwight Shellman and Jennifer Morrell at Colorado’s Implementation of Risk Limiting Audits | US Election Assistance Commission for the perspectives of two election adminstrators who have been instrumental in making it happen.
Information relating to the Colorado audits since 2017 is available in Colorado's Audit Center.
In a historic move, on May 5, 2017, the Colorado Department of State issued a solicitation (using their fast-track "Documented Quote" procedure) for audit support software for the upcoming 2017 elections and beyond: DQ_RLA_0500517_FE_KRT_VAAA_2017-1420.pdf. An official copy of it, along with two attachments, can be found by logging in (e.g. for "Public Access") and searching for "Election" at the ColoradoVSS site.
The contract was awarded in June, 2017 to Free & Fair, a company with extensive experience producing high assurance and open source software. See Free & Fair to build risk-limiting audit system for State of Colorado.
The open source software itself is available at cdos-rla/colorado-rla: Colorado Risk-Limiting Audit (RLA) software, originally developed for the state of Colorado by Free & Fair and DemocracyWorks as ColoradoRLA, and can be customized for use in other states and counties. It is also known as RLATool.
Since we also don't want anyone to have to trust the audit support software itself, or the way it is run, we designed the software to support a Public RLA Oversight Protocol. It exports all the necessary information and facilitates all the necessary public observations to provide full confidence in the audit.
In 2018 there was an RLA of the primary election (2018-06-26).
There was an RLA of the November general election. Input was taken to update the rules and further develop the software to handle coordination of state-wide and other multi-county contests via RFI 2018000035, Risk Limiting Audit - Colorado Elections - CDOS, 2017-12-29. Search for "*AUDIT*ELECTION*" at the ColoradoVSS site. Further materials were posted on the Risk-Limiting Audit Representative Group page in 2018, including a transcript of an explanation of how Colorado's RLAs work.
Democracy Works bid and won the year-long contract to update the open-source ColoradoRLA software to implement multi-county audits, multiple audit boards, and other improvements.
In 2023, Neal McBurnett made a presentation to the Bipartisan Election Advisory Commission of the Colorado Secretary of State recommending improvements to the auditing practices: Colorado's Risk-Limiting Tabulation Audits: Accomplishments and Proposed Improvements
Starting in 2019, VotingWorks has taken on the mantle of producing open-source election auditing software: Arlo. It builds on and extends the lessons from Colorado and its ColoradoRLA software, so they apply to the varying challenges faced in other states, supporting ballot-polling audits, batch-comparison audits and ballot-comparison audits.
Paper ballots are critical, but we also need to look at a sample of them by hand to check up on the equipment and procedures: good audits are critical also. This short video from Alex Halderman for the New York Times explains this well, including a hacked demo election.
This has also been recognized by the Presidential Commission on Election Administration. They say:
Recommendation: Audits of voting equipment must be conducted after each election, as part of a comprehensive audit program, and data concerning machine performance must be publicly disclosed in a common data format. ....
The Commission endorses both risk-limiting audits that ensure the correct winner has been determined according to a sample of votes cast, and performance audits that evaluate whether the voting technology performs as promised and expected.
Risk-limiting audits have also been endorsed by the National Academies of Sciences, Engineering, and Medicine and the League of Women Voters: Report of the Election Audits Task Force.
Many of the innovations in election auditing trace back to the seminal 2007 Post Election Audit Summit, 25-27 October 2007 in Minneapolis, Minnesota, organized by CEIMN: Citizens for Election Integrity Minnesota.
In Colorado, we had first-hand experience with the benefits of audits, and the failures of certification.
In 2003, the group Citizens for Verifiable Voting (CVV) in Boulder successfully headed off the purchase of a paperless DRE system. CVV gathered support from all four of Boulder County's political parties (Democratic, Republican, Green and Libertarian) for paper ballots, and made a presentation to the Boulder County Commissioners that also called for Random Sampling of Ballots Hand Counted to Verify Accuracy + Full-Disclosure of Any Voting System.
We started doing innovative audits in 2004, which identified an incorrect vote count. We described early sampling procedures and verifiably random selection processes with links to some of the literature on auditing and related state laws. We were pioneers in auditing at the ballot level. In 2005 Paul Walmsley wrote A Statistical Method for Auditing Vote-Counting, Whereby Counting a Relatively Small Number of Ballots Can Yield a High Degree of Statistical Confidence. In Eagle County in 2006, Harvie Branscomb led an audit that sampled individual VVPATs for audit, and has been a champion of election transparency for over a decade, identifying issues and solutions for many of the complications that arise from Colorado's high rate of drop-off and mail-in ballots, unsorted by precinct or style. See Preserving Anonymity of Cast Vote Records for a 2017 summary and recommendations.
The Conroy v. Dennis lawsuit in 2006 documented problems with relying on equipment certification for security. In 2008 we started doing robust audits in Boulder County.
In 2009 the state adopted a flexible framework for voting system approval, and a state-wide requirement for risk-limiting audits by 2014, later changed to 2017, via Colorado Revised Statute 1-7-515. In 2010 we conducted the first risk-limiting audit outside of California.
An excellent, up-to-date policy-oriented overview is An Introduction to Risk-Limiting Audits and Evidence-Based Elections, Prepared for California's Little Hoover Commission, 2018. A more comprehensive and technical treatment is Risk-Limiting Post-Election Audits: Why and How., by Bretschneider, J, S. Flaherty, S. Goodman, M. Halvorson, R. Johnston, M. Lindeman, R.L. Rivest, P. Smith, and P.B. Stark, 2012.
In 2011, Colorado was the recipient of a grant from the Election Assistance Commission (EAC) to implement Risk-Limiting Audits.
The CORLA project in Colorado provides experience with Evidence-Based Elections as described by Stark and Wagner.
On November 14, 2013, we did a risk-limiting ballot-level audit of three contests for the general election in Arapahoe County. We used the website Tools for Comparison Risk-Limiting Election Audits to audit cast vote records generated by hardware and software from Clear Ballot Group.
The full report for the EAC is available at Risk-Limiting Audit – Final Report (pdf)
In August of 2014, after the CORLA project was over, Arapahoe County experimented with a follow-on audit, this time of the 2014 primary election, using the open source OpenCount software. It got good news coverage:
There are some misconceptions in the articles though. A ballot-level comparison audit is the most efficient way to determine if the outcome was correct, but it requires more detailed reports (cast vote records which can be matched with individual paper ballots) than most tabulation systems in current use provide. The ballot polling audit, demonstrated in Colorado for the first time in this 2014 audit, is much less efficient for close contests, and has a large variance in the number of ballots that need to be audited, but it can be used with any kind of tally equipment. In other words, both achieve the same goal (verifying the outcome), but differ in their efficiency and applicability.
Given the legislative mandate, and based on the positive experience with CORLA, as part of Colorado's Uniform Voting System project, a series of "Mock Risk-Limiting Audits" were performed in 2015 in several counties using equipment from several different manufacturers. They included innovative demonstrations during the 2015 general election in:
The most recent systems from four major vendors, including ES&S, Clear Ballot, Dominion, and Hart, now provide Cast Vote Records that can be used (despite some unfortunate formatting issues) to do ballot-level Risk-Limiting Audits, so Colorado has really helped move the world of elections forward.
In 2015, Colorado selected a voting system based on Dominion's proposal.
For more information, see:
In 2017 the Secretary of State's office convened a working group including auditing experts from the Election Verification Network (EVN), experienced election administrators from five counties, and representatives from voting system vendors, establishing the Risk-Limiting Audit Representative Group. The group helped the SoS establish official rules for how to conduct RLAs, as required starting with the 2017 elections.
The kickoff presentation provides an official introduction. A good summary of procedures recommended by auditing experts is at Concise recommendations for Colorado Risk-Limiting Audit rule in 2017, and there is a wealth of other information at the Risk-Limiting Audit Representative Group site.
The official audit rules were developed via rulemaking process with extensive input documented at Election Rulemaking Hearing 7/11/2017.
So far the audits in Colorado only limit the risk for contests at the county level, or entirely contained in a single county. The math and algorithms for efficiently coordinating coordinating audits of a single contest between different counties is still a topic of research. See for example Next Steps for the Colorado Risk-Limiting Audit (CORLA) Program and bctool and bptool: Python code to support Bayesian audits.
In 2018, Colorado will have a number of state-wide contests, and many big contests shared between counties, and this research will be distilled into practical procedures and code to support that.
California pioneered risk-limiting audits starting in 2008, thanks to professor Philip Stark at Berkeley.
In September of 2017, Rhode Island became the second state to require risk-limiting audits, for implementation by 2020 (RI Gen L § 17-19-37.4 (2017)). They did extensive pilots in 2019. Nevada also now requires RLAs, and California and Washington states also passed legislation in 2018 to enable RLAs.
Dozens of other states require post-election tabulation audits, including an innovative coordinated, margin-sensitive state-wide audit ("Voting System Check") of several contests in New Mexico, conducted by an independent auditor. See e.g. 2016 General Election Voting System Check.
A growing list of counties, states, countries and towns have done official or pilot RLAs. This list is intended to be comprehensive in listing the first RLAs performed in each jurisdiction, so please send in corrections and updates.
Note that RLAs audit the tabulation of an election, and rely on the paper ballots having been verified by the voters. Note also that in most settings to date, it seems that when voters use Ballot Marking Devices (BMDs), they don't actually do a good job of ensuring that the paper ballot matches their preferences, and attacks on the BMD are possible, and hard to detect. So voters should be be encouraged to use a hand-marked paper ballot, or effectively encouraged to verify the actual paper ballot-of-record.
Most modern optical scan voting equipment scans the paper ballots, producing a digital image. It then interprets the image to produce a Cast Vote Record of how it thinks the ballot was marked. These digital images are often of low quality (200 DPI, one bit per pixel) and sometimes the scanners are configured to filter out certain colors. They can be useful for helping with compliance audits of the chain-of-custody of the paper ballots, quality control of the scanning process, for example when ballots are marked in nonstandard or confusing ways. But it is sometimes hard to determine voter intent from the images.
The computerized image data can lead to differing interpretations than humans would see looking at the ballots, due to error, or due to intentional alteration (hacking) of the scanners and associated computers.
For an example of how images can be hacked in the scanner before even being stored, see UnclearBallot: Automated Ballot Image Manipulation by Matthew Bernhard, Kartikeya Kandula, Jeremy Wink, and J. Alex Halderman.
Thus, while software, e.g. from OpenCount or Clear Ballot or AuditEngine can be used to analyze ballot images to help weed out the poorly-marked ballots from the rest, and to suggest which paper ballots are good candidates for targeted auditing, only manual interpretation of voter-verifiable paper ballots can be used to guard against hacking or cyber attacks on election systems, or to reliably detect all machine misinterpretations of voter intent.
This is discussed in more detail at Some Observations re: Maryland Election Procedures, 2016 and Risk-limiting audit procedures must examine the paper ballots - input for Maryland Election Procedures, 2016.
See also Machine Retabulation is not Auditing, and Retabulations, Machine-Assisted Audits, and Election Verification, both by Mark Lindeman, Ronald L. Rivest, and Philip B. Stark, March 2013.
On the other hand, if voting systems can produce reliable, timestamped digital signatures of ballot images, and the evidence is shared with the public, that can be a great way to help preserve the chain-of-custody of paper ballots. This doesn't cover any issues that might occur prior to the signatures, but can still be helpful.
Site run by Neal McBurnett