Neal McBurnett, at the 2018-12-07 MIT Election Audit Summit
In 2017, Colorado became the first state to regularly conduct risk-limiting audits (RLAs). Colorado's successes are grounded in 15 years of multi-partisan efforts to promote and pilot election auditing. Here are some of the lessons I've learned along the way. For more background, and links to a wealth of material, see The Colorado Risk-Limiting Audit Project (CORLA)
One of the clearest lessons is that pilot audits with input from people experienced with risk-limiting audits are enormously helpful and highly recommended. The whole community learns from pilot audits.
Colorado has demonstrated that with a good systems, processes and data, you can do ballot-level risk-limiting audits which limit the risk that tabulation errors or attacks result in getting the wrong outcome. This can be done at scale, in hundreds of contests, in dozens of counties, and across overlapping districts in a state. They can also be done Efficiently. Colorado audited less than ten thousand ballots statewide. Besides the fully risk-limiting audits, simultaneous "opportunistic" audtis can gather evidence on and report risk levels for all the rest of the contests. Furthermore, Colorado's new statewide system is among the most cost effective and best for auditing: central-count scanners with BMDs available for accessibility.
These highly efficient ballot-level RLAs can be done with equipment from multiple vendors. In 2015, four vendors presented and piloted systems that could do ballot-level comparison RLAs in 2015. They were all central count scanning systems, from Dominion, Hart, ClearBallot and ES&S
Colorado funded the development of software to help manage the audits which is now open source ColoradoRLA. This system continues to be enhanced, and can be used for free by any jurisdiction, with support available from multiple organizations.
There is widespread, transpartisan consensus on the need for both paper ballots and audits. An early example was in 2003, when four parties (Republican, Democrat, Libertarian and Green) announced a joint consensus in Boulder Colorado. An excellent overview of the modern case is in the National Academies report from 2018: Securing the Vote: Protecting American Democracy | The National Academies Press
While we've made huge steps forward, there is still much to do. Why is it taking so long to adopt robust audits?
This underscores why it is critical to support and adopt the Common Data Standards work by the EAC / NIST VVSG-Interoperability task force.
We need format standards! See a helpful overview presentation by John Wack: Overview of VVSG-Interoperability Common Data Formats (two presentations).
Common data formats are published or in-the-works for several use cases. Election Results reporting (SP 1500-100) is used in OH, NC, LA County. Other states are in progress. The Election Log Export CDF will soon be published as SP 1500-101. The Voter Records Interchange CDF is slated for review by VR vendors, to be published as SP 1500-102. I has seen initial use in OH and by OSET.
The Cast Vote Records CDF schema should be published soon as SP 1500-103. The ongoing development and documentation of election process business models and voting method descriptions is also very beneficial.
Audits which are conducted by elections officials should also be highly accessible to the public, and the critical inputs to and results of the audit should be shared openly. Otherwise, audits may be convincing to officials, but leave losing candidates and the public with little evidence to go on.
A document presenting details on what the public should have access to is available at Public RLA Oversight Protocol, by Stephanie Singer and Neal McBurnett, 2017.
Briefly, the elements it covers are:
The ColoradoRLA software includes an rla_export tool to provide necessary data for Oversight Protocol in csv/json formats
rla_report software is in progress to interpret the exported data, confirm that the right ballots were selected, and check the risk level calculations., to help implement these oversight steps. This code will also be open source, and verifiers should be encouraged to check it and/or implement their own oversight processes and code.
In its recent audits, Colorado has shared more useful data on its audits, in more useful ways, than probably any other jurisdiction. Officials can be very proud of their results. Officials with access to all the audit data, including the Cast Vote Records (CVRs) etc, can be more confident in the outcomes of more contests than anywhere else in the country, and certainly more efficiently than anywhere else.
Unfortunately, while this is much more transparency than in the past, losing candidates and the public still encounter several crucial holes in the oversight protocol. Some summary data is not available yet, principly because due to an unusual confluence of challenging circumstances, the state is still wrestling with ballot anonymity issues which have limited the availability of the original CVRs to the public. That means the public can't check tally totals, and can't check ballot interpretations in real time, or sometimes at all.
We give kudos to the amazing ongoing accomplishments by both the state and the counties under very challenging circumstances, and look forward to resolving the various obstacles to full transparency.
A model for that sort of transparency has already been seen in the audits in Boulder CO in 2008, which, before the audit, successfully generated auditable data. In some cases that required merging small sets of ballots into larger sets, all to be audited together, in order to eliminate anonymity concerns. See Boulder County 2008 General Election Audit for the data and open-source software for those batch-comparison audits.
More detail on relevant challenges and good solutions is available at Preserving Anonymity of Cast Vote Records, by Mark Lindeman, John McCarthy, Neal McBurnett, Harvie Branscomb, Ron Rivest, and Philip Stark, 2017-08-03.
Detailed reporting on discrepancies in Colorados audits is still in-progress. But it is evident that there are still some instances of errors in data entry. To avoid that, the software should inform the Audit Board of each discrepancy right after entry. That would help with discrepancy investigations, provide much more useful and actionable quality control feedback, and enhance trust in the process on all sides.
The software needs enhancements in reporting convenience and analysis. It should make it easy to view discrepancies, and risk levels for opportunistically audited contests. That is particularly challenging for the wide variety of districts, each involving samples taken in a variety of counties. The software should also automatically generate an "Audit Center" web site with full data for the public.
The software should be further modularized for use with external risk-level calculation modules, covering additional auditing methods like SUITE, Bayesian RLAs, etc.
We need new approaches to handle in-precinct/vote center scanners, which randomize ballots and/or CVRs. They complicate the process of matching paper ballots with CVRs.
We need upgraded support for batch-comparison audits, which yield risk reduction which is predictable, easy to plan for and easy to understand. We should also provide better support for ballot-polling audits, though they can be unpredictable and impractical for some of the most interesting contests with tight margins.
We should foster collaboration between clerks, privacy experts, and toolsmiths around preserving anonymity, especially for the complicated situation in Colorado.
We should audit more systems involved in elections: voter registration, signature verification, envelope sorting, ballot reconciliation etc.
Often in any given election, public attention is focused on particular circumstances. Random selection of ballots to audit is essential for good risk reduction, but we should also be prepared to directly address specific concerns and unusual circumstances.
We should encourage candidates and the public to identify additional interesting ballots to target for auditing. THey could be chosen based on analysis of the CVRs, based on mark density data, or even based on ballot images.
Finally, we should promote more public participation in audits. We could print ballot tracking pages with QR codes, and provide an app that public observers could use to photograph ballots along with the tracking-sheet QR codes. That could assist the public in conducting their oversight, and facilitate sharing of a series of confidence-inducing tweets like "I verified the votes on this ballot".
I'm deeply grateful to Paul Tiger for being the first to encourage me to get involved in election verification back in 2002. A long list of colleagues since then have offered expertise, insights, enthusiasm, and comraderie, including Joe Pezzillo, Paul Walmsley, Ron Rivest, Philip Stark, Harvie Branscomb, and Hillary Hall and her Boulder County team, The Election Verification Network has been invaluable in all of this work. The Colorado legislators who helped us pass laws in 2005 and 2009 to require audits deserve much credit. I've been incredibly impressed at the dedication of the Colorado SOS staff and Clerks! The Free & Fair Team that took on the daunting challenge of signing up to write the initial ColoradoRLA software under incredible contraints of time and resources deserve my eternal gratitude. They went beyond the call of duty. And the Democracy Works team that has continued to improve the software, enhancing the user interface and digging deep into the internals to re-work it for multi-county contests has been incredible also.
Updated versions of this narrative report will be available at http://bcn.boulder.co.us/~neal/elections/audit-summit.html