Neal McBurnett
Boulder County augmented the Random Audit required by Colorado law. The improved methods, pre-approved by the Secretary of State: (1) compare hand counts of votes within batches against the actual counts produced during the live election, (2) use a larger sample of votes, (3) select paper audit contests that that were actually contested, and (4) focus more auditing on the contests with the smallest reported margins.
During the live vote-counting process, in preparation for the audit, the detailed election results were captured after each Mobile Ballot Box (MBB - the memory card that moves ballot information around) was processed by TALLY. The incremental votes, for each contest, contestant, and over and under votes that were attributable to each MBB, were calculated by subtracting the previous overall election results from the current results.
The SOS randomly selected three paper ballot contests to be audited on the central count machine, all of which were uncontested. The Canvass Board selected both of the two contested contests in addition.
The SOS selected seven eSlate DREs (used on Election Day in polling locations) and specific contests to be audited on each. For each selected eSlate: (1) all of the cast vote records were printed, (2) the VVPAT roll was retrieved, (3) for all of the contests all of the votes recorded on the cast vote record printouts were tabulated by hand (without knowing the computed results), (4) all of the votes recorded on the VVPAT printouts were tabulated by hand (without knowing the computed results), (5) the results of the two hand counts were independently compared, and (6) the results of the hand counts were independently compared to the election night incremental results for the same batch. No discrepancies were detected. A total of 66 ballots were audited this way.
The SOS selected one central count scanner and five contests to audit. None of these contests were contested. One (State Senate Dist 11) was not even in Boulder County, and two others (Rep District Attorney 20th District and Rep State Senate Dist 18) didn't have any candidates at all. These three were rejected for our audit.
The canvass board added the two contests where the races were contested, and a third uncontested race to make a total of 5. For each of the contested contests, without regard to which scanner was used, batches of paper ballots were randomly selected until a pre-determined percent of the total ballots counted for the contest were selected - 7.3 % in the closer race, and 6% in the other. For each contest, for each selected batch: (1) the live ballots were retrieved, (2) the total number of votes for each contestant, over, and under votes for that contest was hand counted (without knowing the computed results), (3) each hand count was compared to the corresponding count in the incremental election night report for the corresponding batch. One discrepancy was detected.
The number of votes hand counted for US Congressional District 2 did not match the system tally for those votes. The physical ballot most likely to have caused the difference was identified and retrieved. It contained "white out" that appears intended to change a vote from one contestant to another. The ballot image was displayed in Ballot Now and and the Audit Log was inspected. Apparently the resolution board made an error when attempting to resolve what appeared to the Ballot Now as a "damaged undervoted contest". A very similar resolution error occurred in the recount of the St. Vrain School District Ballot Issue 3a race in 2004 and was partly attributed to problems with the user interface of the Ballot Now application, which seem to still be there.
Still todo: Document the overall statistical level of confidence in the process based on Principles and Best Practices for Post-Election Audits September 2008
The one discrepancy in the 1726 paper ballots audited in the CD2 race could suggest on the order of 10-15 discrepancies in the race, and this problem is highly unlikely to have affected the outcome of the race.
The current audit procedure called for by the Colorado Secretary of State serves as a limited post election test of the voting machines, but does not actually audit the preliminary election results as reported. After the election, the procedure calls for counting a group of ballots both by machine and by hand count and then comparing the answers. It does not confirm any counts obtained during the election itself. Boulder audited both of the contested races for the primary - the 2nd Congressional District (CD2) and the 18th Senate District (S18). The races which were randomly selected by the Secretary of State's office were mostly uncontested. In addition, rather than rescanning ballots and MBBs and doing a partial recount, Boulder did an audit of the actual detailed preliminary results. These detailed preliminary results were produced by generating incremental reports after tallying each MBB during the original tally process, and subtracting each report from the previous report to get results for each MBB. Since each MBB represented a specific batch of paper ballots or a specific DRE, the original results could be tracked back to the original paper ballots or VVPATs. The audit of the VVPATs for the DREs thus represented acutal results as reported in the election. It was otherwise similar to the procedure documented by the Secretary of State. In Boulder 32980 ballots (about 94% of the total) were cast on paper and counted on a central scanner, and 2147 ballots were cast with a DRE. For the audit of the centrally scanned paper ballots, rather than only selecting 500 ballots for each race and rescanning those, as called for in the audit rule, Boulder audited more ballots, especially in the closer of the two races (CD2). We also decided to toss out random selections of audit units where the reported result is zero, after checking that they contained only results from precincts in the wrong districts. That is because there should be no valid ballots in those batches for the race in question, and some unproductive work could thus be avoided. We first generated a complete auditable report of the results for the two contested races. This was released by the county the day before the audit, in xml format: Public_auditable_report.xls Since the margin in the CD2 race was tighter than the margin in S18, we decided to audit at least 1650 relevant ballots in CD2, and 800 for S18. We selected audit units at random by tossing 2 dice and a coin until we got batches including that many relevant ballots. We ended up with 10 batches comprising 1726 contest ballots and 2878 total ballots in the CD2 race, and 3 batches comprising 803 contest ballots and 1069 total ballots in the S18 race. The audit detected a single error in the preliminary results. One additional vote for Fitz-Gerald in the race for the 2nd Congressional District was discovered. This did not affect the overall result, in which Jared Polis was declared the winner. The error was in audit unit 082 (MBB 563), where the audit uncovered a ballot that was not counted for Fitz-Gerald. It was tallied as an undervote instead. The voter had used whiteout to cover the entire box for Polis. During scanning, it was identified by the system as a "damaged ballot", and interpreted as a "resolved damaged undervoted contest". The system brought it to the attention of the operators during the manual resolution process, at which point it was mistakenly confirmed as an undervote. The same sort of error was detected during the audit of the recount of the St Vrain School District Ballot Issue 3a race in 2004, as previously reported. The result stemmed from a combination of operator error and a confusing user interface.
See the audit results spreadsheet for details on each audit unit selected.
Process note: for Serial A0A93E, there was no incremental report for just the associated MBB (MBB 341). They were combined with the incremental report results for MBB 432. So the team looked up the VVPAT roll for MBB 432 and hand-counted it. The total of the two MBBs matched the total of the two VVPATs (e.g. 7 for Toor, not just the 5 for Toor on Serial A0A93E).
Not that the audit of CD2 was only a partial audit, since parts of the district are also in Clear Creek, Gilpin, Adams and Jefferson counties.
Note that it was necessary, due to privacy concerns, to combine some batches together rather than reporting, e.g., how the only person to vote on a DRE in a particular precinct voted.
During the preparation for the audit we discovered that the "csv" (comma-separated-values) version of the cumulative report is deeply flawed. It is not actually structured as a normal csv file, with columns of data organized into rows for each result. Instead each row has a variable list of headers and associated results. For future elections we plan to use the XML version of the cumulative report, even though it is also very oddly structured.
For the actual selection of batches we tossed one coin and two 10-sided dice, one red and one blue. The coin flip determined the 100's digit of the race to be audited: tails was "0" and heads was "1". The blue die determined the 10's digit, and the red die was the 1's digit. If the resulting number was larger than the number of audit units available (142), we re-flipped the coin and re-rolled both dice.
Note that it is also critical to canvass all the ballots to make sure that only the proper ballots were counted, and that no ballots were missing. That was the other main task of the canvass certification process.
This report still lacks some notes on statistical significance. Thanks to Clerk Hillary Hall for championing this major step forward in auditing. Thanks to Al Kolwicz and Gary Boucher of the Canvass Board for their great work, and their help writing this up. Updated: 2008-09-30