← All posts

'Certified' means it passed a test once. It doesn't mean it's safe today.

A security review cleared the system. Then outside researchers found the flaw—while 280,000 people were already voting. The word 'certified' did not stop that from happening.

It is the third week of March 2015, and the New South Wales state election is underway. Hundreds of thousands of voters have already cast their ballots online through a system called iVote. The election authority has run its security checks. The system has been reviewed. Nobody has raised an alarm.

Then two researchers — J. Alex Halderman at the University of Michigan and Vanessa Teague in Australia — point a browser at the running, live system.

They are not election officials. They have no insider access. They are working from outside, examining what any network-capable adversary could reach.

What they find is not a theoretical vulnerability buried in an obscure protocol. It is a working attack path. The iVote system was loading code from an external analytics server over a connection vulnerable to then-known TLS weaknesses — the class of attack known as FREAK and Logjam. A network-positioned attacker could have used that channel to manipulate votes and compromise the privacy of ballots. And the telephone verification service built in specifically so voters could confirm their vote had been recorded correctly? It was itself susceptible to manipulation, which meant the check was checking nothing.

The pre-election security review had not found any of this.

By the time the researchers reported their findings, roughly 280,000 votes had already been cast.

The word 'certified' is doing enormous work here — and not earning it

Here is what 'certified' actually means in election-technology contexts.

It means that at a defined point in time — before deployment, in a controlled lab environment — a system was tested against a written specification, passed those tests, and received a document saying so.

That document does not update itself. It does not re-run the tests when a dependency changes. It does not know that a third-party analytics script was added to the web stack after the audit. It does not know that the TLS library version in production differs from the one the reviewers examined. It does not know anything that happened after the moment the stamp was applied.

Certification is a snapshot. The system keeps running in a world that doesn't stay still.

This is not a flaw peculiar to New South Wales. It is structural to every certification regime that works by checking a system once and then trusting the result. The snapshot is frozen. The threat environment is not.

What Halderman and Teague found was not the result of a rushed or incompetent pre-election review. It was the result of reviewing a system that then continued to exist and change. The external analytics server the code loaded from — that dependency was part of the attack surface, and attack surfaces are not static.

The lesson is not 'run a better pre-election audit.' The lesson is that a one-time check cannot substitute for continuous, independent, external verifiability. And the specific sting of iVote is that the verification mechanism — the phone-call confirmation — was itself compromised. The system's own proof-of-correctness could have been gamed.

A lock you cannot check is not a lock. It is a prop.

A federal court made the same discovery — about a paper ballot you can hold in your hand

Five years later, in a very different setting, a U.S. federal court in Atlanta was reading expert testimony about a system that does produce a physical artifact. Georgia's Dominion ballot-marking devices print a paper ballot. You can hold it. You can take it to a poll worker. You can, in theory, inspect it.

Except there is a problem.

The ballot-marking device prints two things: human-readable text showing your selections, and a QR code. It is the QR code the scanner reads when it tabulates votes. And a QR code is not human-readable. You cannot look at it and confirm it says what the text says.

In its October 2020 opinion in Curling v. Raffensperger, the U.S. District Court for the Northern District of Georgia reviewed expert testimony including from Dr. Halderman — the same researcher who had looked at iVote — about what a cyberattack could do. The court found the system 'does not provide a verifiable and auditable ballot record because it relies on the QR code for vote tabulation and that code itself cannot be read and verified by the voter.'

The court also quoted the National Academies' 2018 conclusion directly: no technical mechanism currently exists to ensure a vote-counting application produces accurate results, and testing alone cannot ensure systems have not been compromised.

Read that sentence again. Testing alone cannot ensure systems have not been compromised.

That is not a fringe research claim. That is a federal court quoting the National Academies, in a published legal opinion. Testing — which is what certification fundamentally is — cannot guarantee the integrity of what is actually running.

The Georgia court, facing a real election weeks away, declined to order a last-minute switch to hand-marked paper ballots. It weighed the disruption against the risk and chose continuity. That is a defensible pragmatic call. But it does not resolve the underlying problem: the artifact voters could inspect was not the artifact being counted.

Certification had cleared the system. The system's verifiability gap remained.

The pattern behind both cases: the snapshot-versus-live problem

New South Wales iVote and Curling look superficially different. One involves an internet-voting system running on networked servers. The other involves a paper ballot produced in a polling booth.

But they share the same structural failure.

In both cases, what was certified and what was actually running were not provably the same thing — and there was no mechanism available to the public, or to voters individually, to close that gap in real time.

In iVote's case, the gap was between the system as reviewed and the system as deployed, which had a live dependency on an external server no reviewer had assessed. The verification feature meant to catch problems had itself been compromised.

In Georgia's case, the gap was between what the voter could see (human-readable text) and what the machine actually counted (a barcode the voter cannot decode). The paper ballot that was supposed to enable auditing was encoding information the auditor — the voter — cannot read. A hand recount of those paper ballots audits the barcode's output, not the voter's intent. As the court noted, if the barcode encodes something different from what the text says, the hand count of the paper confirms the manipulation, not the voter's choice.

Certification cleared both systems.

Neither gap was detectable by looking at a certification document.

What would actually close this gap?

Certification is not useless. A system that has never been tested by anyone is worse than one that has been tested once. The problem is treating it as sufficient — as the conclusion rather than the beginning of a verification process.

What would make it sufficient?

Three things that are technically achievable and currently rare.

First: attestation of what's actually running, not just what was reviewed. A system that can cryptographically prove to an outside observer that the code executing votes today is the same code that was tested — not a later version, not a patched dependency, not a swapped-out module — closes the snapshot gap. This is not science fiction. It is a solved problem in software engineering. It is simply not required in most election-technology certification regimes.

Second: independent external verifiability that does not depend on the vendor or the authority. The iVote lesson is that the system's own verification mechanism can be attacked. A check that the system itself administers is not independent verification. Verifiability means someone outside — a researcher, a political party, a citizen with a laptop — can confirm the system's behavior against a tamper-evident record they did not receive from the system being checked. Germany's Federal Constitutional Court understood this in 2009 when it held that voters must be able to verify the essential steps of an election 'without any specialist knowledge'. The standard is not 'experts can review it.' It is 'ordinary people can check it.'

Third: the artifact being counted must be the artifact the voter can inspect. A QR code no voter can read fails this. A paper ballot the voter marks by hand, which a scanner reads optically, comes much closer — because a human being looking at the same paper can assess the same mark the machine assessed. The Dutch commission that recommended returning to paper in 2007 called this principle controleerbaarheid — checkability — and treated it as non-negotiable: 'there are no secrets in the election process and questions must be answerable and the answers checkable and verifiable.' That is not a description of a QR-code ballot.

The audits can still leave the question open

The objection you are probably already forming: what about post-election audits? What about hand counts?

They help. A risk-limiting audit that samples paper ballots against machine totals, as Colorado pioneered in 2017, provides real statistical evidence that the machine count reflects the paper record. Georgia's full hand tally of nearly 5 million presidential ballots in November 2020 confirmed the machine result to within about a tenth of a percent.

But audits of QR-code ballot-marking-device output audit the QR code's interpretation, not the voter's intent — unless the auditor separately verifies that the human-readable text and the barcode match. That step is not standard.

And audits happen after the fact. In iVote's case, the election was live before the flaw was found. A post-election audit of an online voting system cannot recover votes that were silently altered in transit, because there is no independent copy of what was cast to audit against. The whole architecture of internet voting makes the 'paper as ground truth' model unavailable.

There is also the irreducible question about hand counts themselves. Antrim County, Michigan, ran a full hand tally of roughly 15,700 presidential ballots in December 2020 — and the hand count still differed from the machine tabulation by about a dozen votes. In a race decided by hundreds of votes, a dozen unexplained discrepancies between the 'gold standard' count and the machine count do not resolve the question; they extend it. The audit confirmed the method is imperfect. 'The audit confirmed it' is a reassurance, not proof, when the auditing method itself carries irreducible error.

The answer is not to abandon audits. The answer is to build systems where the ground truth is cryptographically protected, publicly visible, and independently checkable — so that 'the audit confirmed it' means something falsifiable.

What you still cannot verify — and what would make it verifiable

Here is what remains opaque right now in most jurisdictions running certified voting systems.

You cannot verify that the code running on election day is the code that was certified. No public attestation mechanism exists for most deployed systems.

You cannot verify, as a voter, that a QR-code ballot encodes your choices correctly. No independent decoding tool is provided in the polling place.

You cannot verify that an internet voting system's live dependencies match the system that was reviewed. iVote's analytics-server dependency was not in the audit scope.

You cannot verify that a pre-election security review covered the attack surface as it existed on election day, because that surface changes.

What would make these things verifiable — independently, without trusting the vendor or the certifying body?

Published, versioned software builds with cryptographic hashes, checked by independent parties at the point of deployment. Voter-verifiable paper records where what the voter marks is what the scanner reads and what any auditor can assess. Real-time, machine-readable precinct-level result publication that anyone can download and cross-check. And adversarial public testing — not a one-time pre-election review, but an ongoing, open invitation for outside researchers to find problems, as Brazil's electoral court has institutionalized with its public security testing program.

'Certified' is a statement about the past. What elections need is a statement about right now — one that any member of the public can independently confirm, without taking anyone's word for it.

The New South Wales researchers found the iVote flaw from outside, with no special access, during a live election. That is both the warning and the model. The warning: closed, one-time review leaves gaps. The model: outside researchers with access to a running system can find what insiders miss.

The difference between a disaster and a near-miss is whether the system's architecture lets outsiders check it — before the votes are cast, while they are being cast, and after they are counted.

Certification says the system was checked.

Verifiability says it can be checked.

Those are not the same thing, and no election authority's assurance, however sincere, can close the gap between them.

See how common this verification gap is across the world →

Read the two-minute version →

Explore the full case database →

Sources

See all postsExplore the gaps