Scientific experts are prone to bias, overestimate their certainty, and government systems are not good at adjusting to new science.

Those admonitions come from Supreme Court Justice Sonia Sotomayor in a statement addressing the Supreme Court’s refusal to hear the case of Charles C. McCrory v. Alabama. In the piece, Justice Sotomayor asks what courts should do when faced with convictions “resting on science that has now been wholly discredited?”

The question, and Justice Sotomayor’s discussion, offer lessons for how government and politicians should apply science and technology with humility and flexibility.

These could easily be applied to other areas of government, including environmental policy. Agencies often rely on “experts” within the organization to offer their assessment on the political agenda of the agency. Often the agency sets the rules and context for those “expert” opinions, increasing the risk of bias and reducing the likelihood that agencies will accept contradictory science and admit they were wrong.

Addressing changes in forensic science, Justice Sotomayor wrote, “One in four people exonerated since 1989 were wrongfully convicted based on false or misleading forensic evidence introduced at their trials.” Just such an error is alleged in the McCrory case.

In 1985, Charles McCrory was convicted of the murder of his wife when a bite mark expert testified that “the bite marks on the victim matched the appellant's upper teeth.” Subsequently, that expert recanted his testimony and bite mark analysis has been found to be unreliable. In spite of that, courts have refused to retry the case, nearly forty years after he was first put in prison. Last year he was again denied parole.

Justice Sotomayor noted that the areas of science that are more error prone are those that rely on the judgment of “experts” as opposed to laboratory science. She cited a 2009 report from the National Academy of Sciences that assessed the accuracy of forensic science. Contrasting DNA evidence with the judgment of experts, the authors wrote, “In terms of scientific basis, the analytically based disciplines generally hold a notable edge over disciplines based on expert interpretation.” They added that, “The findings of forensic science experts are vulnerable to cognitive and contextual bias.”

All of these same things could be said about the judgement of experts in other areas of public policy. Their assessments are subject to change over time. They are prone to bias. And they can convey more certainty than is warranted.

Courts, government agencies, and politicians have incentives to ignore the limitations of expert judgment. Courts and policymakers can’t always wait until the science is “settled.” In another Supreme Court case from 1992, Daubert v. Merrell Dow Pharmaceuticals, Inc., the majority wrote, “there are important differences between the quest for truth in the courtroom and the quest for truth in the laboratory. Scientific conclusions are subject to perpetual revision. Law, on the other hand, must resolve disputes finally and quickly.”

To deal with these various pressures and uncertainties, government systems should take into account the fact that scientific understanding will improve over time and may, as in the case of bite mark analysis, change entirely. Justice Sotomayor added that “science evolves slowly rather than in conclusive bursts,” making it important that court procedures allow for new information to be considered long after a sentence has been handed down.

This is particularly true when dealing with the subjective judgements of “experts.”

Although the Court turned down the case, Sotomayor calls on legislatures to address the problem, allowing courts to consider new scientific understanding. Pointing to legislation in Texas and California, she wrote that, “targeted legislative reform can allow courts to address convictions legislative reform can allow courts to address convictions based on trial evidence that has been repudiated by the scientific community.”

Policymakers and courts should also take steps to reduce the impact of bias among experts in all fields.

Calling for an improvement in the study of forensic science, the National Academy of Sciences suggested the creation of an independent agency. Addressing the risk of bias, the authors of the NAS report had several suggestions:

• It must not be in any way committed to the existing system, but should be informed by its experiences.
• It must not be part of a law enforcement agency.
• It must have the funding, independence, and sufficient prominence to raise the profile of the forensic science disciplines and push effectively for improvements.

The NAS authors recognized that forensic experts located within “the existing system” or a law enforcement agency could not act independently. These problems are not unique to the justice system. Independent oversight is an important part of holding government agencies accountable for their decisions and ensuring they are responding to science and data, not ideology or political pressure.

Justice Sotomayor did a service in raising this issue, not only to provide justice for those wrongly accused, but for policymakers everywhere looking to make government policy more accurate and accountable.